PythonCK.decorators package

Submodules

PythonCK.decorators.cache_to_file module

cache_to_file and underlying shelf (dict with timestamp).

class PythonCK.decorators.cache_to_file.cache_to_file(func=None, *args, **kwargs)[source]

Bases: PythonCK.decorators.meta.AbstractClassbasedDecorator

Cache the result of this function, persist on the disk (using pickle/shelve).

Extra flag: force_reload

If flag force_reload=True is given additionally into decorated function, the wrapped function will be called regardless of the cache even if it’s available. This is useful in case where the user intent to refresh the cache to newer value, as well as prolonging cache expiredate.

Extra flag: early_giveup

If flag early_giveup=True is given additionally into decorated function, the wrapped function will return None immediately if there’s no cache available before this call. This is useful in the situation where it’s antipicated that the function call will be slow (expensive), and the calling can be postponed to better context. - If cache is already expired, early_giveup will also return None. - The default return value is None

Note

force_reload and early_giveup are mutually exclusive. Exception will be raised if both of them are True simultaneously.

Note

Cannot be used on instance.method?

Parameters:
  • basedir (str) – Name of the directory where the cache should reside. If None, this will be determined automatically.
  • timeout (int) – Timeout until the cache expire, in seconds.
  • input_skip_write (None, bool, callable) – Input-dependent callback which, if true, will not persist the cache.
  • output_skip_write (None, bool, callable) – Output-dependent callback which, if true, will not persist the cache.
  • sharding (bool) – If True, instead of collection calls from same function into one shelve, shards the result into single-pickled-file, one file per one unique input. This is better in term of concurrency, but may make directory more dirty…

Usages:

>>> func = getfixture('f_cache_to_file')
>>> func.counter        # hits & misses
(0, 0)
>>> func(111)           # simple call
((111,), {})
>>> func.counter        # one miss
(0, 1)
>>> func.contains(111)  # it's cached
True
>>> _ = func(111)       # call again, expect hit
>>> func.counter
(1, 2)
>>> _ = func(222)       # call with second key
>>> func.contains(222)
True
>>> func.counter
(1, 3)
>>> _ = func(222)       # call with second key, again
>>> func.contains(222)
True
>>> func.counter
(2, 4)
>>> func(111)           # call with 1st key, again
((111,), {})
>>> func.contains(111)
True
>>> func.counter
(3, 5)

>>> func(111, force_reload=True)    # Using force_reload
((111,), {})
>>> func.counter
(3, 6)
>>> func(333, early_giveup=True)    # Using early_giveup

>>> func.contains(333)
False
>>> func.counter
(3, 6)
>>> _ = func(333)  # now there's cache
>>> func(333, early_giveup=True)
((333,), {})
>>> func.counter
(4, 8)

>>> func('arg', kw='kwarg')   # same thing for kwargs
(('arg',), {'kw': 'kwarg'})
>>> func.counter
(4, 9)
__abstractmethods__ = frozenset([])
__module__ = 'PythonCK.decorators.cache_to_file'
__setitem__(args_kwargs, result)[source]

Backdoor interface to allow putting data into shelf directly without need for a call to host function at all.

Useful for case where batch-call is prefered, but optimize caching for single entry from the batch.

Accept 2 style of keys:
  • ready-key, made via staticmethod makekey
  • raw-key, compose of 2-tuple of (args,kwargs)
Usage:
>>> func = getfixture('f_cache_to_file')

>>> func.contains(111)
False
>>> func[(111,),{}] = 111     # Set args,kwargs and results manually
>>> func.contains(111)
True
>>> func.counter
(0, 0)
>>> func(111)                 # a new call should hit instantly
111
>>> func.counter
(1, 1)

>>> key = func.makekey('arg', kw='kwarg')   # Another set style, via makekey
>>> func[key] = 'key_via_makekey'
>>> func.counter
(1, 1)
>>> func('arg', kw='kwarg')
'key_via_makekey'
>>> func.counter  # hit instantly
(2, 2)

>>> func[1] = 2   # Other case is not accepted
Traceback (most recent call last):
...
ValueError: Invalid key for backdoor __setitem__
__slots__ = ('_count_hit', '_count_total', '_extra', '_isw', '_osw', '_shelfid', '_shelf', '_timeout')
contains(*args, **kwargs)[source]

Return True if given args-kwargs has been cached.

>>> func = getfixture('f_cache_to_file')
>>> func.contains(0)
False
>>> _ = func('111')
>>> func.contains('111')
True
>>> func.contains('111', kw=222)
False
counter

Return the hit/total stats of this cacher.

is_expired(time_last)[source]

Return True if data already expired. False (not expired) if timeout not given

>>> from datetime import datetime, timedelta
>>> _ = getfixture('chtmpdir')
>>> f = cache_to_file(func0, timeout=3600, basedir='.')
>>> f.is_expired(datetime.now())
False
>>> f.is_expired(datetime.now()-timedelta(10000))
True
>>> f.is_expired(None)
True
static makekey(*args, **kwargs)[source]

Handle making unique key from args, kwargs, exact signature like a function.

report_stats()[source]

Report the hit/total stats.

>>> f = getfixture('f_cache_to_file')
>>> c = getfixture('caplog2')
>>> _ = f(42)
>>> _ = f(42)
>>> f.report_stats()
>>> print(c.record_tuples[-1][-1])
cache_to_file: Hit rate : 1/2 = 0.50
shelf

Use for cache-viewer. Lazily create shelf (instead of opening dangle shelf when func is defined but unused)

shelfid
timeleft(key)[source]

Given a key of the shelf, return the time left until the cache is expired. Used in auxiliary method cache viewer:

>>> f = getfixture('f_cache_to_file')

## By default, there's no cache time-out
>>> _ = f(42)
>>> f.timeleft(f.makekey(42)) is None
True

## Check the timeout
>>> _ = getfixture('chtmpdir')
>>> f = cache_to_file(func0, timeout=60, basedir='.')
>>> _ = f(42)
>>> f.counter
(0, 1)
>>> t = f.timeleft(f.makekey(42)).seconds
>>> (t > 0) and (t < 3600)
True
>>> _ = f(42) # try again, without timeout
>>> f.counter
(1, 2)
PythonCK.decorators.cache_to_file.safe_name(func)[source]

Return safe name to be used as identifier.

PythonCK.decorators.decorators module

Collection of several useful decorators.

PythonCK.decorators.decorators.deprecated(func)[source]

This will print warning whenever this decorated function is called.

>>> caplog = getfixture('caplog2')
>>> @deprecated
... def f_deprecated():
...   pass
>>> f_deprecated()

>>> 'WARNING' in caplog.text and 'deprecated' in caplog.text
True
PythonCK.decorators.decorators.report(*args, **kwargs)[source]

Decorator to print information about a function call for use while debugging. Prints function name, arguments, and call number when the function is called. Prints this information again along with the return value when the function returns.

It it’s used inside Gaudi, the debug level is too polluted. Hence additional flag if given

If hide_output, will show only the call, but not output result

>>> caplog = getfixture('caplog2')

>>> @report
... def foo():
...   return 'line1\nline2'

>>> _ = foo()
PythonCK.decorators.decorators.singleton(cls)[source]

Use this on a class to quickly make it becomes singleton.

Usage:

>>> @singleton
... class Foo(object):
...   pass

>>> x1 = Foo()
>>> x1.val = 555
>>> x2 = Foo()
>>> x1 == x2
True
>>> x2.val
555

PythonCK.decorators.lazy module

Collection of lazy (caching) decorators

PythonCK.decorators.lazy.lazy_instance(func)[source]

To be used on the function that return some instance. The function will delay its call until its resultant instance is used (via __getattr__). A prime example is PyrootCK.import_tree.

Usage:

>>> IS_INIT = False

>>> class Heavy(object):
...   def __init__(self):
...     global IS_INIT
...     IS_INIT = True
...   def foo(self):
...     return 42

>>> @lazy_instance
... def get_heavy():
...   return Heavy()

>>> x = get_heavy()
>>> IS_INIT
False
>>> x
<PythonCK.decorators.lazy...>
>>> x.foo()
42
>>> IS_INIT
True

## Triggered by setter
>>> IS_INIT = False
>>> y = get_heavy()
>>> IS_INIT
False
>>> y.attr = 'val'
>>> IS_INIT
True
>>> y.attr
'val'
PythonCK.decorators.lazy.lazy_property(func)[source]

Lazy function, only for getter instance-method!

Usage:

>>> class Foo(object):
...   slow_count = 0
...   @lazy_property
...   def slow(self):
...     self.slow_count += 1
...     return 'some slow result'

>>> foo = Foo()
>>> foo.slow_count
0
>>> foo.slow
'some slow result'
>>> foo.slow_count # increase
1
>>> foo.slow
'some slow result'
>>> foo.slow_count  # not increase anymore!
1
PythonCK.decorators.lazy.memorized(func)[source]

Memorize the computed result of this function until the end of session:

## Define
>>> @memorized
... def f_mem(arg1, arg2):
...   return arg1+arg2

## Check consistency
>>> x1 = f_mem(10, arg2=20)
>>> x2 = f_mem(10, arg2=20)
>>> x1 == x2
True

PythonCK.decorators.meta module

class PythonCK.decorators.meta.AbstractClassbasedDecorator(func=None, *args, **kwargs)[source]

Bases: object

Baseclass to be used for all class-based decorator involving optional init arg.

The difficulty lies in the fact that there’re 2 ways to init a decorator instance.

Without kwargs:

  • __init__ will have func equal to the decorated function.
  • __call__ will have all (args,kwargs) input for func

With kwargs:

  • __init__ will have all args to init decorator, but no func available yet
  • __call__ will be invoked 2 times:
    • 1st-time: func will be passed as first arg, save it, then return self to invoke 2nd
    • 2nd-time: The (args,kwargs) for decorated function will be received here.

Note: - Not possible to use simple args for init, too ambiguous. - The decorated function is accessible through property self.func

Its setter self.func = xxx will be handled implicitly under the hood, so there should be no need to call this manually.

Test cases:

## Abstract, need implementation

>>> class Foo(AbstractClassbasedDecorator):
...   pass
>>> Foo()
Traceback (most recent call last):
...
TypeError: Can't instantiate abstract class Foo with abstract methods _run, _setup

>>> class foo(AbstractClassbasedDecorator):
...   def _setup(self, *args, **kwargs):
...     pass
...   def _run(self, *args, **kwargs):
...     return self.func(*args, **kwargs)

## Decorating function, with/without decorator kwarg.

>>> @foo
... def func_without_kwargs():
...   return 'without'
>>> func_without_kwargs()
'without'

>>> @foo(kw='kwarg')
... def func_with_kwargs():
...   return 'with'
>>> func_with_kwargs()
'with'

## Decorating instance method

>>> class Aileus:
...   @foo
...   def bondus(self):
...     return 'bondus'
...   @foo(kw='kwargs')
...   def callus(self):
...     return 'callus'
>>> a = Aileus()
>>> a.bondus()
'bondus'
>>> a.callus()
'callus'

REF:

__abstractmethods__ = frozenset(['_run', '_setup'])
__call__(...) <==> x(...)[source]
__dict__ = dict_proxy({'_abc_cache': <_weakrefset.WeakSet object>, '_run': <function _run>, '__dict__': <attribute '__dict__' of 'AbstractClassbasedDecorator' objects>, '_abc_registry': <_weakrefset.WeakSet object>, '__abstractmethods__': frozenset(['_run', '_setup']), '_setup': <function _setup>, '__module__': 'PythonCK.decorators.meta', '__call__': <function __call__>, '_abc_negative_cache_version': 31, 'func': <property object>, '_abc_negative_cache': <_weakrefset.WeakSet object>, '_logd': <function _logd>, '__weakref__': <attribute '__weakref__' of 'AbstractClassbasedDecorator' objects>, '__doc__': "\n Baseclass to be used for all class-based decorator involving optional init arg.\n\n The difficulty lies in the fact that there're 2 ways to init a decorator instance.\n\n Without kwargs:\n\n - ``__init__`` will have ``func`` equal to the decorated function.\n - ``__call__`` will have all (args,kwargs) input for func\n\n With kwargs:\n\n - ``__init__`` will have all args to init decorator, but no func available yet\n - ``__call__`` will be invoked 2 times:\n\n * 1st-time: func will be passed as first arg, save it, then return self to invoke 2nd\n * 2nd-time: The (args,kwargs) for decorated function will be received here.\n\n Note:\n - Not possible to use simple args for init, too ambiguous.\n - The decorated function is accessible through property ``self.func``\n Its setter ``self.func = xxx`` will be handled implicitly under the hood,\n so there should be no need to call this manually.\n\n Test cases::\n\n ## Abstract, need implementation\n \n >>> class Foo(AbstractClassbasedDecorator):\n ... pass\n >>> Foo()\n Traceback (most recent call last):\n ...\n TypeError: Can't instantiate abstract class Foo with abstract methods _run, _setup\n\n >>> class foo(AbstractClassbasedDecorator):\n ... def _setup(self, *args, **kwargs):\n ... pass\n ... def _run(self, *args, **kwargs):\n ... return self.func(*args, **kwargs)\n\n ## Decorating function, with/without decorator kwarg.\n\n >>> @foo\n ... def func_without_kwargs():\n ... return 'without'\n >>> func_without_kwargs()\n 'without'\n\n >>> @foo(kw='kwarg')\n ... def func_with_kwargs():\n ... return 'with'\n >>> func_with_kwargs()\n 'with'\n\n ## Decorating instance method\n\n >>> class Aileus:\n ... @foo\n ... def bondus(self):\n ... return 'bondus'\n ... @foo(kw='kwargs')\n ... def callus(self):\n ... return 'callus'\n >>> a = Aileus()\n >>> a.bondus()\n 'bondus'\n >>> a.callus()\n 'callus'\n\n REF:\n\n - http://tech.novapost.fr/python-class-based-decorators-en.html\n - http://code.activestate.com/recipes/577452-a-memoize-decorator-for-instance-methods/\n\n ", '__init__': <function __init__>, '__get__': <function __get__>})
__get__(instance, owner)[source]
__init__(func=None, *args, **kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

__module__ = 'PythonCK.decorators.meta'
__weakref__

list of weak references to the object (if defined)

func

Return the instance of decorated function

PythonCK.decorators.meta.optional_arg_decorator(decorator)[source]

Meta-decorator to let the function-based decorator takes both argfull and argless constructor signature.

Usage:

## Define decorator
>>> @optional_arg_decorator
... def foo(func, arg=None):
...   def wrap(*args, **kwargs):
...     return func(*args, **kwargs), arg
...   return wrap

## Without arg
>>> @foo
... def bar():
...   return 'bar'
>>> bar()
('bar', None)

## With arg
>>> @foo('more')
... def baz():
...   return 'baz'
>>> baz()
('baz', 'more')

REF:

PythonCK.decorators.safe_makedir module

PythonCK.decorators.safe_makedir.safe_makedir(target)[source]

Workaround the race condition when multiple instances run on same node. https://stackoverflow.com/questions/1586648/race-condition-creating-folder-in-python

Also:

  • make sure it’s string (or convert to string from py.local)
  • expand var to abs path
  • return the string path.
>>> import os
>>> _ = getfixture('chtmpdir')
>>> safe_makedir('subdir')
'subdir'
>>> os.path.exists('./subdir')
True

PythonCK.decorators.shelves module

Shelf is an underlying persistence for decorator cache_to_file where the expiration is also persisted.

class PythonCK.decorators.shelves.ShardedShelf(shelfid)[source]

Bases: _abcoll.MutableMapping

Persist EACH keys into individual file, via pickle.

__abstractmethods__ = frozenset([])
__contains__(key)[source]
__delitem__(key)[source]
__getitem__(key)[source]

Return ( value, moditfed time )

__init__(shelfid)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

__iter__()[source]

Return all keys (not the hashed path)

__len__()[source]
__module__ = 'PythonCK.decorators.shelves'
__setitem__(key, val)[source]

Set the file ( put in both key-val because of non-reversible hashing )

__str__()[source]

Show nice entry store, inspired by GAE.

clear()[source]

Remove all cache.

class PythonCK.decorators.shelves.UnshardedShelf(shelfid)[source]

Bases: _abcoll.MutableMapping

Persist ALL keys into single file, via shelve.

__abstractmethods__ = frozenset([])
__delitem__(key)[source]
__getitem__(key)[source]
__init__(shelfid)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

__iter__()[source]
__len__()[source]
__module__ = 'PythonCK.decorators.shelves'
__setitem__(key, val)[source]
__str__() <==> str(x)[source]
clear()[source]

Remove all cache.

Module contents