FCConditionParser

Contains the mechanism to evaluate whether to use or not a catalog

class DIRAC.Resources.Catalog.FCConditionParser.FCConditionParser(vo=None, ro_methods=None)

Bases: object

This objects allows to evaluate conditions on whether or not a given operation should be evaluated on a given catalog for a given lfn (be glad so many things are given to you !).

The conditions are expressed as boolean logic, where the basic bloc has the form “pluginName=whateverThatWillBePassedToThePlugin”. The basic blocs will be evaluated by the respective plugins, and the result can be combined using the standard boolean operators:

  • ! for not
  • & for and
  • | for or
  • [ ] for prioritizing the operations

All these characters, as well as the ‘=’ symbol cannot be used in any expression to be evaluated by a plugin.

The rule to evaluate can either be given at calling time, or can be retrieved from the CS depending on the context (see doc of __call__ and __getConditionFromCS)

Example of rules are:

  • Filename=startswith(‘/lhcb’) & Proxy=voms.has(/lhcb/Role->production)
  • [Filename=startswith(‘/lhcb’) & !Filename=find(‘/user/’)] | Proxy=group.in(lhcb_mc, lhcb_data)
class PluginOperand(tokens)

Bases: object

This class is a wrapper for a plugin and it’s condition It is instantiated by pyparsing every time it encounters “plugin=condition”

__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'__module__': 'DIRAC.Resources.Catalog.FCConditionParser', '__repr__': <function __str__>, 'eval': <function eval>, '__dict__': <attribute '__dict__' of 'PluginOperand' objects>, '__str__': <function __str__>, '__weakref__': <attribute '__weakref__' of 'PluginOperand' objects>, '__doc__': ' This class is a wrapper for a plugin\n and it\'s condition\n It is instantiated by pyparsing every time\n it encounters "plugin=condition"\n\n ', '__init__': <function __init__>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(tokens)
Parameters:tokens – [ pluginName, =, conditions ] the pluginName is automatically prepended with ‘Plugin’
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__()

x.__str__() <==> str(x)

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__() <==> str(x)
__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

eval(**kwargs)

Forward the evaluation call to the plugin

Parameters:**kwargs – contains all the information given to the plugin namely the lfns
Returns:True or False
class _BoolAnd(token)

Bases: DIRAC.Resources.Catalog.FCConditionParser._BoolBinOp

Represents the ‘and’ operator

__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'__module__': 'DIRAC.Resources.Catalog.FCConditionParser', 'reprsymbol': '&', '__doc__': " Represents the 'and' operator ", 'evalop': <built-in function all>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(token)
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__()

String representation

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__()

String representation

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

eval(**kwargs)

Perform the evaluation of the boolean logic by applying the operator between the two arguments

Parameters:**kwargs – whatever information is given to plugin (typically lfn)
evalop()

all(iterable) -> bool

Return True if bool(x) is True for all values x in the iterable. If the iterable is empty, return True.

reprsymbol = '&'
class _BoolBinOp(token)

Bases: object

Abstract object to represent a binary operator

Parameters:token – the token matching a binary operator it is a list with only one element which itself is a list [ [ Arg1, Operator, Arg2] ] The arguments themselves can be of any type, but they need to provide an “eval” method that takes **kwargs as input, and return a boolean
__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'__module__': 'DIRAC.Resources.Catalog.FCConditionParser', 'reprsymbol': None, '__str__': <function __str__>, '__doc__': ' Abstract object to represent a binary operator\n\n :param token: the token matching a binary operator\n it is a list with only one element which itself is a list\n [ [ Arg1, Operator, Arg2] ]\n The arguments themselves can be of any type, but they need to\n provide an "eval" method that takes \\*\\*kwargs as input,\n and return a boolean\n\n ', '__repr__': <function __str__>, 'eval': <function eval>, '__dict__': <attribute '__dict__' of '_BoolBinOp' objects>, '__weakref__': <attribute '__weakref__' of '_BoolBinOp' objects>, 'evalop': <function <lambda>>, '__init__': <function __init__>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(token)
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__()

String representation

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__()

String representation

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

eval(**kwargs)

Perform the evaluation of the boolean logic by applying the operator between the two arguments

Parameters:**kwargs – whatever information is given to plugin (typically lfn)
evalop()
reprsymbol = None
class _BoolNot(t)

Bases: object

Represents the “not” unitary operator

Parameters:t – the token matching a unitary operator it is a list with only one element which itself is a list [ [ !, Arg1] ] The argument itself can be of any type, but it needs to provide an “eval” method that takes **kwargs as input, and return a boolean
__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'__module__': 'DIRAC.Resources.Catalog.FCConditionParser', '__repr__': <function __str__>, 'eval': <function eval>, '__dict__': <attribute '__dict__' of '_BoolNot' objects>, '__str__': <function __str__>, '__weakref__': <attribute '__weakref__' of '_BoolNot' objects>, '__doc__': ' Represents the "not" unitary operator\n\n :param t: the token matching a unitary operator\n it is a list with only one element which itself is a list\n [ [ !, Arg1] ]\n The argument itself can be of any type, but it needs to\n provide an "eval" method that takes \\*\\*kwargs as input,\n and return a boolean\n\n ', '__init__': <function __init__>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(t)
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__()

x.__str__() <==> str(x)

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__() <==> str(x)
__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

eval(**kwargs)

Perform the evaluation of the boolean logic by returning the negation of the evaluation of the argument

Parameters:**kwargs – whatever information is given to plugin (typically lfn)
class _BoolOr(token)

Bases: DIRAC.Resources.Catalog.FCConditionParser._BoolBinOp

Represents the ‘or’ operator

__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'__module__': 'DIRAC.Resources.Catalog.FCConditionParser', 'reprsymbol': '|', '__doc__': " Represents the 'or' operator ", 'evalop': <built-in function any>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(token)
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__()

String representation

__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__()

String representation

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

eval(**kwargs)

Perform the evaluation of the boolean logic by applying the operator between the two arguments

Parameters:**kwargs – whatever information is given to plugin (typically lfn)
evalop()

any(iterable) -> bool

Return True if bool(x) is True for any x in the iterable. If the iterable is empty, return False.

reprsymbol = '|'
_FCConditionParser__allowedChars = ' '
_FCConditionParser__boolExpr = <MagicMock name='mock.infixNotation()' id='140167333564048'>
_FCConditionParser__evaluateCondition(conditionString, **kwargs)

Evaluate a condition against attributes, typically lfn. CAUTION: lfns are here given one by one

_FCConditionParser__forbidenChars = ('[', ']', '!', '&', '|', '=')
_FCConditionParser__getConditionFromCS(catalogName, operationName)

Retrieves the appropriate condition from the CS The base path is in Operation/[Setup/Default]/DataManagement/FCConditions/[CatalogName] If there are no condition defined for the method, we check the global READ/WRITE condition. If this does not exist either, we check the global ALL condition. If none is defined, we return None

Parameters:
  • catalogName (str) – the catalog we want to work on
  • operationName (str) – the operation we want to perform The operationName must be in the read or write method from FileCatalog
Returns:

a condition string or None

_FCConditionParser__pluginOperand = <MagicMock name='mock.Word().__add__().__add__()' id='140167332126288'>
__call__(catalogName, operationName, lfns, condition=None, **kwargs)

Makes a boolean evaluation of a condition, for a given catalog, a given operation, and a list of lfns. Extra parameters might be given, and will be forwarded to each plugin. If the ‘condition’ attribute is not specified (general case), it is fetched from the CS (see __getConditionFromCS)

If there are no condition at all, return True for everything. A programming error in the plugins will lead to the evaluation being False

Note

if the CS can’t be contacted, the conditions will be evaluated to None (courtesy of the Operation helper), so everything will be evaluated to True. Ultimately, it does not really matter, since you will not be able to find any catalog beforehand if you can’t contact the CS…

Parameters:
  • catalogName (str) – name of the catalog we want to work on
  • operationName (str) – name of the operation we want to perform The operationName must be in the read or write method from FileCatalog if it should be retrieve from the CS
  • lfns

    list/dict of lfns

    Warning

    LFNs are expected to have been through the normalizing process, so it should not be a string

  • condition – condition string. If not specified, will be fetched from the CS
  • kwargs – extra params forwarded to the plugins
Returns:

S_OK with a ‘Successful’ dict {lfn:True/False} where the value is the evaluation of the condition against the given lfn key. Failed dict is always empty

__class__

alias of __builtin__.type

__delattr__

x.__delattr__(‘name’) <==> del x.name

__dict__ = dict_proxy({'_FCConditionParser__allowedChars': ' ', '__module__': 'DIRAC.Resources.Catalog.FCConditionParser', '_FCConditionParser__pluginOperand': <MagicMock name='mock.Word().__add__().__add__()' id='140167332126288'>, '_BoolNot': <class 'DIRAC.Resources.Catalog.FCConditionParser._BoolNot'>, '_BoolBinOp': <class 'DIRAC.Resources.Catalog.FCConditionParser._BoolBinOp'>, '__dict__': <attribute '__dict__' of 'FCConditionParser' objects>, '_FCConditionParser__evaluateCondition': <function __evaluateCondition>, '_FCConditionParser__forbidenChars': ('[', ']', '!', '&', '|', '='), '_FCConditionParser__getConditionFromCS': <function __getConditionFromCS>, '__call__': <function __call__>, 'PluginOperand': <class 'DIRAC.Resources.Catalog.FCConditionParser.PluginOperand'>, '_BoolAnd': <class 'DIRAC.Resources.Catalog.FCConditionParser._BoolAnd'>, '_BoolOr': <class 'DIRAC.Resources.Catalog.FCConditionParser._BoolOr'>, '__weakref__': <attribute '__weakref__' of 'FCConditionParser' objects>, '__doc__': '\n This objects allows to evaluate conditions on whether or not\n a given operation should be evaluated on a given catalog\n for a given lfn (be glad so many things are given to you !).\n\n The conditions are expressed as boolean logic, where the basic bloc has\n the form "pluginName=whateverThatWillBePassedToThePlugin".\n The basic blocs will be evaluated by the respective plugins, and the result can\n be combined using the standard boolean operators:\n\n * ! for not\n * & for and\n * \\| for or\n * [ ] for prioritizing the operations\n\n All these characters, as well as the \'=\' symbol cannot be used in any expression to be\n evaluated by a plugin.\n\n The rule to evaluate can either be given at calling time, or can be retrieved\n from the CS depending on the context (see doc of __call__ and __getConditionFromCS)\n\n\n Example of rules are:\n\n * Filename=startswith(\'/lhcb\') & Proxy=voms.has(/lhcb/Role->production)\n * [Filename=startswith(\'/lhcb\') & !Filename=find(\'/user/\')] | Proxy=group.in(lhcb_mc, lhcb_data)\n\n ', '__init__': <function __init__>, '_FCConditionParser__boolExpr': <MagicMock name='mock.infixNotation()' id='140167333564048'>})
__format__()

default object formatter

__getattribute__

x.__getattribute__(‘name’) <==> x.name

__hash__
__init__(vo=None, ro_methods=None)
Parameters:vo – name of the VO
__module__ = 'DIRAC.Resources.Catalog.FCConditionParser'
__new__(S, ...) → a new object with type S, a subtype of T
__reduce__()

helper for pickle

__reduce_ex__()

helper for pickle

__repr__
__setattr__

x.__setattr__(‘name’, value) <==> x.name = value

__sizeof__() → int

size of object in memory, in bytes

__str__
__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)