Serialization
The serialization mechanism currently used in DIRAC is called DEncode. It is a custom serialization mechanism.
The aim in the medium term is to replace it with standard JSON serialization.
We will describe here these two.
DEncode
DEncode (DEncode
) contains two functions:
encode: returns the string representation of the input.
decode: returns the decoded information from the string input, as well as the length of the information decoded.
from DIRAC.Core.Utilities.DEncode import encode, decode
myData = {'a' : [1,2,3], 2 : 'toto' }
# Encode the structure
myEncodedData = encode(myData)
# myEncodedData is the string 'di2es4:totos1:ali1ei2ei3eee'
# Decode the data back
decode(myEncodedData)
# returns a tuple containing the decoded data
# and the length decoded
# ({2: 'toto', 'a': [1, 2, 3]}, 27)
DEncode supports the following type:
boolean
datetime
dict
int
float (CAUTION, see bellow)
list
long
none
string
tuple
unicode
It is a know fact that DEncode is not stable for floats:
from DIRAC.Core.Utilities.DEncode import encode, decode
d = 133143986190.0
import sys
sys.maxint > d
# True
encode(d)
# 'f1.3314398619e+11e'
decode(encode(d))
# (133143986190.00002, 18)
Notice that 133143986190.0 != 133143986190.00002
JEncode
Warning
This serialization is not in use yet
JEncode (JEncode
) is based on JSON, but exposes the same interface as DEncode, that is an encode and a decode functions.
However, because of the nature of JSON (https://tools.ietf.org/html/rfc7159), there are some limitations and changes with respect to DEncode:
all UTF-8 by default. Non default would be other UTF encoding
Tuples are converted to arrays
the keys of dictionaries are always strings. This means that any other type of key will be cast to a string (including numbers !). As a consequence, it is up to the sender/receiver to cast that in whatever type is desired.
JEncode contains a special serializer and deserializer which enhance the default one with:
Support for datetime: the serialization format is hardcoded and corresponds to %Y-%m-%d %H:%M:%S (see https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior). This means that milliseconds are not kept. Note as well that only dates starting after 01-01-1900 are serializable.
Support for custom object serialization inheriting from JSerializable (
JSerializable
). See the Code documentation for more details on the restrinctions and how to use it.