ElasticSearchDB

This class a wrapper around elasticsearch-py. It is used to query Elasticsearch instances.

class DIRAC.Core.Utilities.ElasticSearchDB.ElasticSearchDB(host, port, user=None, password=None, indexPrefix='', useSSL=True, useCRT=False, ca_certs=None, client_key=None, client_cert=None)

Bases: object

class ElasticSearchDB
Parameters:

  • url (str) – the url to the database for example: el.cern.ch:9200

  • gDebugFile (str) – is used to save the debug information to a file

  • timeout (int) – the default time out to Elasticsearch

  • RESULT_SIZE (int) – The number of data points which will be returned by the query.

RESULT_SIZE = 10000
__init__(host, port, user=None, password=None, indexPrefix='', useSSL=True, useCRT=False, ca_certs=None, client_key=None, client_cert=None)

c’tor

Parameters:

  • self – self reference

  • host (str) – name of the database for example: MonitoringDB

  • port (str) – The full name of the database for example: ‘Monitoring/MonitoringDB’

  • user (str) – user name to access the db

  • password (str) – if the db is password protected we need to provide a password

  • indexPrefix (str) – it is the indexPrefix used to get all indexes

  • useSSL (bool) – We can disable using secure connection. By default we use secure connection.

  • useCRT (bool) – Use certificates.

  • ca_certs (str) – CA certificates bundle.

  • client_key (str) – Client key.

  • client_cert (str) – Client certificate.

bulk_index(indexPrefix, data=None, mapping=None, period='day', withTimeStamp=True)
Parameters:

  • indexPrefix (str) – index name.

  • data (list) – contains a list of dictionary

  • mapping (dict) – the mapping used by elasticsearch

  • period (str) – Accepts ‘day’ and ‘month’. We can specify which kind of indexes will be created.

  • withTimeStamp (bool) – add timestamp to data, if not there already.

Returns:

S_OK/S_ERROR

clusterName = ''
createIndex(indexPrefix, mapping=None, period='day')
Parameters:

  • indexPrefix (str) – it is the index name.

  • mapping (dict) – the configuration of the index.

  • period (str) – We can specify, which kind of index will be created. Currently only daily and monthly indexes are supported.

deleteByQuery(indexName, query)

Delete data by query (careful!)

Parameters:

  • indexName (str) – the name of the index

  • query (str) – the JSON-formatted query for which we want to issue the delete

deleteDoc(index: str, docID: str)

Deletes a document in an index.

Parameters:

  • index – name of the index

  • docID – document ID

deleteIndex(indexName)
Parameters:

indexName (str) – the name of the index to be deleted…

existingIndex(indexName)

Checks the existance of an index, by its name

Parameters:

indexName (str) – the name of the index

Returns:

S_OK/S_ERROR if the request is successful

existsDoc(index: str, docID: str) → bool

Returns information about whether a document exists in an index.

Parameters:

  • index – name of the index

  • docID – document ID

static generateFullIndexName(indexName, period)

Given an index prefix we create the actual index name.

Parameters:

  • indexName (str) – it is the name of the index

  • period (str) – We can specify which kind of indexes will be created (day, week, month, year, null).

Returns:

string with full index name

getDoc(index: str, docID: str) → dict

Retrieves a document in an index.

Parameters:

  • index – name of the index

  • docID – document ID

getDocTypes(indexName)

Returns mappings, by index.

Parameters:

indexName (str) – is the name of the index…

Returns:

S_OK or S_ERROR

getDocs(indexFunc, docIDs: list[str]) → list[dict]

Efficiently retrieve many documents from an index.

Parameters:

  • index – name of the index

  • docIDs – document IDs

getIndexPrefix()

It returns the DIRAC setup.

getIndexes(indexName=None)

It returns the available indexes…

getUniqueValue(indexName, key, orderBy=False)
Parameters:

  • indexName (str) – the name of the index which will be used for the query

  • orderBy (dict) – it is a dictionary in case we want to order the result {key:’desc’} or {key:’asc’}

Returns:

a list of unique value for a certain key from the dictionary.

index(indexName, body=None, docID=None, op_type='index')
Parameters:

  • indexName (str) – the name of the index to be used

  • body (dict) – the data which will be indexed (basically the JSON)

  • id (int) – optional document id

  • op_type (str) – Explicit operation type. (options: ‘index’ (default) or ‘create’)

Returns:

the index name in case of success.

pingDB()

Try to connect to the database

Returns:

S_OK(TRUE/FALSE)

query(index: str, query)

Executes a query and returns its result (uses ES DSL language).

Parameters:

  • self – self reference

  • index (str) – index name

  • query (dict) – It is the query in ElasticSearch DSL language

update(index: str, query=None, updateByQuery: bool = True, docID: str = None)

Executes an update of a document, and returns S_OK/S_ERROR

Parameters:

  • index – index name

  • query – It is the query in ElasticSearch DSL language

  • updateByQuery – A bool to determine update by query or index values using index function.

  • docID – ID for the document to be created.

updateDoc(index: str, docID: str, body) → dict

Update an existing document with a script or partial document

Parameters:

  • index – name of the index

  • docID – document ID

  • body – The request definition requires either script or partial doc

DIRAC.Core.Utilities.ElasticSearchDB.generateDocs(data, withTimeStamp=True)

Generator for fast bulk indexing, yields docs

Parameters:

  • data (list) – list of dictionaries

  • withTimeStamp (bool) – add the timestamps to the docs

Returns:

doc

DIRAC.Core.Utilities.ElasticSearchDB.ifConnected(method)

Decorator for checking that the connection is established.