MonitoringDB

Wrapper on top of ElasticDB. It is used to manage the DIRAC monitoring types.

Configuration Parameters:

The following options can be set in Systems/Monitoring/<Setup>/Databases/MonitoringDB

  • IndexPrefix: Prefix used to prepend to indices created in the ES instance. If this
    is not present in the CS, the indices are prefixed with the setup name.
class DIRAC.MonitoringSystem.DB.MonitoringDB.MonitoringDB(name='Monitoring/MonitoringDB', readOnly=False)

Bases: DIRAC.Core.Base.ElasticDB.ElasticDB

Extension of ElasticDB for Monitoring system DB

RESULT_SIZE = 10000
__init__(name='Monitoring/MonitoringDB', readOnly=False)

c’tor

Parameters:
  • self – self reference
  • dbName (str) – name of the database for example: MonitoringDB
  • fullName (str) – The full name of the database for example: ‘Monitoring/MonitoringDB’
  • indexPrefix (str) – it is the indexPrefix used to get all indexes
bulk_index(indexprefix, doc_type='_doc', data=None, mapping=None, period=None)
Parameters:
  • indexPrefix (str) – index name.
  • doc_type (str) – the type of the document
  • data (list) – contains a list of dictionary
  • mapping (dict) – the mapping used by elasticsearch
  • period (str) – We can specify which kind of indices will be created. Currently only daily and monthly indexes are supported.
clusterName = ''
createIndex(indexPrefix, mapping, period=None)
Parameters:
  • indexPrefix (str) – it is the index name.
  • mapping (dict) – the configuration of the index.
  • period (str) – We can specify, which kind of index will be created. Currently only daily and monthly indexes are supported.
deleteByQuery(indexName, query)

Delete data by query

Parameters:
  • indexName (str) – the name of the index
  • query (str) – the query that we want to issue the delete on
deleteIndex(indexName)
Parameters:indexName (str) – the name of the index to be deleted…
exists(indexName)

it checks the existance of an index

Parameters:indexName (str) – the name of the index
static generateFullIndexName(indexName, period=None)

Given an index prefix we create the actual index name. Each day an index is created.

Parameters:
  • indexName (str) – it is the name of the index
  • period (str) – We can specify, which kind of indexes will be created. Currently only daily and monthly indexes are supported.
getDataForAGivenPeriod(typeName, condDict, initialDate='', endDate='')

Retrieves the history of logging entries for the given component during a given given time period

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
  • initialDate (str) – Indicates the start of the time period in the format ‘DD/MM/YYYY hh:mm’
  • endDate (str) – Indicate the end of the time period in the format ‘DD/MM/YYYY hh:mm’
Returns:

Entries from the database for the given component recorded between the initial and the end dates

getDbHost()

It returns the elasticsearch database host

getDbPort()
It returns the database port
Parameters:self – self reference
getDocTypes(indexName)
Parameters:indexName (str) – is the name of the index…
Returns:S_OK or S_ERROR
getIndexName(typeName)
Parameters:typeName (str) – doc_type and type name is equivalent
getIndexPrefix()

It returns the DIRAC setup.

getIndexes()

It returns the available indexes…

getKeyValues(typeName)

Get all values for a given key field in a type

getLastDayData(typeName, condDict)

It returns the last day data for a given monitoring type.

Returns:

for example

{'sort': [{'timestamp': {'order': 'desc'}}],
 'query': {'bool': {'must': [{'match': {'host': 'dzmathe.cern.ch'}},
                             {'match': {'component': 'Bookkeeping_BookkeepingManager'}}]}}}

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
getLimitedData(typeName, condDict, size=10)

Returns a list of records for a given selection.

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    -> conditions for the query

    • key -> name of the field
    • value -> list of possible values
  • size (int) – Indicates how many entries should be retrieved from the log
Returns:

Up to size entries for the given component from the database

getMapping(monitoringType)

It returns the mapping of a certain monitoring type

Parameters:monitoringType (str) – the monitoring type for example WMSHistory
Returns:an empty dictionary if there is no mapping defenied.
getUniqueValue(indexName, key, orderBy=False)
Parameters:
  • indexName (str) – the name of the index which will be used for the query
  • orderBy (dict) – it is a dictionary in case we want to order the result {key:’desc’} or {key:’asc’}
Returns:

a list of unique value for a certain key from the dictionary.

index(indexName, doc_type='_doc', body=None, docID=None)
Parameters:
  • indexName (str) – the name of the index to be used
  • doc_type (str) – the type of the document
  • body (dict) – the data which will be indexed (basically the JSON)
  • id (int) – optional document id
Returns:

the index name in case of success.

pingDB()

Try to connect to the database

Returns:S_OK(TRUE/FALSE)
put(records, monitoringType)

It is used to insert the data to El.

Parameters:
  • records (list) – it is a list of documents (dictionary)
  • monitoringType (str) – is the type of the monitoring
query(index, query)

Executes a query and returns its result (uses ES DSL language).

Parameters:
  • self – self reference
  • index (basestring) – index name
  • query (dict) – It is the query in ElasticSearch DSL language
registerType(index, mapping, period=None)

It register the type and index, if does not exists

Parameters:
  • index (str) – name of the index
  • mapping (dict) – mapping used to create the index.
  • period (str) – We can specify, which kind of indexes will be created. Currently only daily and monthly indexes are supported.
retrieveAggregatedData(typeName, startTime, endTime, interval, selectFields, condDict, grouping, metainfo)

Get data from the DB using simple aggregations. Note: this method is equivalent to retrieveBucketedData. The different is the dynamic bucketing. We do not perform dynamic bucketing on the raw data.

Parameters:
  • typeName (str) – name of the monitoring type
  • startTime (int) – epoch objects.
  • endtime (int) – epoch objects.
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
retrieveBucketedData(typeName, startTime, endTime, interval, selectFields, condDict, grouping, metainfo)

Get data from the DB

Parameters:
  • typeName (str) – name of the monitoring type
  • startTime (int) – epoch objects.
  • endtime (int) – epoch objects.
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
setDbHost(hostName)
It is used to set the cluster host
Parameters:hostname (str) – it is the host name of the elasticsearch
setDbPort(port)

It is used to set the cluster port

param self:self reference
param str port:the port of the elasticsearch.
update(index, doctype='_doc', query=None, updateByQuery=True, id=None)

Executes an update of a document, and returns S_OK/S_ERROR

Parameters:
  • self – self reference
  • index (basestring) – index name
  • doctype (basestring) – type of document
  • query (dict) – It is the query in ElasticSearch DSL language
  • updateByQuery (bool) – A bool to determine updation by update by query or index values using index function.
  • id (int) – ID for the document to be created.