MonitoringDB

Wrapper on top of ElasticDB. It is used to manage the DIRAC monitoring types.

class DIRAC.MonitoringSystem.DB.MonitoringDB.MonitoringDB(name='Monitoring/MonitoringDB', readOnly=False)

Bases: DIRAC.Core.Base.ElasticDB.ElasticDB

Extension of ElasticDB for Monitoring system DB

RESULT_SIZE = 10000
__init__(name='Monitoring/MonitoringDB', readOnly=False)

c’tor

Parameters:
  • self – self reference
  • dbName (str) – name of the database for example: MonitoringDB
  • fullName (str) – The full name of the database for example: ‘Monitoring/MonitoringDB’

:param str indexPrefix it is the indexPrefix used to get all indexes

bulk_index(indexprefix, doc_type, data, mapping=None, period=None)
Parameters:
  • indexPrefix (str) – index name.
  • doc_type (str) – the type of the document
  • data (list) – contains a list of dictionary
  • period (str) – We can specify which kind of indices will be created. Currently only daily and monthly indexes are supported.
Paran dict mapping:
 

the mapping used by elasticsearch

clusterName = ''
createIndex(indexPrefix, mapping, period=None)
Parameters:
  • indexPrefix (str) – it is the index name.
  • mapping (dict) – the configuration of the index.
  • period (str) – We can specify, which kind of index will be created. Currently only daily and monthly indexes are supported.
deleteIndex(indexName)

:param str indexName the name of the index to be deleted…

exists(indexName)

it checks the existance of an index :param str indexName: the name of the index

getDataForAGivenPeriod(typeName, condDict, initialDate='', endDate='')

Retrieves the history of logging entries for the given component during a given given time period

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
  • initialDate (str) – Indicates the start of the time period in the format ‘DD/MM/YYYY hh:mm’
  • endDate (str) – Indicate the end of the time period in the format ‘DD/MM/YYYY hh:mm’
Returns:

Entries from the database for the given component recorded between the initial and the end dates

getDbHost()

It returns the elasticsearch database host

getDbPort()
It returns the database port
Parameters:self – self reference
getDocTypes(indexName)

:param str indexName is the name of the index… :return S_OK or S_ERROR

getIndexName(typeName)
Parameters:typeName (str) – doc_type and type name is equivalent
getIndexPrefix()

It returns the DIRAC setup.

getIndexes()

It returns the available indexes…

getKeyValues(typeName)

Get all values for a given key field in a type

getLastDayData(typeName, condDict)

It returns the last day data for a given monitoring type.

Returns:

for example

{'sort': [{'timestamp': {'order': 'desc'}}],
 'query': {'bool': {'must': [{'match': {'host': 'dzmathe.cern.ch'}},
                             {'match': {'component': 'Bookkeeping_BookkeepingManager'}}]}}}

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
getLimitedData(typeName, condDict, size=10)

Returns a list of records for a given selection.

Parameters:
  • typeName (str) – name of the monitoring type
  • condDict (dict) –

    -> conditions for the query

    • key -> name of the field
    • value -> list of possible values
  • size (int) – Indicates how many entries should be retrieved from the log
Returns:

Up to size entries for the given component from the database

getMapping(monitoringType)

It returns the mapping of a certain monitoring type

Parameters:monitoringType (str) – the monitoring type for example WMSHistory
Returns:an empty dictionary if there is no mapping defenied.
getUniqueValue(indexName, key, orderBy=False)

:param str indexName the name of the index which will be used for the query :param dict orderBy it is a dictionary in case we want to order the result {key:’desc’} or {key:’asc’} It returns a list of unique value for a certain key from the dictionary.

index(indexName, doc_type, body)
Parameters:
  • indexName (str) – the name of the index to be used…
  • doc_type (str) – the type of the document
  • body (dict) – the data which will be indexed
Returns:

the index name in case of success.

pingDB()

Try to connect to the database :return: S_OK(TRUE/FALSE)

put(records, monitoringType)

It is used to insert the data to El.

Parameters:
  • records (list) – it is a list of documents (dictionary)
  • monitoringType (str) – is the type of the monitoring
query(index, query)

Executes a query and returns its result (uses ES DSL language).

Parameters:
  • self – self reference
  • index (basestring) – index name
  • query (dict) – It is the query in ElasticSearch DSL language
registerType(index, mapping, period=None)

It register the type and index, if does not exists

Parameters:
  • index (str) – name of the index
  • mapping (dict) – mapping used to create the index.
  • period (str) – We can specify, which kind of indexes will be created. Currently only daily and monthly indexes are supported.
retrieveAggregatedData(typeName, startTime, endTime, interval, selectFields, condDict, grouping, metainfo)

Get data from the DB using simple aggregations. Note: this method is equivalent to retrieveBucketedData. The different is the dynamic bucketing. We do not perform dynamic bucketing on the raw data.

Parameters:
  • typeName (str) – name of the monitoring type
  • startTime (int) – epoch objects.
  • endtime (int) – epoch objects.
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
retrieveBucketedData(typeName, startTime, endTime, interval, selectFields, condDict, grouping, metainfo)

Get data from the DB

Parameters:
  • typeName (str) – name of the monitoring type
  • startTime (int) – epoch objects.
  • endtime (int) – epoch objects.
  • condDict (dict) –

    conditions for the query

    • key -> name of the field
    • value -> list of possible values
setDbHost(hostName)
It is used to set the cluster host
Parameters:hostname (str) – it is the host name of the elasticsearch
setDbPort(port)

It is used to set the cluster port

param self:self reference
param str port:the port of the elasticsearch.