.. _centralized_logging: =========================================================================== Centralized logging for DIRAC components with the ELK stack and MQ services =========================================================================== The `ELK stack `_ is now a de facto standard for centralizing logs. DIRAC allows you to send your Agents and Services logs to a MessageQueue system, from which the ELK stack can be fed. We will not detail the installation of the MQ or the ELK stack here. DIRAC Configuration =================== First of all, you should configure a MQ in your ``Resources`` that will be used for the logging purposes. See :ref:`configuration_message_queues` for details. It would look something like:: Resources { MQServices { messagebroker.cern.ch { MQType = Stomp Port = 61713 User = myUser Password = myPassword Host = messagebroker.cern.ch Queues { lhcb.dirac.logging { } } } } } Next, you should define a logging backend using the ``messageQueue`` plugin. See :ref:`gLogger_backends_messagequeue` and :ref:`gLogger_gLogger_basics_backend` for details. It would look something like:: LogBackends { mqLogs { MsgQueue = messagebroker.cern.ch::Queues::lhcb.dirac.logging Plugin = messageQueue } } Finally, you have to instruct your services or agents to use that log backend. Again, see :ref:`gLogger_gLogger_basics_backend`. For the DFC for example, it would look something like:: FileCatalog { [...] LogBackends = stdout, mqLogs } From the DIRAC point of view, that's all there is to do. Logstash and ELK configurations =============================== The suggested logstash configuration (``/etc/logstash/conf.d/configname``) can be found in https://gitlab.cern.ch/ai/it-puppet-module-dirac/-/blob/qa/code/templates/logstash.conf.erb (check the `full documentation `_) The ElasticSearch template ``lhcb-dirac-logs_default`` looks like:: { "order": 1, "template": "lhcb-dirac-logs-*", "settings": { "index.number_of_shards": "1", "index.search.slowlog.threshold.fetch.warn": "1s", "index.search.slowlog.threshold.query.warn": "10s", "indexing.slowlog.level": "info", "indexing.slowlog.threshold.index.warn": "10s", "indexing.slowlog.threshold.index.info": "5s" }, "mappings": { "_default_": { "dynamic_templates": [ { "message_field": { "path_match": "message", "match_mapping_type": "string", "mapping": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, { "varmessage_field": { "path_match": "varmessage", "match_mapping_type": "*", "mapping": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, { "string_fields": { "match": "*", "match_mapping_type": "string", "mapping": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } ], "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "keyword" }, "geoip": { "dynamic": true, "properties": { "ip": { "type": "ip" }, "location": { "type": "geo_point" }, "latitude": { "type": "half_float" }, "longitude": { "type": "half_float" } } } } } } } Kibana dashboard ================ A dashboard for the logs can be found `here `_ in both json and ndjson format, as ES services are dropping support for .json imports in newer versions.