Shinken logs ElasticSearch storage

Shinken broker module used to : - store Shinken logs in an ElasticSearch index

Main features

This module intercepts Shinken logs and analyses each log to store it in ElasticSearch. The logs are then available for an application such as Kibana.

Requirements

Use elasticsearch version > 2.0.0 and elasticsearch-curator > 3.4.0

pip install elasticsearch>=2.0 pip install elasticsearch-curator>=3.4

Enabling Elastic logs

To use the elastic-logs module you must declare it in your broker configuration. ``` define broker { ...

  modules        ..., elastic-logs

} ```

The module configuration is defined in the file: elastic-logs.cfg.

Default configuration needs to be tuned up to your ElasticSearch configuration.

```

Module: elastic-logs

Loaded by: Broker

Store the Shinken logs in Elasticsearch

define module { module_name elastic-logs module_type elastic-logs

# ElasticSearch cluster connection
# EXAMPLE
# hosts es1.example.net:9200,es2.example.net:9200,es3.example.net:9200
hosts           localhost:9200

# The prefix of the index where to store the logs.
# There will be one indexe per day: shinken-YYYY.MM.DD
index_prefix    shinken

# The timeout connection to the ElasticSearch Cluster
timeout         20

# Commit period
# Every commit_period seconds, the module stores the received logs in the index
# Default is to commit every 60 seconds
commit_period     60

# Commit volume
# The module commits at most commit_volume logs in the index at every commit period (bulk operation)
# Default is 200 lines
commit_volume     200

# Logs rotation
#
# Remove indices older than the specified value
# Value is specified as :
# 1d: 1 day
# 3m: 3 months ...
# d = days, w = weeks, m = months, y = years
# Default is 1 month
max_logs_age    1m

# Services filtering
# Filter is declared as a comma separated list of items:
# An item can be a regexp which is matched against service description (hostname/service)
#  ^test*, matches all hosts which name starts with test
#  /test*, matches all services which name starts with test
#
# An item containing : is a specific filter (only bi is supported currently)
#  bi:>x, bi:>=x, bi:<x, bi:<=x, bi:=x to match business impact

# default is to consider only the services which business impact is > 4
# 3 is the default value for business impact if property is not explicitely declared
# Default is only bi>4 (most important services)
#services_filter bi:>4

}

```