Light Log Monitoring and Alerting (Filebeat, ElasticSearch, ElastAlert)

2018, Jul 28

This method is a really light way to monitor logs in real time. Tools used:

  • Filebeat
  • ElasticSearch
  • ElastAlert

Logstash is not required for this setup.

Filebeat watch logs for some patterns and index them in elasticsearch. After that, ElastAlert is sending queries to ElasticSearch to periodically check for new entries.

Filebeat Setup:

Filebeat use a yml configuration in /etc/filebeat/filebeat.yml and logs in /var/log/filebeat/filebeat. It needs to access ElasticSearch on HTTP (there are few other methods to access ES if you don't want to access directly). Example of filebeat.yml:

filebeat:
  prospectors:
  - type: log
    paths:
      - /path/to/logs/*.log
    multiline.pattern: '^.*ERROR.*$'
    multiline.negate: true
    multiline.match: after

    include_lines: ['at java']
    close_inactive: 15m
    ignore_older: 1h
    scan_frequency: 10m
#    tail_files: true

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "log-console-%{+yyyy.MM.dd}"

setup.template:
  name: 'log-console'
  pattern: 'log-console-*'
  enabled: false

This example use ElasticSearch for the output. You can set a file as the output, for testing purposes. Or, if you want to integrate Logstash here, you can set up the appropriate output.

Filebeat documentation: here

ElastAlert Setup:

Example of config.yaml:

rules_folder: /opt/elastalert/rules/

run_every:
   minutes: 1

buffer_time:
  hours: 10

es_host: localhost
es_port: 9200

writeback_index: elastalert

alert_time_limit:
  days: 2

Creating rules in rules/event_rule.yaml

es_host: localhost
es_port: 9200

name: action rule
type: any
index: log-console-*

filter:
 - query:
      query_string:
        query: "java at"

realert:
  minutes: 0

alert:
  - "email"
email: "my.email@gmail.com"

name: "Log monitoring"

alert_text: |
    Something happened on {0}
    {1}
alert_text_args: ["host.name", "message"]
alert_text_type: alert_text_only

alert_subject: "[Log Monitoring] Issue occurred on {0}"
alert_subject_args: ["host.name"]

ElastAlert documentation: here

Start ElastAlert with this:

elastalert --config /opt/elastalert/config.yaml --rule /opt/elastalert/rules/event_rule.yaml --verbose

If you want to set a start date, use --start argument.

You can find updates on these configurations here.

Clear old indices in ElasticSearch

curl -s -XGET 'localhost:9200/_cat/indices?v' | grep filebeat | awk '{print $3}' | sort | head -n -14 | xargs --no-run-if-empty -n 1 -I % /usr/bin/curl -XDELETE 'localhost:9200/%'

It clears filebeat* indices older than 14 days (if there is an index per day).