Deploying ELK to work with IDLive Doc Server¶

This tutorial shows how to install and configure ELK to work with IDLive Doc Server.

Install Docker engine and pull `sebp/elk` image¶

To install ELK, we will use sebp/elk Docker image:

sudo -i
apt update
apt dist-upgrade
apt install docker.io
docker info
docker pull sebp/elk

Increase max virtual memory areas¶

Elasticsearch requires that the vm.max_map_count kernel setting is set to at least 262144:

sudo -i
echo "vm.max_map_count = 262144" > /etc/sysctl.d/20-elk-stack-reqs.conf
sysctl -p

Configure Logstash¶

Create logstash.conf file containing following:

echo 'input {
    tcp {
      port => 5959
      codec => json_lines
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
  }
}' > ./logstash.conf

Run ELK¶

Run sebp/elk container. The following ports must be open in the container:

5601 - port for access to Kibana
5959 - port for access to Logstash
9200 - port for access to Elasticsearch

docker run -d                                                                 \
    -p 5601:5601                                                              \
    -p 5959:5959                                                              \
    -p 9200:9200                                                              \
    -v /path/to/file/logstash.conf:/etc/logstash/conf.d/99-json-input.conf:ro \
    -v /path/to/directory/elk-data:/var/lib/elasticsearch                     \
    --name elk sebp/elk

Wait for the container to start completely. A similar entry should appear in the server logs:

Successfully started Logstash API endpoint {:port=>9600}

Create `logstash-*` index¶

Create create_index.sh script containing following:

echo '#!/usr/bin/env bash
set -euo pipefail
url="http://localhost:5601"
index_pattern="logstash-*"
id="logstash-*"
time_field="@timestamp"
# Create index pattern
# curl -f to fail on error
curl -f -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/saved_objects/index-pattern/$id" \
  -d"{\"attributes\":{\"title\":\"$index_pattern\",\"timeFieldName\":\"$time_field\"}}"
# Make it the default index
curl -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/kibana/settings/defaultIndex" \
  -d"{\"value\":\"$id\"}"' > create_index.sh
sudo chmod +x create_index.sh

Run the script:

./create_index.sh

Make sure that the logstash-* index appears in Kibana. To do this, go to the page http://ip-address:5601/app/management/kibana/indexPatterns

Restart IDLive Doc Server¶

docker run -d --name das               \
    -p 8080:8080                       \
    -e LOGSTASH_HOST=ip-address-of-elk \
    -e LOGSTASH_PORT=5959              \
    ...

Restart the server by specifying the connection to ELK. To do this, you need to pass LOGSTASH_HOST and LOGSTASH_PORT

Now you can make a request to IDLive Doc Server and the logs will be displayed on the Kibana page: http://ip-address:5601/app/discover