Wire up Filebeat and Logstash to Amazon Elasticsearch service on Ubuntu

Short Description

Requirements

• OS — Ubuntu > 16.0
• Elastic search setup

Follow these steps:

1. Installations

• Install Filebeat on the source machine.

Download Filebeat from here. Install using

sudo dpkg -i filebeat-oss-6.7.0-amd64.deb

• Install Logstash on the source machine.
• Download Logstash from here.

sudo dpkg -i logstash-oss-6.7.0-amd64.deb

Note: Make sure the versions of Filebeat, Logstash and Elasticsearch are all same. In this post, I have used version 6.7.0.

Reference download links.

Download Elastic Products | Elastic

Install Java or OpenJDK on your machine.

sudo apt-get install openjdk-8-jdk

Note: Logstash requires Java to run. In this example, we’re using Java version 8 (Open JDK 1.8), which is supported by all versions of Logstash. For more information about the supported versions of Java and Logstash, see the Elasticsearch support matrix on the Elasticsearch website.

2. Verify the configuration files by checking the “/etc/filebeat” and “/etc/logstash” directories.

3. Update your Filebeat YAML configuration file to send Apache access logs to Logstash.

/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
      # Path to the log files you want to crawl and fetch
    - /home/ubuntu/logs/*.log 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
# output.elasticsearch:
  # hosts: ["localhost:9200"]
output.logstash:
  hosts: ["localhost:5044"]
setup.ilm.enabled: false 
ilm.enabled: false

Start the Filebeat and Logstash services with the following commands on each instance.

systemctl start filebeat (service filebeat start)

• Logstash

Create a conf file inside /etc/logstash/conf.d

touch /etc/logstash/conf.d/logstash.conf

Make sure that your Logstash configuration file can access Filebeats on Port 5044. This port access allows Logstash to forward requests to your Amazon ES VPC endpoint.

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["https:/your-elastic-search-domain.ap-south-1.es.amazonaws.com/_cat/indices:443"]
    index => "ubuntu-data-pipeline" # Set name of pipeline
    user => "Your Username"
    password => "Your Password"
  }
}

Start logstash service

systemctl start logstash (service logstash start)

Run a cat indices API call to your Amazon ES domain to confirm that the Filebeat logs are being sent. If your logs are successfully sent, you’ll receive the following response:

curl -XGET -u 'username:password' https:/your-elastic-search-domain.ap-south-1.es.amazonaws.com/_cat/indices

If you successfully configure Elasticsearch, Logstash, and Kibana (ELK) with Amazon EC2 Linux, your pipeline looks like this:

Filbeat > Logstash > AWS Elasticsearch/Kibana

• Kibana

Open your Kibana dashboard.

Navigate to Discover from side navigation.

Click on Create new Index pattern

Provide the name of the pattern that we previously defined in logstash configuration file. (ubuntu-data-pipeline in this case.)