In this tutorial, we will show you an easy way to configure Filebeat-Logstash SSL/TLS Connection. In order to sent encrypted data from Filebeat to Logstash, you need to enable SSL/TLS mutual communication between them.
Table of Contents
Configuring Filebeat-Logstash SSL/TLS Connection
Before you can proceed, we assume that you already have installed and setup ELK stack as well the Filebeat on the end points from where you are collecting event data from.
Install and Setup ELK Stack
You can follow any of the guides below to install and setup Elastic Stack;
Install ELK Stack on Ubuntu 20.04
Installing ELK Stack on CentOS 8
Deploy a Single Node Elastic Stack Cluster on Docker Containers
Install and Setup Filebeat
Follow the links below to install and setup Filebeat;
Install and Configure Filebeat on CentOS 8
Install Filebeat on Fedora 30/Fedora 29/CentOS 7
Install and Configure Filebeat 7 on Ubuntu 18.04/Debian 9.8
Generate ELK Stack CA and Server Certificates
In this demo, we will be creating TLS certificates using elasticsearch-certutil.
elasticsearch-certutil is an Elastic Stack utility that simplifies the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack.
With elasticsearch-certutil, it is possible to generate the certificates for a specific node or multiple nodes. However, in this demo, since we are just running a single node Elastic Stack with all the components in place, then we will just generate the certificates for just this single node.
To silently generate the node certificates, create an YAML file to define you nodes distinguished names (can be hostname) and the node FQDN in the format shown below;
vim $HOME/instances.ymlinstances:
- name: 'elk'
dns: [ 'elk.kifarunix-demo.com' ]Once that is done, run the command below to generate the ELK Stack TLS Certificates.
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key --pem --in $HOME/instances.yml --out $HOME/elk-cert.zip --days 365The command will create the CA key and certificate, the node key and certificate archived in a $HOME/elk-cert.zip file which is valid for an year.
Listing the contents of the archive file;
unzip -l $HOME/elk-cert.zip
Archive: /root/elk-cert.zip
Length Date Time Name
--------- ---------- ----- ----
0 2020-10-16 17:48 ca/
1200 2020-10-16 17:48 ca/ca.crt
1675 2020-10-16 17:48 ca/ca.key
0 2020-10-16 17:48 elk/
1188 2020-10-16 17:48 elk/elk.crt
1675 2020-10-16 17:48 elk/elk.key
--------- -------
5738 6 files
Read more about the elasticsearch-certutil tool on Elasticsearch reference page.
Extract the certificate files to some directory. In the command below, we extract to my home directory.
unzip -d $HOME $HOME/elk-cert.zipYou should now have these files;
ls $HOME/ca/ -1ca.crt
ca.keyls $HOME/elk -1elk.crt
elk.keyBe sure to keep you private keys as secure as possible.
Convert the Keys to Standard Elastic Beats PKCS#8 Key format
For Beat to connect to Logstash via TLS, you need to convert the generated node key to the PKCS#8 standard required for the Elastic Beat – Logstash communication over TLS;
openssl pkcs8 -in $HOME/elk/elk.key -topk8 -nocrypt -out $HOME/elk/elk.pkcs8.keyConfiguring Filebeat-Logstash connection with SSL/TLS
Next, copy the node certificate, $HOME/elk/elk.crt, and the Beats standard key, to the relevant configuration directory. In this setup, we install the certs/keys on the /etc/logstash directory;
cp $HOME/elk/{elk.pkcs8.key,elk.crt} /etc/logstash/Configure SSL/TLS connection;
vim /etc/logstash/conf.d/test.conf
input {
beats {
port => 5044
ssl => true
ssl_key => '/etc/logstash/elk.pkcs8.key'
ssl_certificate => '/etc/logstash/elk.crt'
}
}
output {
# elasticsearch {
# hosts => ["https://localhost:9200"]
# manage_template => false
# index => "ssh_auth-%{+YYYY.MM}"
# cacert => "/etc/logstash/logstash.ca.crt"
#}
stdout { }
}
Save and exit the configuration file.
Test Logstash Configuration
Before you can run Logstash, it is a good idea to check for any configuration errors;
/usr/share/logstash/bin/logstash --path.settings /etc/logstash -tIf all is well, you should see such lines from the command output;
...
Configuration OK
[2020-10-16T19:03:05,994][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting LogstashYou can now run Logstash in debugging mode just to see if any error arises as per your Logstash configuration file;
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash/...
[INFO ] 2020-10-16 19:07:34.788 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-10-16 19:07:34.899 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2020-10-16 19:07:35.212 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
...If you see the line, Successfully started Logstash API endpoint, then you good to go.
Configure Filebeat for Logstash SSL/TLS communication
Assuming you have already installed Filebeat on a system you want to collect logs from, configure it for Logstash TLS communication as follows;
Copy the CA certificate generated above to the remote remote system.
scp $HOME/ca/ca.crt username@filebeathost:Once you have copied the CA certificate to the remote host running filebeat, proceed to configure Filebeat-Logstash SSL/TLS communication.
Place the copied CA certificate at some relevant directory, e.g /etc/filebeat;
cp $HOME/ca.crt /etc/filebeatNow configure Filebeat to use SSL/TLS by specifying the path to CA cert on the Logstash output config section;
output.logstash:
hosts: ["elk.kifarunix-demo.com:5044"]
ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]See our sample Filebeat configuration file. Ensure that the Logstash hostname matches the FQDN used while creating the certificates.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/auth.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.logstash:
hosts: ["elk.kifarunix-demo.com:5044"]
ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
Save the configuration file.
Validate the Logstash server’s certificate
Before you can run Filebeat, you need to validate the Logstash server’s certificate trust.
curl -v --cacert /etc/filebeat/ca.crt https://elk.kifarunix-demo.com:5044If the trust can be established between the Logstash and the Filebeat, the command should return an empty response from the server.
* Trying 192.168.57.3:5044...
* TCP_NODELAY set
* Connected to elk.kifarunix-demo.com (192.168.57.3) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/filebeat/ca.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=elk
* start date: Oct 17 15:06:00 2020 GMT
* expire date: Oct 15 15:06:00 2030 GMT
* subjectAltName: host "elk.kifarunix-demo.com" matched cert's "elk.kifarunix-demo.com"
* issuer: CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: elk.kifarunix-demo.com:5044
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server
If you used IP address whilst generating the TLS certs, then run verification as follows;
curl -v --cacert /etc/filebeat/ca.crt https://192.168.57.3:5044Testing Filebeat Configuration
Filebeat in debugging mode to check if all is well.
filebeat -eYou should see the Filebeat starts to harvest the log files and connects to Logstash host.
...
2020-10-16T20:05:49.564Z INFO cfgfile/reload.go:224 Loading of config files completed.
2020-10-16T20:05:49.563Z INFO log/harvester.go:299 Harvester started for file: /var/log/auth.log
2020-10-16T20:05:52.543Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-10-16T20:05:53.544Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://elk.kifarunix-demo.com:5044))
2020-10-16T20:05:53.547Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-10-16T20:05:53.549Z INFO [publisher] pipeline/retry.go:223 done
2020-10-16T20:05:53.624Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://elk.kifarunix-demo.com:5044)) established
If you were running Logstash in debugging mode, then you should be able to see the logs being populated to standard output;
...
{
"@timestamp" => 2020-10-16T20:05:52.544Z,
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@version" => "1",
"agent" => {
"version" => "7.9.2",
"name" => "elk.kifarunix-demo.com",
"type" => "filebeat",
"hostname" => "elk.kifarunix-demo.com",
"ephemeral_id" => "1241500c-8f5f-401b-a9f9-1526e8651878",
"id" => "726660dc-4b6b-464f-b19b-62f343792a18"
},
"host" => {
"containerized" => false,
"architecture" => "x86_64",
"mac" => [
[0] "08:00:27:5c:05:2a",
[1] "08:00:27:7f:84:15"
],
"name" => "elk.kifarunix-demo.com",
"hostname" => "elk.kifarunix-demo.com",
"os" => {
"codename" => "focal",
"version" => "20.04.1 LTS (Focal Fossa)",
"name" => "Ubuntu",
"platform" => "ubuntu",
"family" => "debian",
"kernel" => "5.4.0-51-generic"
},
"ip" => [
[0] "10.0.2.15",
[1] "fe80::a00:27ff:fe5c:52a",
[2] "192.168.57.3",
[3] "fe80::a00:27ff:fe7f:8415"
],
"id" => "57e55f802e0648f885bfe16101cb8d55"
},
"log" => {
"offset" => 6926,
"file" => {
"path" => "/var/log/auth.log"
}
},
"ecs" => {
"version" => "1.5.0"
},
"message" => "Oct 16 20:03:50 ubuntu20 sshd[8512]: Failed password for johndoe from 192.168.57.1 port 54196 ssh2"
Now stop both Filebeat and Logstash debugging modes by pressing Ctrl+c.
And start and enable the services to start on boot;
systemctl enable --now logstashsystemctl enable --now filebeatAnd that marks the end of our guide.
Further Reading
Filebeat Reference: Secure communication with Logstash
Related Tutorials
Install Nextcloud with Nginx and SSL/TLS Certificates on CentOS 8
Configure Apache with SSL/TLS Certificates on CentOS 8
