Enable HTTPS Connection Between Elasticsearch Nodes

In this tutorial, you will learn how to enable HTTPS connection between Elasticsearch nodes. One of the Elastic security features is to enable encryption between Elasticsearch cluster nodes using HTTPS connection.

If you want to learn how to configure and setup Elasticsearch cluster, check out our previous guide by following the link below;

Configure Multi-node Elasticsearch Cluster

See also;

How to Enable Basic Authentication on ELK Stack

Enable Kibana HTTPS Connection

Enabling HTTPS Connection Between Elasticsearch Nodes

Elasticsearch nodes in a cluster communicate via a transport protocol that uses TCP port 9300. Note that the HTTP connection between Elasticsearch and the REST clients such as Kibana, Filebeat communicate via TCP port 9200.

Checking our Cluster nodes;

curl -XGET es-node-01.kifarunix-demo.com:9200/_cat/nodes?v

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role  master name
192.168.59.19           29          84   1    1.90    1.31     1.06 cdhilmrstw *      es-node-03
192.168.59.17           48          86   2    1.49    1.46     1.17 cdhilmrstw -      es-node-01
192.168.59.18           22          84   1    1.61    1.35     1.14 cdhilmrstw -      es-node-02

Generate Elasticsearch TLS/SSL Certificates on One of the Nodes

You need to generate x509 TLS/SSL certificates to enable you encrypt communication between nodes.

You can generate the TLS certs and key using elasticsearch-certutil tool.

Create directory to store the certs files

On ALL the nodes, run the command below to create a directory to store the certificate files.

mkdir /etc/elasticsearch/certs/

Generate CA in PKCS#12 format

The command below generates a CA certificate and private key and store it it under the directory created above under the name elk-cluster-ca.p12, in PKCS12 format.

The command will also prompt you to enter the password for securing the CA file. We use an empty password by just pressing ENTER in this setup.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca \
	--out /etc/elasticsearch/certs/elk-cluster-ca.p12 \
	--days 3650

Generate CA in PEM format

If you want to generate the CA in PEM format, run the command below. The command generates a zip file containing individual files for the CA certificate and private key.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca \
	--pem \
	--out /etc/elasticsearch/certs/elk-cluster-ca.zip \
	--days 3650

Generate the certificates in PKCS#12 format

By default the command generates a single output file which contains:

• a certificate,
• private key
• the CA certificate.

It also prompts for the CA password (if you set the password above, otherwise press enter to use blank password) and the certificate password.

/usr/share/elasticsearch/bin/elasticsearch-certutil cert \
	--ca /etc/elasticsearch/certs/elk-cluster-ca.p12 \
	--out /etc/elasticsearch/certs/elk-cluster-cert.p12 \
	--days 3650

Generate the certificates in PEM format

To generate the certificates in PEM format, unzip the CA cert and key files;

unzip /etc/elasticsearch/certs/elk-cluster-ca.zip -d /etc/elasticsearch/certs/

Next, generate the certificates in PEM format;

/usr/share/elasticsearch/bin/elasticsearch-certutil cert \
	--pem \
	--ca-cert /etc/elasticsearch/certs/ca/ca.crt \
	--ca-key /etc/elasticsearch/certs/ca/ca.key \
	--days 3650 \
	--out /etc/elasticsearch/certs/elk-cluster-cert.zip

Unzip the files;

unzip /etc/elasticsearch/certs/elk-cluster-cert.zip  -d /etc/elasticsearch/certs/

Set the ownership of the certificate files to elasticsearch user.

chown -R elasticsearch: /etc/elasticsearch/certs/

Enable Elasticsearch Security Features on ALL Nodes

The default installation of Elasticsearch uses basic license which comes with security features disabled by default.

To enable Elasticsearch security features, set the value for xpack.security.enabled to true by running the command below in every node in the Elasticsearch cluster.

echo "xpack.security.enabled: true" >> /etc/elasticsearch/elasticsearch.yml

Similarly, you need to enable Elasticsearch node transport SSL/TLS settings;

echo "xpack.security.transport.ssl.enabled: true" >> /etc/elasticsearch/elasticsearch.yml

Enabling HTTPS Connection Between Elasticsearch Nodes

Once the above is done, you need to specify path to the certificate files generated above.

If you used the PKCS#12 format, enter the following lines in elasticsearch.yml file on every node in Elasticsearch.

xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elk-cluster-cert.p12 
xpack.security.transport.ssl.truststore.path: certs/elk-cluster-cert.p12

If you are using the PEM files, then enter the lines below in elasticsearch.yml file on every node in Elasticsearch.

xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

Copy Certificates to Other Elasticsearch Nodes

Before you can start Elasticsearch on other nodes, ensure that you have copied the certificates;

for i in node02 node03; do rsync -avP /etc/elasticsearch/certs root@$i:/etc/elasticsearch/; done

Set the ownership of the certificate files to elasticsearch user.

chown -R elasticsearch: /etc/elasticsearch/certs/

Add Certificate Password to Elasticsearch Keystore

If you secured your certificates with a password, you need to add the password to the Elasticsearch keystore on all the cluster nodes.

If you used the certificates in PKCS#12 format, run the commands below to add the certificate password to the keystore:

/usr/share/elasticsearch/bin/elasticsearch-keystore add \
xpack.security.transport.ssl.truststore.secure_password

/usr/share/elasticsearch/bin/elasticsearch-keystore add \
xpack.security.transport.ssl.keystore.secure_password

If you used the certificates in PEM format, use the command below to add the certificate key to keystore:

/usr/share/elasticsearch/bin/elasticsearch-keystore add \
xpack.security.transport.ssl.secure_key_passphrase

Restart Elasticsearch

Next, restart Elasticsearch service to effect the changes;

systemctl restart elasticsearch

Create Passwords for Built-in Elastic Users

When you enable basic Elastic security features, you need to set the passwords for the built-in Elasticsearch users especially the elastic and kibana_system users to enable you run the REST queries.

Take for example, you want to check the nodes in the cluster by running the command;

curl -XGET es-node-01.kifarunix-demo.com:9200/_cat/nodes?pretty \
	--cacert /etc/elasticsearch/certs/elk-cluster-ca.p12

Sample output;

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

Passwords can be randomly generated or can be manually set using elasticsearch-setup-passwords utility.

You can only run the password creation ONCE on any node in the cluster.

To automatically generate random passwords for built-in Elastic users, run the command;

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

When the command runs, confirm the process.

Sample output;

Changed password for user apm_system
PASSWORD apm_system = eXvv3x7zs9O7qbYgMvLW

Changed password for user kibana_system
PASSWORD kibana_system = mLhDnalWA8yPpPuE0GFB

Changed password for user kibana
PASSWORD kibana = mLhDnalWA8yPpPuE0GFB

Changed password for user logstash_system
PASSWORD logstash_system = p4nAxkl1MGss7VU18TzT

Changed password for user beats_system
PASSWORD beats_system = L1Xm8xnUM7yKAfBAR3Nx

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = uNHAxTVX0LSXwnX54hzr

Changed password for user elastic
PASSWORD elastic = o8u1zQm93o5Py3huqqt1

Save the passwords in a place where you can easily retrieve when needed.

In most cases, we will be using the passwords for elastic and kibana_system users.

• elastic is a built-in superuser in Elastic stack.

• kibana_system is the user Kibana uses to connect and communicate with Elasticsearch.

For example, you can now specify the username in the curl command ran above.

curl -XGET es-node-01.kifarunix-demo.com:9200/_cat/nodes?pretty \
--cacert /etc/elasticsearch/certs/elk-cluster-ca.p12 -u elastic

Enter the password generated above for elastic user.

Sample output;

192.168.59.17 19 75 1 0.23 0.37 0.58 cdhilmrstw - es-node-01
192.168.59.18 53 73 0 0.01 0.26 0.62 cdhilmrstw * es-node-02
192.168.59.19 42 73 0 0.04 0.28 0.62 cdhilmrstw - es-node-03

Configure Security in Kibana

There are some internal requests that Kibana makes to Elasticsearch. Since you have enabled HTTPS Elasticsearch authentication, such requests that Kibana makes, also need to be authenticated. Credentials for the kibana_system used can be used in this case ad thus you need to add them to Kibana.

Credentials for the user can be defined in plain text in Kibana configuration file, kibana.yml or can be added to Kibana keystore.

The credentials were generated above.

To add the credentials in Kibana configuration file, enter the lines in the kibana.yml.

vim /etc/kibana/kibana.yml

elasticsearch.username: "kibana_system"
elasticsearch.password: "mLhDnalWA8yPpPuE0GFB"

To add the credentials to the Keystore instead of putting them on /etc/kibana/kibana.yml, create Kibana keystore and add them as shown below;

chmod g+w /etc/kibana/

sudo -u kibana /usr/share/kibana/bin/kibana-keystore create

Add the user, kibana_system:

sudo -u kibana /usr/share/kibana/bin/kibana-keystore add elasticsearch.username

Add the password:

sudo -u kibana /usr/share/kibana/bin/kibana-keystore add elasticsearch.password

Restart Kibana Service

Next, restart Kibana service;

systemctl restart kibana

You can also check our guide on how to enable Kibana HTTPS connection.

Quick Way to Enable Kibana HTTPS Connection

Authenticating to Kibana Web Interface

Once you enable Security features, you will be required to authenticate with a valid user before you can access Kibana web interface.

In this setup, you can use the Elastic superuser created above, elastic.

Reference

Configuring TLS in Elasticsearch

Other Tutorials

Integrate Wazuh Manager with ELK Stack

Configure ELK Stack Alerting with ElastAlert

Monitor Linux System Metrics with ELK Stack