Setup Multinode Elasticsearch 8.x Cluster

Follow through this tutorial to learn how to setup multinode Elasticsearch 8.x cluster. As of this writing, Elastic Stack 8.3 is the current release. This means that Elasticsearch 8.3, one of the major components of the Elastics Stack is also the current release version as of this writing.

Setup Multinode Elasticsearch 8.x Cluster

In our previous tutorial, we learnt how to setup a three node Elasticsearch 7.x cluster.

We will as well be configuring a three node Elasticsearch 8.x cluster in this tutorial.

My Environment:

• Node 1: es-node01.kifarunix-demo.com
• Node 2: es-node02.kifarunix-demo.com
• Node 3: es-node03.kifarunix-demo.com

Ensure that the hostnames are resolvable on each node. If you do not have a DNS server, then you can use your hosts file.

192.168.56.125 es-node01.kifarunix-demo.com es-node01
192.168.56.138 es-node02.kifarunix-demo.com es-node02
192.168.56.139 es-node03.kifarunix-demo.com es-node03

With the release of the Elastic 8.x, there are some few changes in Elasticsearch setup procedure.

So, how can you setup multinode Elasticsearch 8.x cluster?

Install Elasticsearch 8.x on All Cluster Nodes

You need to install the same version of Elasticsearch 8.x on all the cluster nodes.

In this tutorial, we will be using Ubuntu 22.04 system.

Thus, to install Elasticsearch 8.x on Ubuntu, you need to install the Elastic APT repositories as follows;

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
gpg --dearmor > /etc/apt/trusted.gpg.d/elastic.gpg

echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" > \
/etc/apt/sources.list.d/elastic-8.x.list

Run system update;

apt update

Once the repos are in place, install Elasticsearch 8.x on all the cluster nodes using the command below;

apt install elasticsearch

During the installation, the Security features will be enabled by default;

• Authentication and authorization are enabled.
• TLS for the transport and HTTP layers is enabled and configured. Self-signed SSL certs are generated and used.
• Elastic super user account (elastic) and its password is created.
• Elasticsearch is configured as a single node cluster.

Sample installation output;

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 26 not upgraded.
Need to get 539 MB of archives.
After this operation, 1,139 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.3.0 [539 MB]
Fetched 539 MB in 3min 51s (2,333 kB/s)                                                                                                                                    
Selecting previously unselected package elasticsearch.
(Reading database ... 108617 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.3.0_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.3.0) ...
Setting up elasticsearch (8.3.0) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : GWucOHqyGX1V5JiL6kC*

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token '
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with 
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with 
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with 
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

DO NOT start Elasticsearch service yet.

Configure Elasticsearch 8.x on Node 01

Open the Elasticsearch configuration file for editing on Node 01;

vim /etc/elasticsearch/elasticsearch.yml

Set the Name of the Cluster

Set the name of the cluster;

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: kifarunix-demo
...

Set the Name of Each Node;

# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es-node01
...

You can add other custom attributes if you want.

Define the Roles of Elasticsearch Node

As stated above, you can assign each node a respective role as master, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both master and data node to make the cluster resilient to the loss of any single node.

...
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es-node01
node.roles: [ master, data ]
...

Enable Memory Lock

To ensure good performance of Elasticsearch, you need to disable memory swapping is by enabling memory lock. Hence, uncomment the line bootstrap.memory_lock: true. This is one of the many ways of disabling swappiness.

...
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
...

Bind Elasticsearch to Non-Loopback Address

Elasticsearch binds to loopback addresses by default. For a node to form a cluster, you need to bind it to non-loopback address. This can be done by setting the IP address of the node as the value of network.host.

Node 01

network.host: 192.168.56.125

Elasticsearch by default uses TCP port 9200 to expose REST APIs. TCP port 9300-9400 is used for node communication

The http.host is set to 0.0.0.0 by default. Since we have bound ES to an interface IP, which we assume it is static in this case, comment out this line;

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0

Discovery and Cluster Formation settings

There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;

• discovery.seed_hosts

discovery.seed_hosts setting Provides a list of master-eligible nodes in the cluster. Each value has the format host:port or host, where port defaults to the setting transport.profiles.default.port. This setting was previously known as discovery.zen.ping.unicast.hosts.

Configure this setting on all Nodes.

HOWEVER, due to auto-configuration of Elasticsearch during the installation, we will need to be able to start Elasticsearch service on each node before we can join them to the cluster.

As a result, we will skip the configuration of this setting for now.

• cluster.initial_master_nodes

cluster.initial_master_nodes setting defines the initial set of master-eligible nodes. This is important when starting an Elasticsearch cluster for the very first time. After the cluster has formed, remove this setting from each node’s configuration. The value of this setting MUST match the value of node.name.

Note that during the installation, Elasticsearch is auto-configured as a single node cluster. For example on Node01.

...
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["es-node01.kifarunix-demo.com"]
...

Update this line such that the name of the node matches the value of node.name.

For example, in setup, this line should be;

cluster.initial_master_nodes: ["es-node01"]

See Discovery and Cluster Formation settings below.

Elasticsearch Cluster HTTPS Connection

By default, Elasticsearch 8.x is auto-configured with self-signed SSL certitificates for both the Transport (connection between the nodes) and HTTP (HTTP API client connections, such as Kibana, Logstash, and Agents).

Later when you add other nodes to the cluster, all the security auto-configurations will be removed on those nodes being added to a cluster.

For communication between the nodes, you should see such configurations on elasticsearch.yml.

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

For HTTP API client connections, such as Kibana, Logstash, and Agents;

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

Save changes made and exit the file.

Configure Other Important Systems Settings

Disable Memory Swapping

Enabling memory lock as done above is on the ways of disabling swappiness. You therefore need to ensure that memory locking is enabled on the Elasticsearch service level. This can be done as follows;

[[ -d /etc/systemd/system/elasticsearch.service.d ]] || mkdir /etc/systemd/system/elasticsearch.service.d

echo -e '[Service]\nLimitMEMLOCK=infinity' > \
/etc/systemd/system/elasticsearch.service.d/override.conf

Whenever a systemd service is modified, you need to reload the systemd configurations.

systemctl daemon-reload

One of the recommended ways to disable swapping is to completely disable swap if Elasticsearch is the only service running on the server.

swapoff -a

Edit the /etc/fstab file and comment out any lines that contain the word swap;

sed -i.bak '/swap/s/^/#/' /etc/fstab

Otherwise, disable swappiness in the kernel configuration;

echo 'vm.swappiness=1' >> /etc/sysctl.conf

sysctl -p

Set JVM Heap Size

Elasticsearch sets the heap size to 1GB by default. As a rule of thump, set Xmx to no more than 50% of your physical RAM but not more than 32GB. Any custom JVM settings should be placed under /etc/elasticsearch/jvm.options.d.

echo -e "-Xms1g\n-Xmx1g" > /etc/elasticsearch/jvm.options.d/jvm.options

Set maximum Open File Descriptor

Set the maximum number of open files for the elasticsearch user to 65,536. This is already set by default in the, /usr/lib/systemd/system/elasticsearch.service.

...
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535
...

You also should set the maximum number of processes.

...
# Specifies the maximum number of processes
LimitNPROC=4096

Virtual Memory Settings

Elasticsearch uses a mmapfs directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the /etc/sysctl.conf and update the value of vm.max_map_count as shown below.

vm.max_map_count=262144

You can simply run the command below to configure virtual memory settings.

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

To apply the changes;

sysctl -p

To that far, below is the configuration file on each node;

grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml

Node 01;

cluster.name: kifarunix-demo
node.name: es-node01
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.125
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["es-node01"]

Start and Enable Elasticsearch Service on Node 01

Next, start and enable Elasticsearch service to run on system boot on node 01.

systemctl enable --now elasticsearch

Confirm that Elasticsearch is running;

systemctl status elasticsearch

The password was in the installation output;

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─override.conf
     Active: active (running) since Wed 2022-06-29 17:15:44 UTC; 1min 57s ago
       Docs: https://www.elastic.co
   Main PID: 2660 (java)
      Tasks: 83 (limit: 3447)
     Memory: 877.7M
        CPU: 33.848s
     CGroup: /system.slice/elasticsearch.service
             ├─2660 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.l>
             ├─2720 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+Alw>
             └─2744 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jun 29 17:15:00 es-node01.kifarunix-demo.com systemd[1]: Starting Elasticsearch...
Jun 29 17:15:44 es-node01.kifarunix-demo.com systemd[1]: Started Elasticsearch.

Confirm the ports are opened.

ss -altnp | grep -iE '92|93'

LISTEN 0      4096   [::ffff:192.168.56.125]:9200            *:*    users:(("java",pid=2720,fd=370))         
LISTEN 0      4096   [::ffff:192.168.56.125]:9300            *:*    users:(("java",pid=2720,fd=368))

curl -k -u elastic https://es-node01:9200

Enter host password for user 'elastic':
{
  "name" : "es-node01",
  "cluster_name" : "kifarunix-demo",
  "cluster_uuid" : "vsBAmUoETA2TfnOgQ4c8dg",
  "version" : {
    "number" : "8.3.0",
    "build_type" : "deb",
    "build_hash" : "5b8b981647acdf1ba1d88751646b49d1b461b4cc",
    "build_date" : "2022-06-23T22:48:49.607492124Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Enroll Other Nodes into Elasticsearch Cluster

At this point, Elasticsearch is running on Node 01 ONLY.

Generate Elasticsearch Cluster Enrollment Token

Next, you need to generate Elasticsearch cluster enrollment token. Do this only on a single node where ES is started.

In this setup, we will generate Elasticsearch cluster enrollment token on ES Node01 ONLY since we have started Elasticsearch service on this node.

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

Sample token;

eyJ2ZXIiOiI4LjMuMCIsImFkciI6WyIxOTIuMTY4LjU2LjEyNTo5MjAwIl0sImZnciI6ImRjZjU3
Y2RlYTVkZmI5ZDRkZWFlODA0MDdmODBlNDU2NzA3ZGE2MWZiMmFiNDUwODViMjhlZjJ
hZGYzMzNhNmMiLCJrZXkiOiJKRmd5c1lFQjR1cUFpSnc2MmdQVDpIRTdTVkg2YlRxYUprb0hCUDBFdUp3In0=

Once you have the token, enroll other nodes.

Enroll Elasticsearch Node 02;

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjMuMCIsImFkciI6WyIxOTIuMTY4LjU2LjEyNTo5MjAwIl0sImZnciI6ImRjZjU3\
Y2RlYTVkZmI5ZDRkZWFlODA0MDdmODBlNDU2NzA3ZGE2MWZiMmFiNDUwODViMjhlZjJh\
ZGYzMzNhNmMiLCJrZXkiOiJKRmd5c1lFQjR1cUFpSnc2MmdQVDpIRTdTVkg2YlRxYUprb0hCUDBFdUp3In0=

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically: 
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

Similarly, run the same enrollment command on other nodes.

Once the enrollment is done, start the Elasticsearch service on the node.

systemctl enable --now elasticsearch

Discovery and Cluster Formation settings

If you noticed, the enrollment command replaces the Cluster initial nodes setting,

cluster.initial_master_nodes

With discovery.seed_hosts: setting bearing the address of the first node.

Now, we need to configure the new nodes, Node02 and Node03 in this setup, as we did with Node01.

The only change you need to do on Node 01 is to define the list of the cluster nodes;

vim /etc/elasticsearch/elasticsearch.yml

Basically, you need to define cluster members and a list of initial set of master-eligible nodes.

# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

Next, comment the initial master configuration set during auto-configuration;

...
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["es-node01"]

Sample config on Node 01;

cluster.name: kifarunix-demo
node.name: es-node01
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.125
http.port: 9200
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Restart Elasticsearch Service.

systemctl restart elasticsearch

Configure Node 02 just like how Node 01 has been configured. Below is our sample Node 02 configuration;

cluster.name: kifarunix-demo
node.name: es-node02
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.138
http.port: 9200
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Restart Elasticsearch Service.

systemctl restart elasticsearch

Configure Node 03 as well. Below is our sample Node 03 configuration;

cluster.name: kifarunix-demo
node.name: es-node03
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.139
http.port: 9200
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Restart Elasticsearch Service.

systemctl restart elasticsearch

Check the Cluster Nodes

curl -k -XGET "https://es-node01:9200/_cat/nodes?v" -u elastic

Enter host password for user 'elastic':
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.139           34          97   2    0.66    0.38     0.22 dm        -      es-node03
192.168.56.138           41          97   5    0.72    0.47     0.30 dm        -      es-node02
192.168.56.125           22          97   1    0.37    0.20     0.13 dm        *      es-node01

And there you go. You now have Elasticsearch 8.x cluster.

Check cluster health status;

curl -k -XGET "https://es-node01:9200/_cat/health?v" -u elastic

Enter host password for user 'elastic':
epoch      timestamp cluster        status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1656539772 21:56:12  kifarunix-demo green           3         3      4   2    0    0        0             0                  -                100.0%

ENSURE that the cluster transport ports (9300/tcp) are opened on the firewall on each node to allow cluster formation.

Once the cluster is formed as above, remove or comment the following line on all the nodes.

cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

You can remove as follows;

sed -i.bak '/cluster.initial_master_nodes/s/^/#/' /etc/elasticsearch/elasticsearch.yml

And that is how you can easily setup multinode Elasticsearch 8.x cluster.

Other Tutorials

Deploy a Single Node Elastic Stack Cluster on Docker Containers

Setup Multi-node Elasticsearch Cluster