Follow through this tutorial to learn how to setup multinode Elasticsearch 8.x cluster. As of this writing, Elastic Stack 8.3 is the current release. This means that Elasticsearch 8.3, one of the major components of the Elastics Stack is also the current release version as of this writing.
Setup Multinode Elasticsearch 8.x Cluster
In our previous tutorial, we learnt how to setup a three node Elasticsearch 7.x cluster.
We will as well be configuring a three node Elasticsearch 8.x cluster in this tutorial.
My Environment:
- Node 1: es-node01.kifarunix-demo.com
- Node 2: es-node02.kifarunix-demo.com
- Node 3: es-node03.kifarunix-demo.com
Ensure that the hostnames are resolvable on each node. If you do not have a DNS server, then you can use your hosts file.
192.168.56.125 es-node01.kifarunix-demo.com es-node01
192.168.56.138 es-node02.kifarunix-demo.com es-node02
192.168.56.139 es-node03.kifarunix-demo.com es-node03
With the release of the Elastic 8.x, there are some few changes in Elasticsearch setup procedure.
So, how can you setup multinode Elasticsearch 8.x cluster?
Install Elasticsearch 8.x on All Cluster Nodes
You need to install the same version of Elasticsearch 8.x on all the cluster nodes.
In this tutorial, we will be using Ubuntu 22.04 system.
Thus, to install Elasticsearch 8.x on Ubuntu, you need to install the Elastic APT repositories as follows;
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
gpg --dearmor > /etc/apt/trusted.gpg.d/elastic.gpg
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" > \
/etc/apt/sources.list.d/elastic-8.x.list
Run system update;
apt update
Once the repos are in place, install Elasticsearch 8.x on all the cluster nodes using the command below;
apt install elasticsearch
During the installation, the Security features will be enabled by default;
- Authentication and authorization are enabled.
- TLS for the transport and HTTP layers is enabled and configured. Self-signed SSL certs are generated and used.
- Elastic super user account (elastic) and its password is created.
- Elasticsearch is configured as a single node cluster.
Sample installation output;
Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: elasticsearch 0 upgraded, 1 newly installed, 0 to remove and 26 not upgraded. Need to get 539 MB of archives. After this operation, 1,139 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.3.0 [539 MB] Fetched 539 MB in 3min 51s (2,333 kB/s) Selecting previously unselected package elasticsearch. (Reading database ... 108617 files and directories currently installed.) Preparing to unpack .../elasticsearch_8.3.0_amd64.deb ... Creating elasticsearch group... OK Creating elasticsearch user... OK Unpacking elasticsearch (8.3.0) ... Setting up elasticsearch (8.3.0) ... --------------------------- Security autoconfiguration information ------------------------------ Authentication and authorization are enabled. TLS for the transport and HTTP layers is enabled and configured. The generated password for the elastic built-in superuser is : GWucOHqyGX1V5JiL6kC* If this node should join an existing cluster, you can reconfigure this with '/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token' after creating an enrollment token on your existing cluster. You can complete the following actions at any time: Reset the password of the elastic built-in superuser with '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'. Generate an enrollment token for Kibana instances with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'. Generate an enrollment token for Elasticsearch nodes with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'. ------------------------------------------------------------------------------------------------- ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service
DO NOT start Elasticsearch service yet.
Configure Elasticsearch 8.x on Node 01
Open the Elasticsearch configuration file for editing on Node 01;
vim /etc/elasticsearch/elasticsearch.yml
Set the Name of the Cluster
Set the name of the cluster;
# ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: kifarunix-demo ...
Set the Name of Each Node;
# ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: es-node01 ...
You can add other custom attributes if you want.
Define the Roles of Elasticsearch Node
As stated above, you can assign each node a respective role as master, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both master
and data
node to make the cluster resilient to the loss of any single node.
... # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: es-node01 node.roles: [ master, data ] ...
Enable Memory Lock
To ensure good performance of Elasticsearch, you need to disable memory swapping is by enabling memory lock. Hence, uncomment the line bootstrap.memory_lock: true. This is one of the many ways of disabling swappiness.
... # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # bootstrap.memory_lock: true ...
Bind Elasticsearch to Non-Loopback Address
Elasticsearch binds to loopback addresses by default. For a node to form a cluster, you need to bind it to non-loopback address. This can be done by setting the IP address of the node as the value of network.host.
Node 01
network.host: 192.168.56.125
Elasticsearch by default uses TCP port 9200 to expose REST APIs. TCP port 9300-9400 is used for node communication
The http.host
is set to 0.0.0.0 by default. Since we have bound ES to an interface IP, which we assume it is static in this case, comment out this line;
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0
Discovery and Cluster Formation settings
There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;
- discovery.seed_hosts
discovery.seed_hosts setting Provides a list of master-eligible nodes in the cluster. Each value has the format host:port
or host
, where port
defaults to the setting transport.profiles.default.port
. This setting was previously known as discovery.zen.ping.unicast.hosts.
Configure this setting on all Nodes.
HOWEVER, due to auto-configuration of Elasticsearch during the installation, we will need to be able to start Elasticsearch service on each node before we can join them to the cluster.
As a result, we will skip the configuration of this setting for now.
- cluster.initial_master_nodes
cluster.initial_master_nodes setting defines the initial set of master-eligible nodes. This is important when starting an Elasticsearch cluster for the very first time. After the cluster has formed, remove this setting from each node’s configuration. The value of this setting MUST match the value of node.name
.
Note that during the installation, Elasticsearch is auto-configured as a single node cluster. For example on Node01.
... # Create a new cluster with the current node only # Additional nodes can still join the cluster later cluster.initial_master_nodes: ["es-node01.kifarunix-demo.com"] ...
Update this line such that the name of the node matches the value of node.name.
For example, in setup, this line should be;
cluster.initial_master_nodes: ["es-node01"]
See Discovery and Cluster Formation settings below.
Elasticsearch Cluster HTTPS Connection
By default, Elasticsearch 8.x is auto-configured with self-signed SSL certitificates for both the Transport (connection between the nodes) and HTTP (HTTP API client connections, such as Kibana, Logstash, and Agents).
Later when you add other nodes to the cluster, all the security auto-configurations will be removed on those nodes being added to a cluster.
For communication between the nodes, you should see such configurations on elasticsearch.yml
.
# Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12
For HTTP API client connections, such as Kibana, Logstash, and Agents;
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
Save changes made and exit the file.
Configure Other Important Systems Settings
Disable Memory Swapping
Enabling memory lock as done above is on the ways of disabling swappiness. You therefore need to ensure that memory locking is enabled on the Elasticsearch service level. This can be done as follows;
[[ -d /etc/systemd/system/elasticsearch.service.d ]] || mkdir /etc/systemd/system/elasticsearch.service.d
echo -e '[Service]\nLimitMEMLOCK=infinity' > \
/etc/systemd/system/elasticsearch.service.d/override.conf
Whenever a systemd service is modified, you need to reload the systemd configurations.
systemctl daemon-reload
One of the recommended ways to disable swapping is to completely disable swap if Elasticsearch is the only service running on the server.
swapoff -a
Edit the /etc/fstab
file and comment out any lines that contain the word swap
;
sed -i.bak '/swap/s/^/#/' /etc/fstab
Otherwise, disable swappiness in the kernel configuration;
echo 'vm.swappiness=1' >> /etc/sysctl.conf
sysctl -p
Set JVM Heap Size
Elasticsearch sets the heap size to 1GB by default. As a rule of thump, set Xmx
to no more than 50% of your physical RAM but not more than 32GB. Any custom JVM settings should be placed under /etc/elasticsearch/jvm.options.d
.
echo -e "-Xms1g\n-Xmx1g" > /etc/elasticsearch/jvm.options.d/jvm.options
Set maximum Open File Descriptor
Set the maximum number of open files for the elasticsearch
user to 65,536. This is already set by default in the, /usr/lib/systemd/system/elasticsearch.service
.
...
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535
...
You also should set the maximum number of processes.
...
# Specifies the maximum number of processes
LimitNPROC=4096
Virtual Memory Settings
Elasticsearch uses a mmapfs
directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the /etc/sysctl.conf and update the value of vm.max_map_count as shown below.
vm.max_map_count=262144
You can simply run the command below to configure virtual memory settings.
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
To apply the changes;
sysctl -p
To that far, below is the configuration file on each node;
grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml
Node 01;
cluster.name: kifarunix-demo node.name: es-node01 node.roles: [ master, data ] path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.56.125 http.port: 9200 xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 cluster.initial_master_nodes: ["es-node01"]
Start and Enable Elasticsearch Service on Node 01
Next, start and enable Elasticsearch service to run on system boot on node 01.
systemctl enable --now elasticsearch
Confirm that Elasticsearch is running;
systemctl status elasticsearch
The password was in the installation output;
● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─override.conf Active: active (running) since Wed 2022-06-29 17:15:44 UTC; 1min 57s ago Docs: https://www.elastic.co Main PID: 2660 (java) Tasks: 83 (limit: 3447) Memory: 877.7M CPU: 33.848s CGroup: /system.slice/elasticsearch.service ├─2660 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.l> ├─2720 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+Alw> └─2744 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Jun 29 17:15:00 es-node01.kifarunix-demo.com systemd[1]: Starting Elasticsearch... Jun 29 17:15:44 es-node01.kifarunix-demo.com systemd[1]: Started Elasticsearch.
Confirm the ports are opened.
ss -altnp | grep -iE '92|93'
LISTEN 0 4096 [::ffff:192.168.56.125]:9200 *:* users:(("java",pid=2720,fd=370)) LISTEN 0 4096 [::ffff:192.168.56.125]:9300 *:* users:(("java",pid=2720,fd=368))
curl -k -u elastic https://es-node01:9200
Enter host password for user 'elastic': { "name" : "es-node01", "cluster_name" : "kifarunix-demo", "cluster_uuid" : "vsBAmUoETA2TfnOgQ4c8dg", "version" : { "number" : "8.3.0", "build_type" : "deb", "build_hash" : "5b8b981647acdf1ba1d88751646b49d1b461b4cc", "build_date" : "2022-06-23T22:48:49.607492124Z", "build_snapshot" : false, "lucene_version" : "9.2.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }
Enroll Other Nodes into Elasticsearch Cluster
At this point, Elasticsearch is running on Node 01 ONLY.
Generate Elasticsearch Cluster Enrollment Token
Next, you need to generate Elasticsearch cluster enrollment token. Do this only on a single node where ES is started.
In this setup, we will generate Elasticsearch cluster enrollment token on ES Node01 ONLY since we have started Elasticsearch service on this node.
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
Sample token;
eyJ2ZXIiOiI4LjMuMCIsImFkciI6WyIxOTIuMTY4LjU2LjEyNTo5MjAwIl0sImZnciI6ImRjZjU3
Y2RlYTVkZmI5ZDRkZWFlODA0MDdmODBlNDU2NzA3ZGE2MWZiMmFiNDUwODViMjhlZjJ
hZGYzMzNhNmMiLCJrZXkiOiJKRmd5c1lFQjR1cUFpSnc2MmdQVDpIRTdTVkg2YlRxYUprb0hCUDBFdUp3In0=
Once you have the token, enroll other nodes.
Enroll Elasticsearch Node 02;
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjMuMCIsImFkciI6WyIxOTIuMTY4LjU2LjEyNTo5MjAwIl0sImZnciI6ImRjZjU3\
Y2RlYTVkZmI5ZDRkZWFlODA0MDdmODBlNDU2NzA3ZGE2MWZiMmFiNDUwODViMjhlZjJh\
ZGYzMzNhNmMiLCJrZXkiOiJKRmd5c1lFQjR1cUFpSnc2MmdQVDpIRTdTVkg2YlRxYUprb0hCUDBFdUp3In0=
This node will be reconfigured to join an existing cluster, using the enrollment token that you provided. This operation will overwrite the existing configuration. Specifically: - Security auto configuration will be removed from elasticsearch.yml - The [certs] config directory will be removed - Security auto configuration related secure settings will be removed from the elasticsearch.keystore Do you want to continue with the reconfiguration process [y/N]y
Similarly, run the same enrollment command on other nodes.
Once the enrollment is done, start the Elasticsearch service on the node.
systemctl enable --now elasticsearch
Discovery and Cluster Formation settings
If you noticed, the enrollment command replaces the Cluster initial nodes setting,
cluster.initial_master_nodes
With discovery.seed_hosts:
setting bearing the address of the first node.
Now, we need to configure the new nodes, Node02 and Node03 in this setup, as we did with Node01.
The only change you need to do on Node 01 is to define the list of the cluster nodes;
vim /etc/elasticsearch/elasticsearch.yml
Basically, you need to define cluster members and a list of initial set of master-eligible nodes.
# --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
Next, comment the initial master configuration set during auto-configuration;
... # Create a new cluster with the current node only # Additional nodes can still join the cluster later #cluster.initial_master_nodes: ["es-node01"]
Sample config on Node 01;
cluster.name: kifarunix-demo node.name: es-node01 node.roles: [ master, data ] path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.56.125 http.port: 9200 discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"] cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12
Restart Elasticsearch Service.
systemctl restart elasticsearch
Configure Node 02 just like how Node 01 has been configured. Below is our sample Node 02 configuration;
cluster.name: kifarunix-demo node.name: es-node02 node.roles: [ master, data ] path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.56.138 http.port: 9200 discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"] cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12
Restart Elasticsearch Service.
systemctl restart elasticsearch
Configure Node 03 as well. Below is our sample Node 03 configuration;
cluster.name: kifarunix-demo node.name: es-node03 node.roles: [ master, data ] path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 192.168.56.139 http.port: 9200 discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"] cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12
Restart Elasticsearch Service.
systemctl restart elasticsearch
Check the Cluster Nodes
curl -k -XGET "https://es-node01:9200/_cat/nodes?v" -u elastic
Enter host password for user 'elastic': ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 192.168.56.139 34 97 2 0.66 0.38 0.22 dm - es-node03 192.168.56.138 41 97 5 0.72 0.47 0.30 dm - es-node02 192.168.56.125 22 97 1 0.37 0.20 0.13 dm * es-node01
And there you go. You now have Elasticsearch 8.x cluster.
Check cluster health status;
curl -k -XGET "https://es-node01:9200/_cat/health?v" -u elastic
Enter host password for user 'elastic': epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1656539772 21:56:12 kifarunix-demo green 3 3 4 2 0 0 0 0 - 100.0%
ENSURE that the cluster transport ports (9300/tcp) are opened on the firewall on each node to allow cluster formation.
Once the cluster is formed as above, remove or comment the following line on all the nodes.
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
You can remove as follows;
sed -i.bak '/cluster.initial_master_nodes/s/^/#/' /etc/elasticsearch/elasticsearch.yml
And that is how you can easily setup multinode Elasticsearch 8.x cluster.
Other Tutorials
Deploy a Single Node Elastic Stack Cluster on Docker Containers
Setup Multi-node Elasticsearch Cluster
I don’t think this instruction is clear, my nodes are not connecting to each other. I only have one node on the _cat/nodes endpoint and it only works on node01 (others says that elastic password is invalid). The enrollment part is not clear, it doesn’t change my configuration in any way.
Sorry for that. At what point is password failing?
Thanks for this guide, my cluster are running.
curl -k -XGET “https://10.249.213.5:9200/_cat/nodes?v” -u elastic
Enter host password for user ‘elastic’:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.249.214.5 15 23 0 0.22 0.35 0.39 dm * es-node-2
10.249.213.6 9 50 0 1.12 1.00 0.82 dm – es-node-3
10.249.213.7 11 50 17 1.12 1.00 0.82 dm – es-node-1
Glad it helped you. Enjoy
I spent a few days failing to spin-up a 3-node cluster until I found found this guide. Thank you VERY much for all the effort required to put it together!!!