Setup Multinode Elasticsearch 8.x Cluster

Follow through this tutorial to learn how to setup multinode Elasticsearch 8.x cluster. As of this writing, Elastic Stack 8.3 is the current release. This means that Elasticsearch 8.3, one of the major components of the Elastics Stack is also the current release version as of this writing.

Setup Multinode Elasticsearch 8.x Cluster

In our previous tutorial, we learnt how to setup a three node Elasticsearch 7.x cluster.

We will as well be configuring a three node Elasticsearch 8.x cluster in this tutorial.

My Environment:

• Node 1: es-node01.kifarunix-demo.com
• Node 2: es-node02.kifarunix-demo.com
• Node 3: es-node03.kifarunix-demo.com

Ensure that the hostnames are resolvable on each node. If you do not have a DNS server, then you can use your hosts file.

192.168.122.12 es-node01.kifarunix-demo.com es-node01
192.168.122.73 es-node02.kifarunix-demo.com es-node02
192.168.122.50 es-node03.kifarunix-demo.com es-node03

With the release of the Elastic 8.x, there are some few changes in Elasticsearch setup procedure.

Install Elasticsearch 8.x on All Cluster Nodes

You need to install the same version of Elasticsearch 8.x on all the cluster nodes.

In this tutorial, we will be using Ubuntu 22.04 system.

Thus, to install Elasticsearch 8.x on Ubuntu, you need to install the Elastic APT repositories as follows;

apt install apt-transport-https \
	ca-certificates \
	curl \
	gnupg2 \
	software-properties-common

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
	gpg --dearmor -o /etc/apt/trusted.gpg.d/elastic.gpg

echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" > \
/etc/apt/sources.list.d/elastic-8.x.list

Run system update;

apt update

Once the repos are in place, install Elasticsearch 8.x on all the cluster nodes using the command below;

apt install elasticsearch

During the installation, the Security features will be enabled by default;

• Authentication and authorization are enabled.
• TLS for the transport and HTTP layers is enabled and configured. Self-signed SSL certs are generated and used.
• Elastic super user account (elastic) and its password is created.
• Elasticsearch is configured as a single node cluster.

Sample installation output;

...
Setting up elasticsearch (8.11.1) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : 2BEkVDLjbN77ODLsTGmA

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token '
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with 
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with 
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with 
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

DO NOT start Elasticsearch service yet!

Configure Elasticsearch 8.x

Open the Elasticsearch configuration file for editing;

vim /etc/elasticsearch/elasticsearch.yml

Set the Name of the Cluster on All Cluster Nodes

Optionally set the name of the cluster on each Node;

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: kifarunix-demo
...

Set the Node Name on Each of the cluster Nodes;

# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es-node01
...

You can add other custom attributes if you want.

Define the Roles of Elasticsearch Node

As stated above, you can assign each node a respective role as master, data node, ingest node, coordinating node.. In this setup, we will configure all the three nodes to act as both master and data node to make the cluster resilient to the loss of any single node.

...
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es-node01
node.roles: [ master, data ]
...

Enable Memory Lock

To ensure good performance of Elasticsearch, you need to disable memory swapping is by enabling memory lock. Hence, uncomment the line bootstrap.memory_lock: true. This is one of the many ways of disabling swappiness.

...
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
...

Bind Elasticsearch to Non-Loopback Address

Elasticsearch binds to loopback addresses by default. For a node to form a cluster, you need to bind it to non-loopback address. This can be done by setting the IP address of the node as the value of network.host.

Node 01

network.host: 192.168.56.125

Update the rest of the nodes, accordingly.

Elasticsearch by default uses TCP port 9200 to expose REST APIs. TCP port 9300-9400 is used for node communication;

You can comment out the port definition;

http.port: 9200

The http.host is set to 0.0.0.0 by default. Since we have bound ES to an interface IP, which MUST be static in this case, comment out this line;

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0

Discovery and Cluster Formation settings

There are two important discovery and cluster formation settings that should be configured before going to production so that nodes in the cluster can discover each other and elect a master node;

• discovery.seed_hosts

discovery.seed_hosts setting provides a list of master-eligible nodes in the cluster. Each value has the format host:port or host, where port defaults to the setting transport.profiles.default.port. This setting was previously known as discovery.zen.ping.unicast.hosts.

Configure this setting on all Nodes.

HOWEVER, due to auto-configuration of Elasticsearch during the installation, we will need to be able to start Elasticsearch service on each node before we can join them to the cluster.

As a result, we will skip the configuration of this setting for now.

• cluster.initial_master_nodes

cluster.initial_master_nodes setting defines the initial set of master-eligible nodes. This is important when starting an Elasticsearch cluster for the very first time. After the cluster has formed, remove this setting from each node’s configuration. The value of this setting MUST match the value of node.name.

Note that during the installation, Elasticsearch is auto-configured as a single node cluster. For example on Node01.

...
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["es-node01.kifarunix-demo.com"]
...

Update this line such that the name of the node matches the value of node.name.

For example, in setup, this line should be;

cluster.initial_master_nodes: ["es-node01"]

See Discovery and Cluster Formation settings below.

Configure Elasticsearch Cluster HTTPS Connection

By default, Elasticsearch 8.x is auto-configured with self-signed SSL certitificates for both the Transport (connection between the nodes) and HTTP (HTTP API client connections, such as Kibana, Logstash, and Agents).

Later when you add other nodes to the cluster, all the security auto-configurations will be removed on those nodes being added to a cluster and certificates from first node copied over to the node being enrolled.

For communication between the nodes, you should see such configurations on elasticsearch.yml.

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

For HTTP API client connections, such as Kibana, Logstash, and Agents;

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

Save changes made and exit the file.

Kindly note that the CA certificate is generated and stored as /etc/elasticsearch/certs/http_ca.crt.

The CA key for the CA certificate, is stored in the /etc/elasticsearch/certs/http.p12 file. This file is password protected and the password used to protect it is found in the Elasticsearch Keystore.

Thus, to get the CA key, you first need to retrieve the password that was used to protect it by executing the command below;

/usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password

The command will print the Keystore password to standard output.

Once you have the Keystore password, extract the CA key;

openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -nocerts	-nodes

You will be prompted to enter the password. Use the one retrieved above.

The command will print two keys each with a friendlyName, e.g, friendlyName: http_ca and friendlyName: http.

Bag Attributes
    friendlyName: http_ca
    localKeyID: 54 69 6D 65 20 31 37 30 30 37 35 38 35 31 39 32 33 39 
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQC4y7ivLZ2UJJqp
9xKj2q5yWO6RFSXoJo92fNtaVdfu4QULNLSn540Z4nGE+pjkP1u15/H5mFzQLQQ0
rNvoZUxGgzmOLmo8eXsY3iyh5Q3suVNfYY3v+zTmFbD6A7f87MKLLSU6ee2ojdFl
OKT6j00DgmqaPYuqd6CYY7FP2/3asAiSZC9o48RplvkSRBx4Hda7SpOAzlPDGgcm
vnWQqf0Nd+TH7Y8M2Cn+//95YVuNOL03kaXeoZixjPK3zWpYCc6nVRQkR5e9ihbf
WWCBPtJq6nKRkZXWCpE8VsEwI0Pd/0z5aqqXgM9r7/YaOABZksYpX0HR49y8SkU2
C28BMN8X3229Zy2tkdYbGUOaGMuzhb/ce+SsUZDN43gYNiEcgIxNe11Xp+R0dbwc
LkolkoYNddSqi75Vem2PsdSLTR4z1xn7BWf7BxP+PY2/jcvVD4NDNAMDWwKAIW+P
qBrsJizYFLWBz0p280XI6Vm+4KMfvF9wI6wCA4UdLJBvK6Syh3CaJ/27ufiFLgQ3
xlxjY8dmWxAp8rW3pPF00wwcYF6MBjC8uGijga1fgvqjlz9qJqk0/CK2OGXRQdyK
NBh+YmYJccrShxh3wPE42S1NTsHNhiaKub6TbBn6ANKAV8luq425buH/qZhyNBy+
AP5ewwCF41H0U8eKOycbmmodYhX2QwIDAQABAoICABbv/TCL8kSdITAk9toCdCdw
BSIFBKYtgx6AnkmMNjf0aPKHIZVlwbc2IEO9Rz+tjZilUv0EKHZpNV3EcaywRijK
c65uRx7ShVuvgs1lkmDmcAQle79kkvWsOOy6NsWHgxj7YDpDQCNCgdHql1pyccC/
GLRPHLLqvl2r93+jctMI3pzXyqVeVYGLDRzIpNqpTtSt4Q6FOGhoe/1seUic5kgc
WvgQnfJI3ThFq6meqAUoxx51Tv1fWYX6/3WeYwDNLCrLqKsIJ+bi6QvixX5wFHYo
5lJN8SdBkIgeIe9JBqIw5Ou14w7Ycjvs68M9XZhrW3rEfssABDoVje8IuZ065Qs/
XFWFhPKQwkxyhkgcYnthbydI+BRWXSNPm6cfzag08brao9Ozzm6R9+Zj9J4SoH01
kWaECWbEFk+SZfjxDlUozpSIBmFcRA9crHqGKXqnUrBHNWgd+aqlgt/kxf0Nd2Qi
NiLyXO9YMZCS7h7mkXTadHPt+rD+PZubXGYEGQP5LgwQqJ7knxhP/S0wpdqblOXl
I79MT6vdKCWEKUyh79sHfDgbTa8VnzALLUme/PzL8/zR+TUMTuY7fxYWSGNSoXj5
brinxHpyMogeSXD9h/49Fk8ENjPhIVneQj16q/JR8+3yB1lcZsTaB5sqcRQCUkvg
nyqUtSMCsukLfqL0grghAoIBAQDsHobEPD0O5OaiD/q+0wrDZLdMR7iFygMsjPVE
zvFcwtytnmDDFuZGoTLK6n3XuamTIDDLAiGQDqA7OgADNmFGS/Sxq3Br1wlEJ6uK
EklCGq6+6CVZCEwf10JfVYa8MBTVIBZoR5N6E1hymvtXAV/XoT09ubXauHxZDaXB
VbjyUdZ3LsQVPB0AaKN3MTd86nIWl14CbaIPg+0QeZ4M4p1GJ3P+vHaRil4jFQJI
cPINm1JDYuSvj876UwxvUycX9uR01Ybp9mmMAgVgZWqTOY1DeM+iu0xWjyTjOqI7
TByTm0FRtwHdVo9j0sn6PDCw0LXPJh+5C0kBDIGO2apX+XpVAoIBAQDIWu8+37rF
AKZ86ALMJdDgPGhcUiNNWHemy9+9Q/xQaec+/xcMArpwhwg7y1hyzDo89qT3/+Fq
ptL960vqzee9oGnmocTj06Xwl5Mc/ByKUEwkXLllU0PvYNP87nW2Nx8B1J4IZPrk
y/KxcmFe56x3+42/7o2sDaaLVsTeMQNVBm4h3rdycmbHw925cToKCpo9iecVyNrl
p9rfsCJqQNJymeRpxASNVSM0QiCmFfnvfSPdx0a5WAGIXcLAE8xZWNWB+2XYOC7P
RpLSZqT1EiPRvwPlSxxjt7g3mvBglpdvqElZt8mbaVTLcvPrhMG/lAdf5dLjVewi
9xyCEfzdM/Y3AoIBACQ9rzuxb0G4+nlHcJoXdGB3NfccTwMh9YjB/edYyA0Pz6VK
WNm4yxFuLoico0IYOiHd3/9YzWP6CrfseBIGJ7oNcHpQsQmsULzIRfkEQ4BE7itZ
IFpg/qYoeQR+8RYzXw8zRJ0u10D2dS3qAcfh6x8CnY660WHHzTYx9Q4OpTt0OjHC
2M/VkTH93ZEu2bBvn6E+DGzc/PHasulDcXHS2obpCVHBBYXjMf26fMY3p79U5rR3
Glij/wXG93Ki1a5E893G5FVGv/6AVOjtyATe4YLIqT3194qVn/jMiBgH3578RnNP
XpblcU7GDMA+us54yl8IH/hb9hGWJNCTMiUcop0CggEAFl9BE847tr/J12y40z7t
wfw80wJA+uQECFX/l8y9oeURc1Fcq3SqiOIrIzIjoD35ytDnvuGNtDEIol3hzkhL
tjrxWFV7/MynXOQwAy6LqU4qEm1gLlZL4bD4OugNsNka10N8mgdclKvJX9Mb2FMa
SROH0oS6wRmNHUYFGaJzcs0TAykxZtArdecDHS/tgS2J55E8gow5FSfXIt8yGJdA
aJeSj9TZhUaadb1kk+ckT3+zv5H+7bdMBIQjuPr8+IUY4jjGOIydfXut7VyU0uDB
qywfLT1j4CalpV/hs4ddRZSPDDufgYETfNxLLGNYlyHaaPTqxYPUqFrbjhKgH7sr
pQKCAQAxW3yYgmPjwpyglWtiSwVLs9mTDh4ahRl+FSUwk8ZSwRD1k9APEpoPmEbP
qyalnv61aQTQN7cbEzgGm0D0t93sRWtEMU7Efq/kU/7acvc/dNJ8p4hRZXxUNNaN
reZ42/IssirFJTpTFauxSPtYb/7R6gIDAxf3J4+lGInIjWwZtvTb406rOKztvMqj
D8B5Cf9vGrO/CAjnY87BJNRFBrehhnLNFeh1pbdEMAORibfMxtn7k9EGYRdXSdrN
Xv+Zfn4rcd4zFjtMZ8fjOZXXhansJrmBAwAX9SmFtliD96OaNhEV1+3HLScDoR/K
0FZM/3K5DrR8Ed3vWKqAtEOgi5AK
-----END PRIVATE KEY-----
Bag Attributes
    friendlyName: http
    localKeyID: 54 69 6D 65 20 31 37 30 30 37 35 38 35 31 39 32 36 32 
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQC8kJYcWvgzcjRd
qzMagpo3Op94hNDJ2AX2gKP3V5B1kX4tlbjZxWwGLknfBA/Sz5fTkle8z/P0dVCf
SXuRM9e92RyQoP1gwrPXJ+McFmmgH/NwETN5aOaTThGmNN756xdyZt9qnxI3pdMs
/KUJcBM428YqlY6xFxkFWNQvkNSgC+fujwwmSGUEJ2+WdLO1UUkW2PcanRGrFj5y
VBu7KaQG9PZ6DFDcdclJQGFnxkJC6eEvWgUxA2vJQUxsfw3+NUdZeXoh3vZ+Nsvw
0Qw2DP6lZQN3IULw4kgk/Sf1tPwVVN93QfTtZK899QYwAywKjJQmkCj9oTXQuZMb
SZgf0ZMgLNpWlluRhUXxXf9GQcVbtL6kFWxxa22wGrWEmmDsd//kn4b47rPKpmu4
iSGqS035xLXV+mutjWzDtZFRvzvI4a4/va9Q4XCr3oACUDcgpHdOMpEreuaVt6o7
JdF+W4Sn80FHpsLenfh36meQ4G5LjASF1pdzpMgPJRmz2PtO30afVAQivT3jOPuJ
v/szAdIwekTLCuFEPJ3nbCQ7osxIcKUB3z3Zbin7Rr8njeev6WlvWiR3uKso29xG
80sOLzmYhLGiv6xVR2ESR/BJ+1TJXUiLn9GNBt1x8xZ3bzZGzFxIHub6nmx+7K9S
FxJ1i8Y2sjN5bE1+2LBjKRLqhQuOjwIDAQABAoICAEzTJ6WD1q23MOic0KRTY5tu
0fwbOKmTztIglS03ys5rqZnO71IiYiGeiFBJalz8YSI89Gwj2YPWrDNJkAOXuKVL
qJywSqs6iGT3hksyQnSPr/IPwAYOHCsV2pD554KxKotcqFfbWWO8tu4UPPU8aUh4
GiPNbSZvf60zBLggbNCMKUGDnHKYejeYDZmiEHmmFX4uhXadWc53sJeJ9wZpL7/5
29Xb4LIBdzHV/hl2qBZ1DV7qcUUH3MlRRl7J4RyZ/mjQ2lakbY78qliCA1SeNL2U
CzZlIc3ck4ImdjpgM3hJbQY8pBp4NAGtK4kyMaZWoiGYCGtVXASfJvUuJcGI9ESq
lxKlC+ndLfPmSFT9Yyi+34qIhCBlqpcfwT3aw8tVlVvHeI2rpPjs01s+8YnGJACY
BapcjNk0GWQZj+HrkLcOs3MUbqdpQbFIggI6US1zAzq3MrfvY03t8Qdzxsiln2nW
34nKMACAwKRcXeSSnxHE8J50VoWVZ1jaV81RqnvknGxhR4DoSmL9RWB3S9fTD1ff
qaiGzriqrl88LXqnsqiUDkgdB4brchDs6c3OF06/WS3oNRGzh/oAPFN69lTMScMg
xeFwv+auJW+c4V1Wz4ynwBZiqCy5yep2GXzJqNbl8Ad/b3hfo/K4E7mIifS/4p0H
rjLpkFkX+9DQS9GIg61xAoIBAQDFlESZdeVRkWXzMOGoYrf4a3PuLiPgq+odbYmI
Xj+kR+0hNYLWviDOxQySyxLScjT7LRuqTd9We8lItHPa/jfXEsrGkDUIOQ+WBzyd
W0SbnW0u5c6oJpj6GaQz6rcGpEAlqYgE00Ng5scXRoN/ntLiHvta0vA9P7v0iA62
nXbfGeCAaHpkoLRk/XcDBmpJueUSSJ5cZDUgSfNkLoxOM7/ZJa5oLWjcylJP/35d
wo3yM+A+VaZUW1pSLljzWnDfrEvK3ju7Kz0V9sVBoPAewtciowdp3Mi2MdlLiwLL
h0YhexDJ20QmEf4QAOOJQKaxqnoI2b3hDhsK9BaXRo+4y3WXAoIBAQD0UfnTamU7
tnOoIGweWuatJRseD8jP/8TpiWQ31k1vkaQ1BHOvaN9y66KStJUKOBTJUhf5ArQe
FZxaA3sxIxHlQitN8yuI5oNOKf42+4BCfE5ZDYn1/euksd/LNojHHOijpxoEnTaK
sHr6+RT3dvbFHzKEYxzxHbUwKPN65qAfd0LBblNLyGuqQP9fyjueV0HgQdFEtxHD
OEsteCfCYY4nxhTQ/YOg3hHX6titPmNbVXRnPmjXoCVwM64iNT7HtWuJFQI54pPi
B0DMH2kfm6njKWI+gMCnKfp3s+0bXDySxHpOn1EKj+wecFiOr/X4nN+4hBhCQ43f
APfEALR1Wv3JAoIBAQCkcNd7Zc07Sykv86IMdEUro7r0JEDGEC8k2PLbpZ2QTDDH
L/M6aTmC/iuNzShYMKd52bwpzPAx2YrYUSAPxv+QqFpOUt1gf6VCN7myObDrV2X0
311VQ/KUTV0FvLch9vhQoD3NzktIziorbAur3vMjMaf/mopKX84+IjMkt/+khbP9
C3e6YqphVzeDNAaoX+iQhBIRUXGJ8yJ4YelyeM2WnC8BE7Iv+M/zNvDkdjWPvK9J
Eh4CHZoZxetQdxh0gMEAEmBiUgVgT2czTRAseft2H3vfFtuSEAELR6JG5MpwuS1q
42xfxx/OD/9QT5etTPaOgTLwqb3GKwrWUurxYZz3AoIBAQCVzCU2wMcs42LqNGbV
/bntcxC7T8Y58YHUBk8SBS0ZONzLPN2JMO2/0kWtWVcAGv5zqQfVvxicUXe5oOTJ
bc0tGXLfqjaJC3x6UjjxkSZEnV6ULz7lOjhelEi4mckm+8yPeobzSUkFo8rjUhDO
4XvPxJ7+mJ5IH4TV8GXIdtq6MC8X1gLwNgP9MTjijGYdYTacvv4F+ZDEdyFw0Xjf
L10veb2NncI0wbODBRSws83LVAu4uYUin6gUsTsU8jx5yfwS4nSo9Qjizrul66RS
ouh4xQjddaepVo2hEwpPejARdLHgvsG7jh6hUxSY974CHnz07tjI2A6GKlu/Kwd5
5L4xAoIBABIIJOsPkaMS2Nc3iRz9GweT00QFKoZekUiwrafJa4U8bc9B5gMXja77
ozTZShXDFKxVrvTLsraCUCK4RsFMTQE+pG5Q/SEt8Lnc6D5VvY1oQu4zdbMsNfaO
vI3vVB0+IQazDIRLhV6JNCaUZAr8jxnShyz2y4N5ZJYToOp20QT5BB5z2uRdtb9Q
LrXnUNXS07bOVrBhYoi7pNbrvfiGrbrZ5aInn+NVSKy7Mkav7VaiwfhxMBwhD0kj
esbwv62ZEoAziXeW95iQxvprroZgEAgUsyZJ/cHilJ4c5YIkv2en21pGcGEtoWpv
Lc00BYUVRYhNU3H1h6CRQkbnHsNB5X4=
-----END PRIVATE KEY-----

The CA key will the one under the friendlyName: http_ca.

You can copy the key, anything between -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- and store them in a file of your choice, e.g /etc/elasticsearch/certs/http_ca_key.crt.

Configure Other Important Elasticsearch Systems Settings

Disable Memory Swapping on All Cluster Nodes

Enabling memory lock as done above is on the ways of disabling swappiness. You therefore need to ensure that memory locking is enabled on the Elasticsearch service level. This can be done as follows;

[[ -d /etc/systemd/system/elasticsearch.service.d ]] || mkdir /etc/systemd/system/elasticsearch.service.d

echo -e '[Service]\nLimitMEMLOCK=infinity' > \
/etc/systemd/system/elasticsearch.service.d/override.conf

Whenever a systemd service is modified, you need to reload the systemd configurations.

systemctl daemon-reload

One of the recommended ways to disable swapping is to completely disable swap if Elasticsearch is the only service running on the server.

swapoff -a

Edit the /etc/fstab file and comment out any lines that contain the word swap;

sed -i.bak '/swap/s/^/#/' /etc/fstab

Otherwise, disable swappiness in the kernel configuration;

echo 'vm.swappiness=1' >> /etc/sysctl.conf

sysctl -p

Set JVM Heap Size on All Cluster Nodes

Elasticsearch usually sets the heap size automatically based on the role of the node. However, if you want to go with manual configuration, as a rule of thump, set Xmx to no more than 50% of your physical RAM. Any custom JVM settings should be placed under /etc/elasticsearch/jvm.options.d.

echo -e "-Xms1g\n-Xmx1g" > /etc/elasticsearch/jvm.options.d/jvm.options

Set maximum Open File Descriptor on All Cluster Nodes

Set the maximum number of open files for the elasticsearch user to 65,536. This is already set by default in the, /usr/lib/systemd/system/elasticsearch.service.

less /usr/lib/systemd/system/elasticsearch.service

...
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535
...

You also should set the maximum number of processes.

...
# Specifies the maximum number of processes
LimitNPROC=4096

Update Virtual Memory Settings on All Cluster Nodes

Elasticsearch uses a mmapfs directory by default to store its indices. To ensure that you do not run out of virtual memory, edit the /etc/sysctl.conf and update the value of vm.max_map_count as shown below.

vm.max_map_count=262144

You can simply run the command below to configure virtual memory settings.

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

To apply the changes;

sysctl -p

To that far, below is the configuration file on each node;

grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml

Node 01;

cluster.name: kifarunix-demo
node.name: es-node01
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.12
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["es-node01"]

Node 02

cluster.name: kifarunix-demo
node.name: es-node02
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.73
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["es-node02"]

Node 03

cluster.name: kifarunix-demo
node.name: es-node03
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.50
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["en-node03"]

Start and Enable Elasticsearch Service on Node 01

For now, start and enable Elasticsearch service to run on system boot on Node 01 ONLY.

systemctl enable --now elasticsearch

Confirm that Elasticsearch is running;

systemctl status elasticsearch

The password was in the installation output;

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─override.conf
     Active: active (running) since Thu 2023-11-23 12:25:26 EST; 4s ago
       Docs: https://www.elastic.co
   Main PID: 580 (java)
      Tasks: 80 (limit: 4645)
     Memory: 1.7G
        CPU: 27.822s
     CGroup: /system.slice/elasticsearch.service
             ├─580 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcl>
             ├─638 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+>
             └─658 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 23 12:25:12 es-node01.kifarunix-demo.com systemd[1]: Starting elasticsearch.service - Elasticsearch...
Nov 23 12:25:14 es-node01.kifarunix-demo.com systemd-entrypoint[580]: Nov 23, 2023 12:25:14 PM sun.util.locale.provider.LocaleProviderAdapter 
Nov 23 12:25:14 es-node01.kifarunix-demo.com systemd-entrypoint[580]: WARNING: COMPAT locale provider will be removed in a future release
Nov 23 12:25:26 es-node01.kifarunix-demo.com systemd[1]: Started elasticsearch.service - Elasticsearch.

Confirm the ports are opened.

ss -altnp | grep -iE '92|93'

LISTEN 0      4096   [::ffff:192.168.122.12]:9200            *:*    users:(("java",pid=638,fd=438))
LISTEN 0      4096   [::ffff:192.168.122.12]:9300            *:*    users:(("java",pid=638,fd=433))

curl -k -u elastic https://es-node01:9200

{
  "name" : "es-node01",
  "cluster_name" : "kifarunix-demo",
  "cluster_uuid" : "KuX8vWFOTry9GDJCtXvo_g",
  "version" : {
    "number" : "8.11.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "6f9ff581fbcde658e6f69d6ce03050f060d1fd0c",
    "build_date" : "2023-11-11T10:05:59.421038163Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Enroll Other Nodes into Elasticsearch Cluster

At this point, Elasticsearch is running on Node 01 ONLY.

Generate Elasticsearch Cluster Enrollment Token

Next, you need to generate Elasticsearch cluster enrollment token. Do this only on a single node where ES is already started.

In this setup, we will generate Elasticsearch cluster enrollment token on ES Node01 ONLY since we have started Elasticsearch service on this node.

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

Sample token;

eyJ2ZXIiOiI4LjExLjEiLCJhZHIiOlsiMTkyLjE2OC4xMjIuMTI6OTIwMCJdLCJmZ3IiOiI0MDAxNjNiYmEzNDA3NjM4ZGEyNmNmYWI4OTVmNTFjNDU1NzU2NWNmYzljNGZhMzY4OGNhOTEwNDE1MWQxM2ViIiwia2V5IjoiMnRRNl9Zc0JsYjlwbjlBcGk2Qk06emZqU1ZPMXNRbmFvTDZ5N0FORVdHZyJ9

Once you have the token, enroll other nodes.

Enroll Elasticsearch Node 02;

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <PASTE TOKEN ABOVE> 

For example;

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjExLjEiLCJhZHIiOlsiMTkyLjE2OC4xMjIuMTI6OTIwMCJdLCJmZ3IiOiI0MDAxNjNiYmEzNDA3NjM4ZGEyNmNmYWI4OTVmNTFjNDU1NzU2NWNmYzljNGZhMzY4OGNhOTEwNDE1MWQxM2ViIiwia2V5IjoiMnRRNl9Zc0JsYjlwbjlBcGk2Qk06emZqU1ZPMXNRbmFvTDZ5N0FORVdHZyJ9

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically: 
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

Similarly, run the same enrollment command on other nodes.

Start Elasticsearch on Other Nodes

Once the enrollment is done, start the Elasticsearch service on the node.

systemctl enable --now elasticsearch

Discovery and Cluster Formation settings

If you noticed, the enrollment command reconfigures the cluster initial nodes setting, cluster.initial_master_nodes, on other nodes that are enrolled into the cluster.

Also discovery.seed_hosts: is configured with the address of the first node.

Now, we need to configure the new nodes, Node02 and Node03 in this setup, as we did with Node01.

The only change you need to do on one of the nodes, for example, we do this on Node 01 is to define the list of the cluster nodes;

vim /etc/elasticsearch/elasticsearch.yml

Basically, you need to define:

• cluster members (discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"])
• list of initial set of master-eligible nodes (cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]).

# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

Save and exit the configuration file.

Sample config on Node 01;

cat /etc/elasticsearch/elasticsearch.yml

cluster.name: kifarunix-demo
node.name: es-node01
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.12
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]

Restart Elasticsearch Service.

systemctl restart elasticsearch

Configure Node 02 just like how Node 01 has been configured. Below is our sample Node 02 configuration;

cat /etc/elasticsearch/elasticsearch.yml

cluster.name: kifarunix-demo
node.name: es-node02
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.73
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

Restart Elasticsearch Service.

systemctl restart elasticsearch

Configure Node 03 as well. Below is our sample Node 03 configuration;

cat /etc/elasticsearch/elasticsearch.yml

cluster.name: kifarunix-demo
node.name: es-node03
node.roles: [ master, data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.122.50
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["es-node01", "es-node02", "es-node03"]
cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

Restart Elasticsearch Service.

systemctl restart elasticsearch

Check the Cluster Nodes

curl -k -XGET "https://es-node01:9200/_cat/nodes?v" -u elastic

Enter host password for user 'elastic':
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.122.12           57          50   3    0.08    0.10     0.08 dm        -      es-node01
192.168.122.50           33          50   8    0.20    0.13     0.05 dm        -      es-node03
192.168.122.73           49          50   3    0.02    0.13     0.08 dm        *      es-node02

And there you go. You now have Elasticsearch 8.x cluster.

Check cluster health status;

curl -k -XGET "https://es-node01:9200/_cat/health?v" -u elastic

Enter host password for user 'elastic':
epoch      timestamp cluster        status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1700761464 17:44:24  kifarunix-demo green           3         3      2   1    0    0        0             0                  -                100.0%

As you can see, our cluster status is GREEN!!

ENSURE that the cluster transport ports (9300/tcp) are opened on the firewall on each node to allow cluster formation.

Once the cluster is formed as above, remove or comment the following line on ALL the nodes.

cluster.initial_master_nodes: ["es-node01", "es-node02", "es-node03"]

You can remove as follows;

sed -i.bak '/cluster.initial_master_nodes/s/^/#/' /etc/elasticsearch/elasticsearch.yml

Multinode cluster is up and running!

Other Tutorials

Deploy a Single Node Elastic Stack Cluster on Docker Containers

Setup Multi-node Elasticsearch Cluster