How to Fix Filebeat Glibc Related Errors on Ubuntu 22.04

In this tutorial, you will learn how to fix Filebeat Glibc related errors on Ubuntu 22.04 that is affecting users using glibc >= 2.35.

ldd --version

ldd (Ubuntu GLIBC 2.35-0ubuntu3) 2.35
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

How to Fix Filebeat Glibc Related Errors on Ubuntu 22.04

Have you installed Filebeat on Ubuntu 22.04 but realized that it cannot run due to some Glibc related errors?

In my demo setup, I have Elastic Stack 7.17.0 running on Debian 11.

curl localhost:9200

Sample output “number” : “7.17.0”;

{
  "name" : "debian11",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_JanDf4yRcCxVBnLgN0a5A",
  "version" : {
    "number" : "7.17.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
    "build_date" : "2022-01-28T08:36:04.875279988Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

It is always recommended to install similar versions of all components. That is to say, if you are running Elastic Stack v7.17.0, then Elasticsearch, Kibana, Logstash and Beasts should all be of the same versions.

Having said that, we installed Filebeat 7.17.0 and other Filebeat versions on Ubuntu 22.04 trying to check if the issue would go away. However, the same errors were experienced with some versions of Filebeat!

hostnamectl

   Static hostname: jellyfish
         Icon name: computer-convertible
           Chassis: convertible
        Machine ID: a892921910db4c7aa544a53d6f775666
           Boot ID: af6f2a87fb394cada730d00fbd56c9b1
  Operating System: Ubuntu 22.04 LTS
            Kernel: Linux 5.15.0-27-generic
      Architecture: x86-64

Sample Glibc Errors experienced on Ubuntu 22.04 with different versions of Filebeat;

Fatal glibc error: rseq registration failed

...
2022-06-08T17:13:39.465Z	INFO	instance/beat.go:328	Setup Beat: filebeat; Version: 7.17.0
2022-06-08T17:13:39.465Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.17.0' as ILM is enabled.
2022-06-08T17:13:39.465Z	INFO	[esclientleg]	eslegclient/connection.go:105	elasticsearch url: http://192.168.58.22:9200
2022-06-08T17:13:39.467Z	INFO	[publisher]	pipeline/module.go:113	Beat name: ceph-admin
2022-06-08T17:13:39.479Z	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
Fatal glibc error: rseq registration failed
Aborted (core dumped)

...
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f928b9c6a7c m=5 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f928b9c6a7c
stack: frame={sp:0x7f92634578a0, fp:0x0} stack=[0x7f9262c581e8,0x7f9263457de8)
...

If you experienced these errors, how would you fix them?

Well, according to this topic on Elastic Forum, MacKenzie mentioned that “glibc >= 2.35 added a new rseq syscall that is not in our default list of allowed syscalls“

He went ahead and suggested some fixes.

1. Upgrade to Elastic 7.17.2 and use Filebeat 7.17.2
2. Customize allowed Filebeat Syscalls to include rseq syscall

rseq, an acronym for Restartable Seqeunces, is a system call that provides synchronization mechanism for per-CPU data which super-fast update operations on per-cpu data in user-space.

Install Filebeat 7.17.2

As already stated before, it is always recommended to install similar versions of all components. As such, MacKenzie suggested that one can upgrade their Elastic Stack version to 7.17.2 and use beta release versions of Filebeat 7.17.2, the first release which ships with the fix to errors experienced when glibc >= 2.35 is used;

We installed Filebeat 7.17.2, and it indeed fixed the issue;

filebeat version

filebeat version 7.17.2 (amd64), libbeat 7.17.2 [f6042bc3407cc10201cfd8c7574d8b0a88a699db built 2022-03-28 09:47:58 +0000 UTC]

filebeat -e

...
INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(http://x.x.x.x:9200)) established

Customize Allowed Filebeat Syscalls

Use this solution if you cannot upgrade your Elastic Stack for one or the other reasons.

Filebeat is setup to utilize Linux secure computing mode (seccomp), which exposes only specific system calls to Filebeat program which thus minimizes the impact of unknown vulnerabilities that might be found in the process.

As MacKenzie mentioned, GLIBC >= 2.35 added a new req syscall which is not exposed to Filebeat by default.

Thus, in order to be able fix the Filebeat issue with Glibc without the need to upgrade your Elastic Stack, you need to configure Filebeat to allow this specific system call.

On Linux, Filebeat is set to enable seccomp by default.

To allow rseq syscall, edit the filebeat.yml configuration file and add the lines below;

seccomp:
  default_action: allow 
  syscalls:
  - action: allow
    names:
    - rseq

You can simply copy and paste the command below on the terminal to update the Filebeat configuration file with the content above;

cat >> /etc/filebeat/filebeat.yml << 'EOL'
seccomp:
  default_action: allow 
  syscalls:
  - action: allow
    names:
    - rseq
EOL

Next, you can test the changes by running;

filebeat -e

You Filebeat should successfully connect to Elasticsearch or whatever your stash output is.

Other Tutorials

Install ELK Stack 8.x on Ubuntu