Install Sysdig System Visibility Tool on Ubuntu 22.04

0
138

This tutorial will take you through how to install Sysdig system visibility tool on Ubuntu 22.04. Sysdig is a simple visibility tool that provides deep visibility into your system. According to sysdig man pages;

  • sysdig is a tool for system troubleshooting, analysis and exploration.
  • It can be used to capture, filter and decode system calls and other OS events.
  • It can be both used to inspect live systems, or to generate trace files that can be analyzed at a later stage.
  • sysdig includes a powerful filtering language, has customizable output, and can be extended through Lua scripts, called chisels.

Install Sysdig System Visibility Tool on Ubuntu 22.04

In this tutorial, we will be installing Sysdig on an Ubuntu 22.04 system.

Sysdig is provided by the default Ubuntu Universe repositories. However, the available version is a bit dated.

apt-cache policy sysdig
sysdig:
  Installed: (none)
  Candidate: 0.27.1-0.3build1
  Version table:
     0.27.1-0.3build1 500
        500 http://ke.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

The current release version of Sysdig as of this writing is v0.29.3.

Hence, there are two ways in which you can install current versions of Sysdig on Ubuntu 22.04;

  1. Automated Installation via the install script
  2. Manual Installation

Automatically Install Sysdig on Ubuntu

To automatically install Sysdig on Ubuntu, all you have to do is to download the installation script and execute it as follows. The script will basically install Sysdig Draios APT repository on your system and install Sysdig and other required packages via that repo.

curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

Sample installation output;

* Detecting operating system
* Installing Sysdig public key
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
* Installing Sysdig repository
W: https://download.sysdig.com/stable/deb/stable-amd64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
* Installing kernel headers
* Installing Sysdig
Extracting templates from packages: 100%
Preconfiguring packages ...
Selecting previously unselected package gcc-11-base:amd64.
(Reading database ... 73190 files and directories currently installed.)
Preparing to unpack .../gcc-11-base_11.2.0-19ubuntu1_amd64.deb ...
Unpacking gcc-11-base:amd64 (11.2.0-19ubuntu1) ...
Preparing to unpack .../libc6_2.35-0ubuntu3.1_amd64.deb ...
Unpacking libc6:amd64 (2.35-0ubuntu3.1) over (2.35-0ubuntu3) ...
Setting up libc6:amd64 (2.35-0ubuntu3.1) ...
Selecting previously unselected package libisl23:amd64.
(Reading database ... 73195 files and directories currently installed.)
Preparing to unpack .../00-libisl23_0.24-2build1_amd64.deb ...
Unpacking libisl23:amd64 (0.24-2build1) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../01-libmpc3_1.2.1-2build1_amd64.deb ...
Unpacking libmpc3:amd64 (1.2.1-2build1) ...
Selecting previously unselected package cpp-11.
Preparing to unpack .../02-cpp-11_11.2.0-19ubuntu1_amd64.deb ...
Unpacking cpp-11 (11.2.0-19ubuntu1) ...
Selecting previously unselected package cpp.
Preparing to unpack .../03-cpp_4%3a11.2.0-1ubuntu1_amd64.deb ...
Unpacking cpp (4:11.2.0-1ubuntu1) ...
Selecting previously unselected package libcc1-0:amd64.
Preparing to unpack .../04-libcc1-0_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libcc1-0:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libgomp1:amd64.
Preparing to unpack .../05-libgomp1_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libgomp1:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libitm1:amd64.
Preparing to unpack .../06-libitm1_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libitm1:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../07-libatomic1_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libatomic1:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libasan6:amd64.
Preparing to unpack .../08-libasan6_11.2.0-19ubuntu1_amd64.deb ...
Unpacking libasan6:amd64 (11.2.0-19ubuntu1) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../09-liblsan0_12-20220319-1ubuntu1_amd64.deb ...
Unpacking liblsan0:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libtsan0:amd64.
Preparing to unpack .../10-libtsan0_11.2.0-19ubuntu1_amd64.deb ...
Unpacking libtsan0:amd64 (11.2.0-19ubuntu1) ...
Selecting previously unselected package libubsan1:amd64.
Preparing to unpack .../11-libubsan1_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libubsan1:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libquadmath0:amd64.
Preparing to unpack .../12-libquadmath0_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libquadmath0:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libgcc-11-dev:amd64.
Preparing to unpack .../13-libgcc-11-dev_11.2.0-19ubuntu1_amd64.deb ...
Unpacking libgcc-11-dev:amd64 (11.2.0-19ubuntu1) ...
Selecting previously unselected package gcc-11.
Preparing to unpack .../14-gcc-11_11.2.0-19ubuntu1_amd64.deb ...
Unpacking gcc-11 (11.2.0-19ubuntu1) ...
Selecting previously unselected package gcc.
Preparing to unpack .../15-gcc_4%3a11.2.0-1ubuntu1_amd64.deb ...
Unpacking gcc (4:11.2.0-1ubuntu1) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../16-libdpkg-perl_1.21.1ubuntu2.1_all.deb ...
Unpacking libdpkg-perl (1.21.1ubuntu2.1) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../17-bzip2_1.0.8-5build1_amd64.deb ...
Unpacking bzip2 (1.0.8-5build1) ...
Selecting previously unselected package make.
Preparing to unpack .../18-make_4.3-4.1build1_amd64.deb ...
Unpacking make (4.3-4.1build1) ...
Selecting previously unselected package lto-disabled-list.
Preparing to unpack .../19-lto-disabled-list_24_all.deb ...
Unpacking lto-disabled-list (24) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../20-dpkg-dev_1.21.1ubuntu2.1_all.deb ...
Unpacking dpkg-dev (1.21.1ubuntu2.1) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../21-libc-dev-bin_2.35-0ubuntu3.1_amd64.deb ...
Unpacking libc-dev-bin (2.35-0ubuntu3.1) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../22-linux-libc-dev_5.15.0-43.46_amd64.deb ...
Unpacking linux-libc-dev:amd64 (5.15.0-43.46) ...
Selecting previously unselected package libcrypt-dev:amd64.
Preparing to unpack .../23-libcrypt-dev_1%3a4.4.27-1_amd64.deb ...
Unpacking libcrypt-dev:amd64 (1:4.4.27-1) ...
Selecting previously unselected package rpcsvc-proto.
Preparing to unpack .../24-rpcsvc-proto_1.4.2-0ubuntu6_amd64.deb ...
Unpacking rpcsvc-proto (1.4.2-0ubuntu6) ...
Preparing to unpack .../25-libtirpc-common_1.3.2-2ubuntu0.1_all.deb ...
Unpacking libtirpc-common (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
Setting up libtirpc-common (1.3.2-2ubuntu0.1) ...
(Reading database ... 75034 files and directories currently installed.)
Preparing to unpack .../libtirpc3_1.3.2-2ubuntu0.1_amd64.deb ...
Unpacking libtirpc3:amd64 (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
Setting up libtirpc3:amd64 (1.3.2-2ubuntu0.1) ...
Selecting previously unselected package libtirpc-dev:amd64.
(Reading database ... 75034 files and directories currently installed.)
Preparing to unpack .../00-libtirpc-dev_1.3.2-2ubuntu0.1_amd64.deb ...
Unpacking libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
Selecting previously unselected package libnsl-dev:amd64.
Preparing to unpack .../01-libnsl-dev_1.3.0-2build2_amd64.deb ...
Unpacking libnsl-dev:amd64 (1.3.0-2build2) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../02-libc6-dev_2.35-0ubuntu3.1_amd64.deb ...
Unpacking libc6-dev:amd64 (2.35-0ubuntu3.1) ...
Selecting previously unselected package libstdc++-11-dev:amd64.
Preparing to unpack .../03-libstdc++-11-dev_11.2.0-19ubuntu1_amd64.deb ...
Unpacking libstdc++-11-dev:amd64 (11.2.0-19ubuntu1) ...
Selecting previously unselected package g++-11.
Preparing to unpack .../04-g++-11_11.2.0-19ubuntu1_amd64.deb ...
Unpacking g++-11 (11.2.0-19ubuntu1) ...
Selecting previously unselected package g++.
Preparing to unpack .../05-g++_4%3a11.2.0-1ubuntu1_amd64.deb ...
Unpacking g++ (4:11.2.0-1ubuntu1) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../06-build-essential_12.9ubuntu3_amd64.deb ...
Unpacking build-essential (12.9ubuntu3) ...
Selecting previously unselected package dctrl-tools.
Preparing to unpack .../07-dctrl-tools_2.24-3build2_amd64.deb ...
Unpacking dctrl-tools (2.24-3build2) ...
Selecting previously unselected package dkms.
Preparing to unpack .../08-dkms_2.8.7-2ubuntu2_all.deb ...
Unpacking dkms (2.8.7-2ubuntu2) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../09-libfakeroot_1.28-1ubuntu1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.28-1ubuntu1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../10-fakeroot_1.28-1ubuntu1_amd64.deb ...
Unpacking fakeroot (1.28-1ubuntu1) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../11-fonts-dejavu-core_2.37-2build1_all.deb ...
Unpacking fonts-dejavu-core (2.37-2build1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../12-fontconfig-config_2.13.1-4.2ubuntu5_all.deb ...
Unpacking fontconfig-config (2.13.1-4.2ubuntu5) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../13-libalgorithm-diff-perl_1.201-1_all.deb ...
Unpacking libalgorithm-diff-perl (1.201-1) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../14-libalgorithm-diff-xs-perl_0.04-6build3_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-6build3) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../15-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../16-libfontconfig1_2.13.1-4.2ubuntu5_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
Selecting previously unselected package libjpeg-turbo8:amd64.
Preparing to unpack .../17-libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb ...
Unpacking libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
Selecting previously unselected package libjpeg8:amd64.
Preparing to unpack .../18-libjpeg8_8c-2ubuntu10_amd64.deb ...
Unpacking libjpeg8:amd64 (8c-2ubuntu10) ...
Selecting previously unselected package libdeflate0:amd64.
Preparing to unpack .../19-libdeflate0_1.10-2_amd64.deb ...
Unpacking libdeflate0:amd64 (1.10-2) ...
Selecting previously unselected package libjbig0:amd64.
Preparing to unpack .../20-libjbig0_2.1-3.1build3_amd64.deb ...
Unpacking libjbig0:amd64 (2.1-3.1build3) ...
Selecting previously unselected package libwebp7:amd64.
Preparing to unpack .../21-libwebp7_1.2.2-2_amd64.deb ...
Unpacking libwebp7:amd64 (1.2.2-2) ...
Selecting previously unselected package libtiff5:amd64.
Preparing to unpack .../22-libtiff5_4.3.0-6_amd64.deb ...
Unpacking libtiff5:amd64 (4.3.0-6) ...
Selecting previously unselected package libxpm4:amd64.
Preparing to unpack .../23-libxpm4_1%3a3.5.12-1build2_amd64.deb ...
Unpacking libxpm4:amd64 (1:3.5.12-1build2) ...
Selecting previously unselected package libgd3:amd64.
Preparing to unpack .../24-libgd3_2.3.0-2ubuntu2_amd64.deb ...
Unpacking libgd3:amd64 (2.3.0-2ubuntu2) ...
Selecting previously unselected package libc-devtools.
Preparing to unpack .../25-libc-devtools_2.35-0ubuntu3.1_amd64.deb ...
Unpacking libc-devtools (2.35-0ubuntu3.1) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../26-libfile-fcntllock-perl_0.22-3build7_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-3build7) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../27-manpages-dev_5.10-1ubuntu1_all.deb ...
Unpacking manpages-dev (5.10-1ubuntu1) ...
Selecting previously unselected package sysdig.
Preparing to unpack .../28-sysdig_0.29.3_amd64.deb ...
Unpacking sysdig (0.29.3) ...
Setting up gcc-11-base:amd64 (11.2.0-19ubuntu1) ...
Setting up manpages-dev (5.10-1ubuntu1) ...
Setting up lto-disabled-list (24) ...
Setting up libxpm4:amd64 (1:3.5.12-1build2) ...
Setting up libfile-fcntllock-perl (0.22-3build7) ...
Setting up libalgorithm-diff-perl (1.201-1) ...
Setting up libdeflate0:amd64 (1.10-2) ...
Setting up linux-libc-dev:amd64 (5.15.0-43.46) ...
Setting up libgomp1:amd64 (12-20220319-1ubuntu1) ...
Setting up bzip2 (1.0.8-5build1) ...
Setting up libjbig0:amd64 (2.1-3.1build3) ...
Setting up libfakeroot:amd64 (1.28-1ubuntu1) ...
Setting up libasan6:amd64 (11.2.0-19ubuntu1) ...
Setting up fakeroot (1.28-1ubuntu1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
Setting up rpcsvc-proto (1.4.2-0ubuntu6) ...
Setting up make (4.3-4.1build1) ...
Setting up libquadmath0:amd64 (12-20220319-1ubuntu1) ...
Setting up libmpc3:amd64 (1.2.1-2build1) ...
Setting up libatomic1:amd64 (12-20220319-1ubuntu1) ...
Setting up fonts-dejavu-core (2.37-2build1) ...
Setting up libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
Setting up libdpkg-perl (1.21.1ubuntu2.1) ...
Setting up libwebp7:amd64 (1.2.2-2) ...
Setting up libubsan1:amd64 (12-20220319-1ubuntu1) ...
Setting up libnsl-dev:amd64 (1.3.0-2build2) ...
Setting up libcrypt-dev:amd64 (1:4.4.27-1) ...
Setting up libisl23:amd64 (0.24-2build1) ...
Setting up libc-dev-bin (2.35-0ubuntu3.1) ...
Setting up libalgorithm-diff-xs-perl (0.04-6build3) ...
Setting up libcc1-0:amd64 (12-20220319-1ubuntu1) ...
Setting up liblsan0:amd64 (12-20220319-1ubuntu1) ...
Setting up dctrl-tools (2.24-3build2) ...
Setting up libitm1:amd64 (12-20220319-1ubuntu1) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up libtsan0:amd64 (11.2.0-19ubuntu1) ...
Setting up libjpeg8:amd64 (8c-2ubuntu10) ...
Setting up cpp-11 (11.2.0-19ubuntu1) ...
Setting up fontconfig-config (2.13.1-4.2ubuntu5) ...
Setting up dpkg-dev (1.21.1ubuntu2.1) ...
Setting up libgcc-11-dev:amd64 (11.2.0-19ubuntu1) ...
Setting up gcc-11 (11.2.0-19ubuntu1) ...
Setting up cpp (4:11.2.0-1ubuntu1) ...
Setting up libc6-dev:amd64 (2.35-0ubuntu3.1) ...
Setting up libtiff5:amd64 (4.3.0-6) ...
Setting up libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
Setting up gcc (4:11.2.0-1ubuntu1) ...
Setting up dkms (2.8.7-2ubuntu2) ...
Setting up libgd3:amd64 (2.3.0-2ubuntu2) ...
Setting up sysdig (0.29.3) ...
Loading new scap-e5c53d648f3c4694385bbe488e7d47eaa36c229a DKMS files...
Building for 5.15.0-27-generic
Building initial module for 5.15.0-27-generic
Done.

scap.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.15.0-27-generic/updates/dkms/

depmod.....
Setting up libstdc++-11-dev:amd64 (11.2.0-19ubuntu1) ...
Setting up libc-devtools (2.35-0ubuntu3.1) ...
Setting up g++-11 (11.2.0-19ubuntu1) ...
Setting up g++ (4:11.2.0-1ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.9ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-27-generic
NEEDRESTART-KEXP: 5.15.0-27-generic
NEEDRESTART-KSTA: 1
NEEDRESTART-SVC: cron.service
NEEDRESTART-SVC: dbus.service
NEEDRESTART-SVC: irqbalance.service
NEEDRESTART-SVC: ModemManager.service
NEEDRESTART-SVC: multipathd.service
NEEDRESTART-SVC: networkd-dispatcher.service
NEEDRESTART-SVC: packagekit.service
NEEDRESTART-SVC: polkit.service
NEEDRESTART-SVC: rsyslog.service
NEEDRESTART-SVC: snapd.service
NEEDRESTART-SVC: ssh.service
NEEDRESTART-SVC: systemd-journald.service
NEEDRESTART-SVC: systemd-logind.service
NEEDRESTART-SVC: systemd-networkd.service
NEEDRESTART-SVC: systemd-resolved.service
NEEDRESTART-SVC: systemd-timesyncd.service
NEEDRESTART-SVC: systemd-udevd.service
NEEDRESTART-SVC: udisks2.service
NEEDRESTART-SVC: unattended-upgrades.service
NEEDRESTART-SVC: [email protected]

Install Sysdig Manually on Ubuntu 22.04

To manually install Sysdig on Ubuntu 22.04;

  • Install Sysdig Draios APT Repository
curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public \
| gpg --dearmor > /etc/apt/trusted.gpg.d/sysdig.gpg
curl -s https://s3.amazonaws.com/download.draios.com/stable/deb/draios.list \
-o /etc/apt/sources.list.d/sysdig-draios.list 
  • Run system package cache update
apt update
  • Install Kernel Headers
apt install linux-headers-$(uname -r)
  • Install Sysdig system visibility tool on Ubuntu 22.04
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  build-essential bzip2 cpp cpp-11 dctrl-tools dkms dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6 libc6-dev libcc1-0 libcrypt-dev libdeflate0 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev
  libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto
Suggested packages:
  bzip2-doc cpp-doc gcc-11-locales debtags menu debian-keyring g++-multilib g++-11-multilib gcc-11-doc gcc-multilib autoconf automake libtool flex bison gdb gcc-doc
  gcc-11-multilib glibc-doc bzr libgd-tools libstdc++-11-doc make-doc
Recommended packages:
  libnss-nis libnss-nisplus
The following NEW packages will be installed:
  build-essential bzip2 cpp cpp-11 dctrl-tools dkms dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0 libdpkg-perl libfakeroot
  libfile-fcntllock-perl libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev libquadmath0
  libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto sysdig
The following packages will be upgraded:
  libc6
1 upgraded, 55 newly installed, 0 to remove and 37 not upgraded.
Need to get 81.6 MB of archives.
After this operation, 246 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Kernel headers might be upgrade in the process. You will therefore have to reboot the system.

systemctl reboot -i

Check Installed Sysdig Version

Confirming the installed version;

sysdig --version

Sample output;

sysdig version 0.29.3

Running Sysdig System Visibility Tool

Sysdig can be executed from the command line by just running;

sysdig

When run with no command line arguments, Sysdig prints the information for each captured system event on a single line, with the following format:

*%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.info

Where;

  • evt.num is the incremental event number
  • evt.time is the event timestamp
  • evt.cpu is the CPU number where the event was captured
  • proc.name is the name of the process that generated the event
  • thread.tid id the TID that generated the event, which corresponds to the PID for single thread processes
  • evt.dir is the event direction, > for enter events and < for exit events
  • evt.type is the name of the event, e.g. ‘open’ or ‘read’
  • evt.args is the list of event arguments.

Sample output;

180 13:24:24.825700257 0 sudo (1417.1417) > write fd=8(/dev/tty) size=2 
181 13:24:24.825701386 0 sudo (1417.1417) < write res=2 data=.. 
182 13:24:24.825701876 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
183 13:24:24.825703162 1 sshd (1407.1407) > getpid 
184 13:24:24.825703262 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
185 13:24:24.825704070 1 sshd (1407.1407) < getpid 
186 13:24:24.825704104 0 sudo (1417.1417) > rt_sigaction 
187 13:24:24.825704316 0 sudo (1417.1417) < rt_sigaction 
188 13:24:24.825704837 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 
189 13:24:24.825706544 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
191 13:24:24.825714684 1 sshd (1407.1407) > rt_sigprocmask 
193 13:24:24.825715005 1 sshd (1407.1407) < rt_sigprocmask 
194 13:24:24.825716551 1 sshd (1407.1407) > ppoll fds=4:41 4:44 10:f1 7:f0 timeout=none sigmask= 
195 13:24:24.825718500 1 sshd (1407.1407) < ppoll res=2 fds=4:44 10:f1 
196 13:24:24.825719227 1 sshd (1407.1407) > rt_sigprocmask 
197 13:24:24.825719449 1 sshd (1407.1407) < rt_sigprocmask 
198 13:24:24.825720103 1 sshd (1407.1407) > read fd=10(/dev/ptmx) size=32768 
200 13:24:24.825721199 1 sshd (1407.1407) < read res=2 data=.. 
201 13:24:24.825722309 1 sshd (1407.1407) > write fd=4(<4t>192.168.56.1:48792->192.168.56.143:22) size=356 
202 13:24:24.825724281 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 
203 13:24:24.825727253 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
204 13:24:24.825729171 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 
205 13:24:24.825731162 0 sudo (1417.1417) > rt_sigaction 
206 13:24:24.825731625 0 sudo (1417.1417) < rt_sigaction 
207 13:24:24.825732245 0 sudo (1417.1417) > read fd=9(/dev/ptmx) size=65536 
208 13:24:24.825733171 0 sudo (1417.1417) < read res=109 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 
209 13:24:24.825733991 0 sudo (1417.1417) > rt_sigaction 
210 13:24:24.825734162 0 sudo (1417.1417) < rt_sigaction 
211 13:24:24.825734643 0 sudo (1417.1417) > rt_sigprocmask 
212 13:24:24.825734870 0 sudo (1417.1417) < rt_sigprocmask 
213 13:24:24.825735287 0 sudo (1417.1417) > rt_sigprocmask 
214 13:24:24.825735442 0 sudo (1417.1417) < rt_sigprocmask 
215 13:24:24.825736119 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 8:f4 8:f1 9:f1 timeout=none sigmask= 
216 13:24:24.825738008 0 sudo (1417.1417) < ppoll res=1 fds=8:f4 
217 13:24:24.825739012 0 sudo (1417.1417) > rt_sigaction 
218 13:24:24.825739197 0 sudo (1417.1417) < rt_sigaction 
219 13:24:24.825739623 0 sudo (1417.1417) > write fd=8(/dev/tty) size=109 
220 13:24:24.825740912 0 sudo (1417.1417) < write res=109 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 
221 13:24:24.825741696 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
222 13:24:24.825743109 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
223 13:24:24.825743972 0 sudo (1417.1417) > rt_sigaction 
224 13:24:24.825744338 0 sudo (1417.1417) < rt_sigaction 
225 13:24:24.825744876 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 
226 13:24:24.825746787 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
228 13:24:24.825749220 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 
229 13:24:24.825750789 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
230 13:24:24.825752221 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 
231 13:24:24.825753358 0 sudo (1417.1417) > rt_sigaction 
232 13:24:24.825753560 0 sudo (1417.1417) < rt_sigaction 
233 13:24:24.825754261 0 sudo (1417.1417) > read fd=9(/dev/ptmx) size=65536 
234 13:24:24.825754931 0 sudo (1417.1417) < read res=2 data=.. 
235 13:24:24.825755384 0 sudo (1417.1417) > rt_sigaction 
236 13:24:24.825755548 0 sudo (1417.1417) < rt_sigaction 
237 13:24:24.825756138 0 sudo (1417.1417) > rt_sigprocmask 
238 13:24:24.825756347 0 sudo (1417.1417) < rt_sigprocmask 
239 13:24:24.825756827 0 sudo (1417.1417) > rt_sigprocmask 
240 13:24:24.825756982 0 sudo (1417.1417) < rt_sigprocmask 
241 13:24:24.825757601 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 8:f4 8:f1 9:f1 timeout=none sigmask= 
242 13:24:24.825758860 0 sudo (1417.1417) < ppoll res=1 fds=8:f4 
243 13:24:24.825759579 0 sudo (1417.1417) > rt_sigaction 
244 13:24:24.825759872 0 sudo (1417.1417) < rt_sigaction 
245 13:24:24.825760299 0 sudo (1417.1417) > write fd=8(/dev/tty) size=2 
246 13:24:24.825761578 0 sudo (1417.1417) < write res=2 data=.. 
247 13:24:24.825762138 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
248 13:24:24.825763510 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
249 13:24:24.825764361 0 sudo (1417.1417) > rt_sigaction 
250 13:24:24.825764540 0 sudo (1417.1417) < rt_sigaction 
251 13:24:24.825765051 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 
252 13:24:24.825766787 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 
255 13:24:24.825776112 1 sshd (1407.1407) < write res=356 data=^.......3.p9...cI.../.eu.....f..I.0\D..50.$..kGL...K..S`...A..$...........J..[.. 
256 13:24:24.825778159 1 sshd (1407.1407) > getpid 
257 13:24:24.825778500 1 sshd (1407.1407) < getpid 
259 13:24:24.825781130 1 sshd (1407.1407) > rt_sigprocmask 
260 13:24:24.825781448 1 sshd (1407.1407) < rt_sigprocmask 
261 13:24:24.825782499 1 sshd (1407.1407) > ppoll fds=4:41 4:44 10:f1 7:f0 timeout=none sigmask= 
262 13:24:24.825784482 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 
263 13:24:24.825784780 1 sshd (1407.1407) < ppoll res=2 fds=4:44 10:f1 
264 13:24:24.825785407 1 sshd (1407.1407) > rt_sigprocmask 
265 13:24:24.825785602 1 sshd (1407.1407) < rt_sigprocmask 
266 13:24:24.825786280 1 sshd (1407.1407) > read fd=10(/dev/ptmx) size=32768 
267 13:24:24.825786972 0  (.4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 
268 13:24:24.825787376 1 sshd (1407.1407) < read res=111 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 
269 13:24:24.825788199 1 sshd (1407.1407) > write fd=4(<4t>192.168.56.1:48792->192.168.56.143:22) size=36 
270 13:24:24.825788929 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 
271 13:24:24.825789978 1 sshd (1407.1407) < write res=36 data=#.Z.,.}[email protected]~+...>.z.e^8.>.1.+.....\ 

Filtering Sysdig command Output

The output of sysdig can be filtered using various event fields.

You can check available sysdig event fields using the command;

sysdig -l

Sample output;

-------------------------------
Field Class:                  evt (All event types)
Description:                  These fields can be used for all event types 

evt.num                       event number. 
evt.time                      event timestamp as a time string that includes the nanosecond part. 
evt.time.s                    event timestamp as a time string with no nanoseconds. 
evt.time.iso8601              event timestamp in ISO 8601 format, including nanoseconds and time zone offset (in UTC). 
evt.datetime                  event timestamp as a time string that includes the date. 
evt.datetime.s                event timestamp as a datetime string with no nanoseconds. 
evt.rawtime                   absolute event timestamp, i.e. nanoseconds from epoch. 
evt.rawtime.s                 integer part of the event timestamp (e.g. seconds since epoch). 
evt.rawtime.ns                fractional part of the absolute event timestamp. 
evt.reltime                   number of nanoseconds from the beginning of the capture. 
evt.reltime.s                 number of seconds from the beginning of the capture. 
evt.reltime.ns                fractional part (in ns) of the time from the beginning of the capture. 
evt.pluginname                if the event comes from a plugin, the name of the plugin that generated it. The plugin 
                              must be currently loaded. 
evt.plugininfo                if the event comes from a plugin, a summary of the event as formatted by the plugin. The 
                              plugin must be currently loaded. 

-------------------------------
Field Class:                  evt (Syscall events only)
Description:                  Event fields applicable to syscall events. Note that for most events you can access the 
                              individual arguments/parameters of each syscall via evt.arg, e.g. evt.arg.filename. 

evt.latency                   delta between an exit event and the correspondent enter event, in nanoseconds. 
evt.latency.s                 integer part of the event latency delta. 
evt.latency.ns                fractional part of the event latency delta. 
evt.latency.human             delta between an exit event and the correspondent enter event, as a human readable string 
                              (e.g. 10.3ms). 
evt.deltatime                 delta between this event and the previous event, in nanoseconds. 
evt.deltatime.s               integer part of the delta between this event and the previous event. 
evt.deltatime.ns              fractional part of the delta between this event and the previous event. 
evt.dir                       event direction can be either '>' for enter events or '<' for exit events. 
evt.type                      The name of the event (e.g. 'open'). 
evt.type.is                   allows one to specify an event type, and returns 1 for events that are of that type. For 
                              example, evt.type.is.open returns 1 for open events, 0 for any other event. 
syscall.type                  For system call events, the name of the system call (e.g. 'open'). Unset for other events 
                              (e.g. switch or internal events). Use this field instead of evt.type if you need to make 
                              sure that the filtered/printed value is actually a system call. 
evt.category                  The event category. Example values are 'file' (for file operations like open and close), 
                              'net' (for network operations like socket and bind), memory (for things like brk or 
                              mmap), and so on. 
evt.cpu                       number of the CPU where this event happened. 
evt.args                      all the event arguments, aggregated into a single string. 
evt.arg                       one of the event arguments specified by name or by number. Some events (e.g. return codes 
                              or FDs) will be converted into a text representation when possible. E.g. 'evt.arg.fd' or 
                              'evt.arg[0]'. 
evt.rawarg                    one of the event arguments specified by name. E.g. 'evt.rawarg.fd'. 
evt.info                      for most events, this field returns the same value as evt.args. However, for some events 
                              (like writes to /dev/log) it provides higher level information coming from decoding the 
                              arguments. 
evt.buffer                    the binary data buffer for events that have one, like read(), recvfrom(), etc. Use this 
                              field in filters with 'contains' to search into I/O data buffers. 
evt.buflen                    the length of the binary data buffer for events that have one, like read(), recvfrom(), 
                              etc. 
evt.res                       event return value, as a string. If the event failed, the result is an error code string 
                              (e.g. 'ENOENT'), otherwise the result is the string 'SUCCESS'. 
evt.rawres                    event return value, as a number (e.g. -2). Useful for range comparisons. 
evt.failed                    'true' for events that returned an error status. 
evt.is_io                     'true' for events that read or write to FDs, like read(), send, recvfrom(), etc. 
evt.is_io_read                'true' for events that read from FDs, like read(), recv(), recvfrom(), etc. 
evt.is_io_write               'true' for events that write to FDs, like write(), send(), etc. 
evt.io_dir                    'r' for events that read from FDs, like read(); 'w' for events that write to FDs, like 
                              write(). 
evt.is_wait                   'true' for events that make the thread wait, e.g. sleep(), select(), poll(). 
evt.wait_latency              for events that make the thread wait (e.g. sleep(), select(), poll()), this is the time 
                              spent waiting for the event to return, in nanoseconds. 
evt.is_syslog                 'true' for events that are writes to /dev/log. 
evt.count                     This filter field always returns 1 and can be used to count events from inside chisels. 
evt.count.error               This filter field returns 1 for events that returned with an error, and can be used to 
                              count event failures from inside chisels. 
evt.count.error.file          This filter field returns 1 for events that returned with an error and are related to 
                              file I/O, and can be used to count event failures from inside chisels. 
evt.count.error.net           This filter field returns 1 for events that returned with an error and are related to 
                              network I/O, and can be used to count event failures from inside chisels. 
evt.count.error.memory        This filter field returns 1 for events that returned with an error and are related to 
                              memory allocation, and can be used to count event failures from inside chisels. 
evt.count.error.other         This filter field returns 1 for events that returned with an error and are related to 
                              none of the previous categories, and can be used to count event failures from inside 
                              chisels. 
evt.count.exit                This filter field returns 1 for exit events, and can be used to count single events from 
                              inside chisels. 
evt.around                    (FILTER ONLY) Accepts the event if it's around the specified time interval. The syntax is 
                              evt.around[T]=D, where T is the value returned by %evt.rawtime for the event and D is a 
                              delta in milliseconds. For example, evt.around[1404996934793590564]=1000 will return the 
                              events with timestamp with one second before the timestamp and one second after it, for a 
                              total of two seconds of capture. 
evt.abspath                   Absolute path calculated from dirfd and name during syscalls like renameat and symlinkat. 
                              Use 'evt.abspath.src' or 'evt.abspath.dst' for syscalls that support multiple paths. 
evt.is_open_read              'true' for open/openat/openat2 events where the path was opened for reading 
evt.is_open_write             'true' for open/openat/openat2 events where the path was opened for writing 
evt.is_open_exec              'true' for open/openat/openat2 or creat events where a file is created with execute 
                              permissions 

-------------------------------
Field Class:                  process
Description:                  Additional information about the process and thread executing the syscall event. 

proc.pid                      the id of the process generating the event. 
proc.exe                      the first command line argument (usually the executable name or a custom one). 
proc.name                     the name (excluding the path) of the executable generating the event. 
proc.args                     the arguments passed on the command line when starting the process generating the event. 
proc.env                      the environment variables of the process generating the event. 
proc.cmdline                  full process command line, i.e. proc.name + proc.args. 
proc.exeline                  full process command line, with exe as first argument, i.e. proc.exe + proc.args. 
proc.cwd                      the current working directory of the event. 
proc.nthreads                 the number of threads that the process generating the event currently has, including the 
                              main process thread. 
proc.nchilds                  the number of child threads that the process generating the event currently has. This 
                              excludes the main process thread. 
proc.ppid                     the pid of the parent of the process generating the event. 
proc.pname                    the name (excluding the path) of the parent of the process generating the event. 
proc.pcmdline                 the full command line (proc.name + proc.args) of the parent of the process generating the 
                              event. 
proc.apid                     the pid of one of the process ancestors. E.g. proc.apid[1] returns the parent pid, 
                              proc.apid[2] returns the grandparent pid, and so on. proc.apid[0] is the pid of the 
                              current process. proc.apid without arguments can be used in filters only and matches any 
                              of the process ancestors, e.g. proc.apid=1234. 
proc.aname                    the name (excluding the path) of one of the process ancestors. E.g. proc.aname[1] returns 
                              the parent name, proc.aname[2] returns the grandparent name, and so on. proc.aname[0] is 
                              the name of the current process. proc.aname without arguments can be used in filters only 
                              and matches any of the process ancestors, e.g. proc.aname=bash. 
proc.loginshellid             the pid of the oldest shell among the ancestors of the current process, if there is one. 
                              This field can be used to separate different user sessions, and is useful in conjunction 
                              with chisels like spy_user. 
proc.duration                 number of nanoseconds since the process started. 
proc.fdopencount              number of open FDs for the process 
proc.fdlimit                  maximum number of FDs the process can open. 
proc.fdusage                  the ratio between open FDs and maximum available FDs for the process. 
proc.vmsize                   total virtual memory for the process (as kb). 
proc.vmrss                    resident non-swapped memory for the process (as kb). 
proc.vmswap                   swapped memory for the process (as kb). 
thread.pfmajor                number of major page faults since thread start. 
thread.pfminor                number of minor page faults since thread start. 
thread.tid                    the id of the thread generating the event. 
thread.ismain                 'true' if the thread generating the event is the main one in the process. 
thread.exectime               CPU time spent by the last scheduled thread, in nanoseconds. Exported by switch events 
                              only. 
thread.totexectime            Total CPU time, in nanoseconds since the beginning of the capture, for the current 
                              thread. Exported by switch events only. 
thread.cgroups                all the cgroups the thread belongs to, aggregated into a single string. 
thread.cgroup                 the cgroup the thread belongs to, for a specific subsystem. E.g. thread.cgroup.cpuacct. 
thread.vtid                   the id of the thread generating the event as seen from its current PID namespace. 
proc.vpid                     the id of the process generating the event as seen from its current PID namespace. 
thread.cpu                    the CPU consumed by the thread in the last second. 
thread.cpu.user               the user CPU consumed by the thread in the last second. 
thread.cpu.system             the system CPU consumed by the thread in the last second. 
thread.vmsize                 For the process main thread, this is the total virtual memory for the process (as kb). 
                              For the other threads, this field is zero. 
thread.vmrss                  For the process main thread, this is the resident non-swapped memory for the process (as 
                              kb). For the other threads, this field is zero. 
proc.sid                      the session id of the process generating the event. 
proc.sname                    the name of the current process's session leader. This is either the process with 
                              pid=proc.sid or the eldest ancestor that has the same sid as the current process. 
proc.tty                      The controlling terminal of the process. 0 for processes without a terminal. 
proc.exepath                  The full executable path of the process. 
proc.vpgid                    the process group id of the process generating the event, as seen from its current PID 
                              namespace. 
proc.is_container_healthcheck true if this process is running as a part of the container's health check. 
proc.is_container_liveness_probe
                              true if this process is running as a part of the container's liveness probe. 
proc.is_container_readiness_probe
                              true if this process is running as a part of the container's readiness probe. 
proc.is_exe_writable          true if this process' executable file is writable by the same user that spawned the 
                              process. 

-------------------------------
Field Class:                  user
Description:                  Information about the user executing the specific event. 

user.uid                      user ID. 
user.name                     user name. 
user.homedir                  home directory of the user. 
user.shell                    user's shell. 
user.loginuid                 audit user id (auid). 
user.loginname                audit user name (auid). 

-------------------------------
Field Class:                  group
Description:                  Information about the user group. 

group.gid                     group ID. 
group.name                    group name. 

-------------------------------
Field Class:                  container
Description:                  Container information. If the event is not happening inside a container, both id and name 
                              will be set to 'host'. 

container.id                  the container id. 
container.name                the container name. 
container.image               the container image name (e.g. falcosecurity/falco:latest for docker). 
container.image.id            the container image id (e.g. 6f7e2741b66b). 
container.type                the container type, eg: docker or rkt 
container.privileged          true for containers running as privileged, false otherwise 
container.mounts              A space-separated list of mount information. Each item in the list has the format 
                              :::: 
container.mount               Information about a single mount, specified by number (e.g. container.mount[0]) or mount 
                              source (container.mount[/usr/local]). The pathname can be a glob 
                              (container.mount[/usr/local/*]), in which case the first matching mount will be returned. 
                              The information has the format ::::. If there is 
                              no mount with the specified index or matching the provided source, returns the string 
                              "none" instead of a NULL value. 
container.mount.source        the mount source, specified by number (e.g. container.mount.source[0]) or mount 
                              destination (container.mount.source[/host/lib/modules]). The pathname can be a glob. 
container.mount.dest          the mount destination, specified by number (e.g. container.mount.dest[0]) or mount source 
                              (container.mount.dest[/lib/modules]). The pathname can be a glob. 
container.mount.mode          the mount mode, specified by number (e.g. container.mount.mode[0]) or mount source 
                              (container.mount.mode[/usr/local]). The pathname can be a glob. 
container.mount.rdwr          the mount rdwr value, specified by number (e.g. container.mount.rdwr[0]) or mount source 
                              (container.mount.rdwr[/usr/local]). The pathname can be a glob. 
container.mount.propagation   the mount propagation value, specified by number (e.g. container.mount.propagation[0]) or 
                              mount source (container.mount.propagation[/usr/local]). The pathname can be a glob. 
container.image.repository    the container image repository (e.g. falcosecurity/falco). 
container.image.tag           the container image tag (e.g. stable, latest). 
container.image.digest        the container image registry digest (e.g. 
                              sha256:d977378f890d445c15e51795296e4e5062f109ce6da83e0a355fc4ad8699d27). 
container.healthcheck         The container's health check. Will be the null value ("N/A") if no healthcheck 
                              configured, "NONE" if configured but explicitly not created, and the healthcheck command 
                              line otherwise 
container.liveness_probe      The container's liveness probe. Will be the null value ("N/A") if no liveness probe 
                              configured, the liveness probe command line otherwise 
container.readiness_probe     The container's readiness probe. Will be the null value ("N/A") if no readiness probe 
                              configured, the readiness probe command line otherwise 

-------------------------------
Field Class:                  fd
Description:                  Every syscall that has a file descriptor in its arguments has these fields set with 
                              information related to the file. 

fd.num                        the unique number identifying the file descriptor. 
fd.type                       type of FD. Can be 'file', 'directory', 'ipv4', 'ipv6', 'unix', 'pipe', 'event', 
                              'signalfd', 'eventpoll', 'inotify' or 'signalfd'. 
fd.typechar                   type of FD as a single character. Can be 'f' for file, 4 for IPv4 socket, 6 for IPv6 
                              socket, 'u' for unix socket, p for pipe, 'e' for eventfd, 's' for signalfd, 'l' for 
                              eventpoll, 'i' for inotify, 'o' for unknown. 
fd.name                       FD full name. If the fd is a file, this field contains the full path. If the FD is a 
                              socket, this field contain the connection tuple. 
fd.directory                  If the fd is a file, the directory that contains it. 
fd.filename                   If the fd is a file, the filename without the path. 
fd.ip                         (FILTER ONLY) matches the ip address (client or server) of the fd. 
fd.cip                        client IP address. 
fd.sip                        server IP address. 
fd.lip                        local IP address. 
fd.rip                        remote IP address. 
fd.port                       (FILTER ONLY) matches the port (either client or server) of the fd. 
fd.cport                      for TCP/UDP FDs, the client port. 
fd.sport                      for TCP/UDP FDs, server port. 
fd.lport                      for TCP/UDP FDs, the local port. 
fd.rport                      for TCP/UDP FDs, the remote port. 
fd.l4proto                    the IP protocol of a socket. Can be 'tcp', 'udp', 'icmp' or 'raw'. 
fd.sockfamily                 the socket family for socket events. Can be 'ip' or 'unix'. 
fd.is_server                  'true' if the process owning this FD is the server endpoint in the connection. 
fd.uid                        a unique identifier for the FD, created by chaining the FD number and the thread ID. 
fd.containername              chaining of the container ID and the FD name. Useful when trying to identify which 
                              container an FD belongs to. 
fd.containerdirectory         chaining of the container ID and the directory name. Useful when trying to identify which 
                              container a directory belongs to. 
fd.proto                      (FILTER ONLY) matches the protocol (either client or server) of the fd. 
fd.cproto                     for TCP/UDP FDs, the client protocol. 
fd.sproto                     for TCP/UDP FDs, server protocol. 
fd.lproto                     for TCP/UDP FDs, the local protocol. 
fd.rproto                     for TCP/UDP FDs, the remote protocol. 
fd.net                        (FILTER ONLY) matches the IP network (client or server) of the fd. 
fd.cnet                       (FILTER ONLY) matches the client IP network of the fd. 
fd.snet                       (FILTER ONLY) matches the server IP network of the fd. 
fd.lnet                       (FILTER ONLY) matches the local IP network of the fd. 
fd.rnet                       (FILTER ONLY) matches the remote IP network of the fd. 
fd.connected                  for TCP/UDP FDs, 'true' if the socket is connected. 
fd.name_changed               True when an event changes the name of an fd used by this event. This can occur in some 
                              cases such as udp connections where the connection tuple changes. 
fd.cip.name                   Domain name associated with the client IP address. 
fd.sip.name                   Domain name associated with the server IP address. 
fd.lip.name                   Domain name associated with the local IP address. 
fd.rip.name                   Domain name associated with the remote IP address. 
fd.dev                        device number (major/minor) containing the referenced file 
fd.dev.major                  major device number containing the referenced file 
fd.dev.minor                  minor device number containing the referenced file 

-------------------------------
Field Class:                  syslog
Description:                  Content of Syslog messages. 

syslog.facility.str           facility as a string. 
syslog.facility               facility as a number (0-23). 
syslog.severity.str           severity as a string. Can have one of these values: emerg, alert, crit, err, warn, 
                              notice, info, debug 
syslog.severity               severity as a number (0-7). 
syslog.message                message sent to syslog. 

-------------------------------
Field Class:                  fdlist
Description:                  Poll event related fields. 

fdlist.nums                   for poll events, this is a comma-separated list of the FD numbers in the 'fds' argument, 
                              returned as a string. 
fdlist.names                  for poll events, this is a comma-separated list of the FD names in the 'fds' argument, 
                              returned as a string. 
fdlist.cips                   for poll events, this is a comma-separated list of the client IP addresses in the 'fds' 
                              argument, returned as a string. 
fdlist.sips                   for poll events, this is a comma-separated list of the server IP addresses in the 'fds' 
                              argument, returned as a string. 
fdlist.cports                 for TCP/UDP FDs, for poll events, this is a comma-separated list of the client TCP/UDP 
                              ports in the 'fds' argument, returned as a string. 
fdlist.sports                 for poll events, this is a comma-separated list of the server TCP/UDP ports in the 'fds' 
                              argument, returned as a string. 

-------------------------------
Field Class:                  k8s
Description:                  Kubernetes related context. Available when configured to fetch k8s meta-data from API 
                              Server. 

k8s.pod.name                  Kubernetes pod name. 
k8s.pod.id                    Kubernetes pod id. 
k8s.pod.label                 Kubernetes pod label. E.g. 'k8s.pod.label.foo'. 
k8s.pod.labels                Kubernetes pod comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
k8s.rc.name                   Kubernetes replication controller name. 
k8s.rc.id                     Kubernetes replication controller id. 
k8s.rc.label                  Kubernetes replication controller label. E.g. 'k8s.rc.label.foo'. 
k8s.rc.labels                 Kubernetes replication controller comma-separated key/value labels. E.g. 
                              'foo1:bar1,foo2:bar2'. 
k8s.svc.name                  Kubernetes service name (can return more than one value, concatenated). 
k8s.svc.id                    Kubernetes service id (can return more than one value, concatenated). 
k8s.svc.label                 Kubernetes service label. E.g. 'k8s.svc.label.foo' (can return more than one value, 
                              concatenated). 
k8s.svc.labels                Kubernetes service comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
k8s.ns.name                   Kubernetes namespace name. 
k8s.ns.id                     Kubernetes namespace id. 
k8s.ns.label                  Kubernetes namespace label. E.g. 'k8s.ns.label.foo'. 
k8s.ns.labels                 Kubernetes namespace comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
k8s.rs.name                   Kubernetes replica set name. 
k8s.rs.id                     Kubernetes replica set id. 
k8s.rs.label                  Kubernetes replica set label. E.g. 'k8s.rs.label.foo'. 
k8s.rs.labels                 Kubernetes replica set comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
k8s.deployment.name           Kubernetes deployment name. 
k8s.deployment.id             Kubernetes deployment id. 
k8s.deployment.label          Kubernetes deployment label. E.g. 'k8s.rs.label.foo'. 
k8s.deployment.labels         Kubernetes deployment comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 

-------------------------------
Field Class:                  mesos
Description:                  Mesos related context. 

mesos.task.name               Mesos task name. 
mesos.task.id                 Mesos task id. 
mesos.task.label              Mesos task label. E.g. 'mesos.task.label.foo'. 
mesos.task.labels             Mesos task comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
mesos.framework.name          Mesos framework name. 
mesos.framework.id            Mesos framework id. 
marathon.app.name             Marathon app name. 
marathon.app.id               Marathon app id. 
marathon.app.label            Marathon app label. E.g. 'marathon.app.label.foo'. 
marathon.app.labels           Marathon app comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'. 
marathon.group.name           Marathon group name. 
marathon.group.id             Marathon group id. 

-------------------------------
Field Class:                  span
Description:                  Fields used if information about distributed tracing is available. 

span.id                       ID of the span. This is a unique identifier that is used to match the enter and exit 
                              tracer events for this span. It can also be used to match different spans belonging to a 
                              trace. 
span.time                     time of the span's enter tracer as a human readable string that includes the nanosecond 
                              part. 
span.ntags                    number of tags that this span has. 
span.nargs                    number of arguments that this span has. 
span.tags                     dot-separated list of all of the span's tags. 
span.tag                      one of the span's tags, specified by 0-based offset, e.g. 'span.tag[1]'. You can use a 
                              negative offset to pick elements from the end of the tag list. For example, 
                              'span.tag[-1]' returns the last tag. 
span.args                     comma-separated list of the span's arguments. 
span.arg                      one of the span arguments, specified by name or by 0-based offset. E.g. 'span.arg.xxx' or 
                              'span.arg[1]'. You can use a negative offset to pick elements from the end of the tag 
                              list. For example, 'span.arg[-1]' returns the last argument. 
span.enterargs                comma-separated list of the span's enter tracer event arguments. For enter tracers, this 
                              is the same as evt.args. For exit tracers, this is the evt.args of the corresponding 
                              enter tracer. 
span.enterarg                 one of the span's enter arguments, specified by name or by 0-based offset. For enter 
                              tracer events, this is the same as evt.arg. For exit tracer events, this is the evt.arg 
                              of the corresponding enter event. 
span.duration                 delta between this span's exit tracer event and the enter tracer event. 
span.duration.human           delta between this span's exit tracer event and the enter event, as a human readable 
                              string (e.g. 10.3ms). 

-------------------------------
Field Class:                  evtin
Description:                  Fields used if information about distributed tracing is available. 

evtin.span.id                 accepts all the events that are between the enter and exit tracers of the spans with the 
                              given ID and are generated by the same thread that generated the tracers. 
evtin.span.ntags              accepts all the events that are between the enter and exit tracers of the spans with the 
                              given number of tags and are generated by the same thread that generated the tracers. 
evtin.span.nargs              accepts all the events that are between the enter and exit tracers of the spans with the 
                              given number of arguments and are generated by the same thread that generated the 
                              tracers. 
evtin.span.tags               accepts all the events that are between the enter and exit tracers of the spans with the 
                              given tags and are generated by the same thread that generated the tracers. 
evtin.span.tag                accepts all the events that are between the enter and exit tracers of the spans with the 
                              given tag and are generated by the same thread that generated the tracers. See the 
                              description of span.tag for information about the syntax accepted by this field. 
evtin.span.args               accepts all the events that are between the enter and exit tracers of the spans with the 
                              given arguments and are generated by the same thread that generated the tracers. 
evtin.span.arg                accepts all the events that are between the enter and exit tracers of the spans with the 
                              given argument and are generated by the same thread that generated the tracers. See the 
                              description of span.arg for information about the syntax accepted by this field. 
evtin.span.p.id               same as evtin.span.id, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.ntags            same as evtin.span.ntags, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.nargs            same as evtin.span.nargs, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.tags             same as evtin.span.tags, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.tag              same as evtin.span.tag, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.args             same as evtin.span.args, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.p.arg              same as evtin.span.arg, but also accepts events generated by other threads in the same 
                              process that produced the span. 
evtin.span.s.id               same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.ntags            same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.nargs            same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.tags             same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.tag              same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.args             same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.s.arg              same as evtin.span.id, but also accepts events generated by the script that produced the 
                              span, i.e. by the processes whose parent PID is the same as the one of the process 
                              generating the span. 
evtin.span.m.id               same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.ntags            same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.nargs            same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.tags             same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.tag              same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.args             same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 
evtin.span.m.arg              same as evtin.span.id, but accepts all the events generated on the machine during the 
                              span, including other threads and other processes. 

For example, to print only SSH related events;

sysdig proc.name=sshd

And press ENTER.

It is also possible to use comparison operators: =, !=, <, <=, >, >=, contains, icontains, in, exists. For example;

sysdig fd.name contains etc

Process must exist;

sysdig proc.name exists

You can also filter multiple events using the boolean operators: and, or, not;

sysdig "not (fd.name contains /proc or fd.name contains /dev)"

Sysdig Chisels

Sysdig also comes bundled with lua scripts that called chisels. These scripts can analyze the sysdig event stream to perform useful actions. For example, monitor user activity, monitor specific IP addresses etc.

To list Sysdig chisels;

sysdig -cl
Category: Application
---------------------
httplog         HTTP requests log
httptop         Top HTTP requests
memcachelog     memcached requests log

Category: CPU Usage
-------------------
spectrogram     Visualize OS latency in real time.
subsecoffset    Visualize subsecond offset execution time.
topcontainers_cpu
                Top containers by CPU usage
topprocs_cpu    Top processes by CPU usage

Category: Errors
----------------
topcontainers_error
                Top containers by number of errors
topfiles_errors Top files by number of errors
topprocs_errors top processes by number of errors

Category: I/O
-------------
echo_fds        Print the data read and written by processes.
fdbytes_by      I/O bytes, aggregated by an arbitrary filter field
fdcount_by      FD count, aggregated by an arbitrary filter field
fdtime_by       FD time group by
iobytes         Sum of I/O bytes on any type of FD
iobytes_file    Sum of file I/O bytes
spy_file        Echo any read/write made by any process to all files. Optionall
                y, you can provide the name of one file to only intercept reads
                /writes to that file.
stderr          Print stderr of processes
stdin           Print stdin of processes
stdout          Print stdout of processes
topcontainers_file
                Top containers by R+W disk bytes
topfiles_bytes  Top files by R+W bytes
topfiles_time   Top files by time
topprocs_file   Top processes by R+W disk bytes
udp_extract     extract data from UDP streams to files.

Category: Logs
--------------
spy_logs        Echo any write made by any process to a log file. Optionally, e
                xport the events around each log message to file.
spy_syslog      Print every message written to syslog. Optionally, export the e
                vents around each syslog message to file.

Category: Misc
--------------
around          Export to file the events around the time range where the given
                 filter matches.

Category: Net
-------------
iobytes_net     Show total network I/O bytes
spy_ip          Show the data exchanged with the given IP address
spy_port        Show the data exchanged using the given IP port number
topconns        Top network connections by total bytes
topcontainers_net
                Top containers by network I/O
topports_server Top TCP/UDP server ports by R+W bytes
topprocs_net    Top processes by network I/O

Category: Performance
---------------------
bottlenecks     Slowest system calls
fileslower      Trace slow file I/O
netlower        Trace slow network I/0
proc_exec_time  Show process execution time
scallslower     Trace slow syscalls
topscalls       Top system calls by number of calls
topscalls_time  Top system calls by time

Category: Security
------------------
list_login_shells
                List the login shell IDs
shellshock_detect
                print shellshock attacks
spy_users       Display interactive user activity

Category: System State
----------------------
lscontainers    List the running containers
lsof            List (and optionally filter) the open file descriptors.
netstat         List (and optionally filter) network connections.
ps              List (and optionally filter) the machine processes.

Category: Tracers
-----------------
tracers_2_statsd
                Export spans duration as statds metrics.

Use the -i flag to get detailed information about a specific chisel

For example, to list running processes;

sysdig -c ps

To get top processes by CPU usage;

sysdig -c topprocs_cpu

Show the data exchanged with the given IP address;

sysdig -c spy_ip 192.168.100.1

Display interactive user activity;

sysdig -c spy_users

And many more.

Csysdig: Intuitive sysdig UI tool

Sysdig ships with an intuitive UI tool called Csysdig. It works in a similar way like top/htop command.

You can simply launch Csysdig from command line.

csysdig
install Sysdig system visibility tool on Ubuntu

Csysdig has different views. Press Fn+F12.

install Sysdig system visibility tool on Ubuntu

You can scroll up/down the views using arrow keys to select a specific view. Press Enter to display the view.

Sample spy_users view.

csysdig spy users

Awesome, isn’t it?

That brings us to a close of our tutorial on how to install Sysdig system visibility tool on Ubuntu 22.04.

Further Reading

Sysdig User Guide

Other Tutorials

Install ntopng on Rocky Linux 8

How to copy paste lines in vim

Example Usage of ps Command in Linux

LEAVE A REPLY

Please enter your comment!
Please enter your name here