This tutorial will take you through how to install Sysdig system visibility tool on Ubuntu 22.04. Sysdig is a simple visibility tool that provides deep visibility into your system. According to sysdig man pages;
sysdig is a tool for system troubleshooting, analysis and exploration.
It can be used to capture, filter and decode system calls and other OS events.
It can be both used to inspect live systems, or to generate trace files that can be analyzed at a later stage.
sysdig includes a powerful filtering language, has customizable output, and can be extended through Lua scripts, called chisels.
Install Sysdig System Visibility Tool on Ubuntu 22.04
In this tutorial, we will be installing Sysdig on an Ubuntu 22.04 system.
Sysdig is provided by the default Ubuntu Universe repositories. However, the available version is a bit dated.
To automatically install Sysdig on Ubuntu, all you have to do is to download the installation script and execute it as follows. The script will basically install Sysdig Draios APT repository on your system and install Sysdig and other required packages via that repo.
The output of sysdig can be filtered using various event fields.
You can check available sysdig event fields using the command;
sysdig -l
Sample output;
-------------------------------
Field Class: evt (All event types)
Description: These fields can be used for all event types
evt.num event number.
evt.time event timestamp as a time string that includes the nanosecond part.
evt.time.s event timestamp as a time string with no nanoseconds.
evt.time.iso8601 event timestamp in ISO 8601 format, including nanoseconds and time zone offset (in UTC).
evt.datetime event timestamp as a time string that includes the date.
evt.datetime.s event timestamp as a datetime string with no nanoseconds.
evt.rawtime absolute event timestamp, i.e. nanoseconds from epoch.
evt.rawtime.s integer part of the event timestamp (e.g. seconds since epoch).
evt.rawtime.ns fractional part of the absolute event timestamp.
evt.reltime number of nanoseconds from the beginning of the capture.
evt.reltime.s number of seconds from the beginning of the capture.
evt.reltime.ns fractional part (in ns) of the time from the beginning of the capture.
evt.pluginname if the event comes from a plugin, the name of the plugin that generated it. The plugin
must be currently loaded.
evt.plugininfo if the event comes from a plugin, a summary of the event as formatted by the plugin. The
plugin must be currently loaded.
-------------------------------
Field Class: evt (Syscall events only)
Description: Event fields applicable to syscall events. Note that for most events you can access the
individual arguments/parameters of each syscall via evt.arg, e.g. evt.arg.filename.
evt.latency delta between an exit event and the correspondent enter event, in nanoseconds.
evt.latency.s integer part of the event latency delta.
evt.latency.ns fractional part of the event latency delta.
evt.latency.human delta between an exit event and the correspondent enter event, as a human readable string
(e.g. 10.3ms).
evt.deltatime delta between this event and the previous event, in nanoseconds.
evt.deltatime.s integer part of the delta between this event and the previous event.
evt.deltatime.ns fractional part of the delta between this event and the previous event.
evt.dir event direction can be either '>' for enter events or '<' for exit events.
evt.type The name of the event (e.g. 'open').
evt.type.is allows one to specify an event type, and returns 1 for events that are of that type. For
example, evt.type.is.open returns 1 for open events, 0 for any other event.
syscall.type For system call events, the name of the system call (e.g. 'open'). Unset for other events
(e.g. switch or internal events). Use this field instead of evt.type if you need to make
sure that the filtered/printed value is actually a system call.
evt.category The event category. Example values are 'file' (for file operations like open and close),
'net' (for network operations like socket and bind), memory (for things like brk or
mmap), and so on.
evt.cpu number of the CPU where this event happened.
evt.args all the event arguments, aggregated into a single string.
evt.arg one of the event arguments specified by name or by number. Some events (e.g. return codes
or FDs) will be converted into a text representation when possible. E.g. 'evt.arg.fd' or
'evt.arg[0]'.
evt.rawarg one of the event arguments specified by name. E.g. 'evt.rawarg.fd'.
evt.info for most events, this field returns the same value as evt.args. However, for some events
(like writes to /dev/log) it provides higher level information coming from decoding the
arguments.
evt.buffer the binary data buffer for events that have one, like read(), recvfrom(), etc. Use this
field in filters with 'contains' to search into I/O data buffers.
evt.buflen the length of the binary data buffer for events that have one, like read(), recvfrom(),
etc.
evt.res event return value, as a string. If the event failed, the result is an error code string
(e.g. 'ENOENT'), otherwise the result is the string 'SUCCESS'.
evt.rawres event return value, as a number (e.g. -2). Useful for range comparisons.
evt.failed 'true' for events that returned an error status.
evt.is_io 'true' for events that read or write to FDs, like read(), send, recvfrom(), etc.
evt.is_io_read 'true' for events that read from FDs, like read(), recv(), recvfrom(), etc.
evt.is_io_write 'true' for events that write to FDs, like write(), send(), etc.
evt.io_dir 'r' for events that read from FDs, like read(); 'w' for events that write to FDs, like
write().
evt.is_wait 'true' for events that make the thread wait, e.g. sleep(), select(), poll().
evt.wait_latency for events that make the thread wait (e.g. sleep(), select(), poll()), this is the time
spent waiting for the event to return, in nanoseconds.
evt.is_syslog 'true' for events that are writes to /dev/log.
evt.count This filter field always returns 1 and can be used to count events from inside chisels.
evt.count.error This filter field returns 1 for events that returned with an error, and can be used to
count event failures from inside chisels.
evt.count.error.file This filter field returns 1 for events that returned with an error and are related to
file I/O, and can be used to count event failures from inside chisels.
evt.count.error.net This filter field returns 1 for events that returned with an error and are related to
network I/O, and can be used to count event failures from inside chisels.
evt.count.error.memory This filter field returns 1 for events that returned with an error and are related to
memory allocation, and can be used to count event failures from inside chisels.
evt.count.error.other This filter field returns 1 for events that returned with an error and are related to
none of the previous categories, and can be used to count event failures from inside
chisels.
evt.count.exit This filter field returns 1 for exit events, and can be used to count single events from
inside chisels.
evt.around (FILTER ONLY) Accepts the event if it's around the specified time interval. The syntax is
evt.around[T]=D, where T is the value returned by %evt.rawtime for the event and D is a
delta in milliseconds. For example, evt.around[1404996934793590564]=1000 will return the
events with timestamp with one second before the timestamp and one second after it, for a
total of two seconds of capture.
evt.abspath Absolute path calculated from dirfd and name during syscalls like renameat and symlinkat.
Use 'evt.abspath.src' or 'evt.abspath.dst' for syscalls that support multiple paths.
evt.is_open_read 'true' for open/openat/openat2 events where the path was opened for reading
evt.is_open_write 'true' for open/openat/openat2 events where the path was opened for writing
evt.is_open_exec 'true' for open/openat/openat2 or creat events where a file is created with execute
permissions
-------------------------------
Field Class: process
Description: Additional information about the process and thread executing the syscall event.
proc.pid the id of the process generating the event.
proc.exe the first command line argument (usually the executable name or a custom one).
proc.name the name (excluding the path) of the executable generating the event.
proc.args the arguments passed on the command line when starting the process generating the event.
proc.env the environment variables of the process generating the event.
proc.cmdline full process command line, i.e. proc.name + proc.args.
proc.exeline full process command line, with exe as first argument, i.e. proc.exe + proc.args.
proc.cwd the current working directory of the event.
proc.nthreads the number of threads that the process generating the event currently has, including the
main process thread.
proc.nchilds the number of child threads that the process generating the event currently has. This
excludes the main process thread.
proc.ppid the pid of the parent of the process generating the event.
proc.pname the name (excluding the path) of the parent of the process generating the event.
proc.pcmdline the full command line (proc.name + proc.args) of the parent of the process generating the
event.
proc.apid the pid of one of the process ancestors. E.g. proc.apid[1] returns the parent pid,
proc.apid[2] returns the grandparent pid, and so on. proc.apid[0] is the pid of the
current process. proc.apid without arguments can be used in filters only and matches any
of the process ancestors, e.g. proc.apid=1234.
proc.aname the name (excluding the path) of one of the process ancestors. E.g. proc.aname[1] returns
the parent name, proc.aname[2] returns the grandparent name, and so on. proc.aname[0] is
the name of the current process. proc.aname without arguments can be used in filters only
and matches any of the process ancestors, e.g. proc.aname=bash.
proc.loginshellid the pid of the oldest shell among the ancestors of the current process, if there is one.
This field can be used to separate different user sessions, and is useful in conjunction
with chisels like spy_user.
proc.duration number of nanoseconds since the process started.
proc.fdopencount number of open FDs for the process
proc.fdlimit maximum number of FDs the process can open.
proc.fdusage the ratio between open FDs and maximum available FDs for the process.
proc.vmsize total virtual memory for the process (as kb).
proc.vmrss resident non-swapped memory for the process (as kb).
proc.vmswap swapped memory for the process (as kb).
thread.pfmajor number of major page faults since thread start.
thread.pfminor number of minor page faults since thread start.
thread.tid the id of the thread generating the event.
thread.ismain 'true' if the thread generating the event is the main one in the process.
thread.exectime CPU time spent by the last scheduled thread, in nanoseconds. Exported by switch events
only.
thread.totexectime Total CPU time, in nanoseconds since the beginning of the capture, for the current
thread. Exported by switch events only.
thread.cgroups all the cgroups the thread belongs to, aggregated into a single string.
thread.cgroup the cgroup the thread belongs to, for a specific subsystem. E.g. thread.cgroup.cpuacct.
thread.vtid the id of the thread generating the event as seen from its current PID namespace.
proc.vpid the id of the process generating the event as seen from its current PID namespace.
thread.cpu the CPU consumed by the thread in the last second.
thread.cpu.user the user CPU consumed by the thread in the last second.
thread.cpu.system the system CPU consumed by the thread in the last second.
thread.vmsize For the process main thread, this is the total virtual memory for the process (as kb).
For the other threads, this field is zero.
thread.vmrss For the process main thread, this is the resident non-swapped memory for the process (as
kb). For the other threads, this field is zero.
proc.sid the session id of the process generating the event.
proc.sname the name of the current process's session leader. This is either the process with
pid=proc.sid or the eldest ancestor that has the same sid as the current process.
proc.tty The controlling terminal of the process. 0 for processes without a terminal.
proc.exepath The full executable path of the process.
proc.vpgid the process group id of the process generating the event, as seen from its current PID
namespace.
proc.is_container_healthcheck true if this process is running as a part of the container's health check.
proc.is_container_liveness_probe
true if this process is running as a part of the container's liveness probe.
proc.is_container_readiness_probe
true if this process is running as a part of the container's readiness probe.
proc.is_exe_writable true if this process' executable file is writable by the same user that spawned the
process.
-------------------------------
Field Class: user
Description: Information about the user executing the specific event.
user.uid user ID.
user.name user name.
user.homedir home directory of the user.
user.shell user's shell.
user.loginuid audit user id (auid).
user.loginname audit user name (auid).
-------------------------------
Field Class: group
Description: Information about the user group.
group.gid group ID.
group.name group name.
-------------------------------
Field Class: container
Description: Container information. If the event is not happening inside a container, both id and name
will be set to 'host'.
container.id the container id.
container.name the container name.
container.image the container image name (e.g. falcosecurity/falco:latest for docker).
container.image.id the container image id (e.g. 6f7e2741b66b).
container.type the container type, eg: docker or rkt
container.privileged true for containers running as privileged, false otherwise
container.mounts A space-separated list of mount information. Each item in the list has the format
::::
container.mount Information about a single mount, specified by number (e.g. container.mount[0]) or mount
source (container.mount[/usr/local]). The pathname can be a glob
(container.mount[/usr/local/*]), in which case the first matching mount will be returned.
The information has the format ::::. If there is
no mount with the specified index or matching the provided source, returns the string
"none" instead of a NULL value.
container.mount.source the mount source, specified by number (e.g. container.mount.source[0]) or mount
destination (container.mount.source[/host/lib/modules]). The pathname can be a glob.
container.mount.dest the mount destination, specified by number (e.g. container.mount.dest[0]) or mount source
(container.mount.dest[/lib/modules]). The pathname can be a glob.
container.mount.mode the mount mode, specified by number (e.g. container.mount.mode[0]) or mount source
(container.mount.mode[/usr/local]). The pathname can be a glob.
container.mount.rdwr the mount rdwr value, specified by number (e.g. container.mount.rdwr[0]) or mount source
(container.mount.rdwr[/usr/local]). The pathname can be a glob.
container.mount.propagation the mount propagation value, specified by number (e.g. container.mount.propagation[0]) or
mount source (container.mount.propagation[/usr/local]). The pathname can be a glob.
container.image.repository the container image repository (e.g. falcosecurity/falco).
container.image.tag the container image tag (e.g. stable, latest).
container.image.digest the container image registry digest (e.g.
sha256:d977378f890d445c15e51795296e4e5062f109ce6da83e0a355fc4ad8699d27).
container.healthcheck The container's health check. Will be the null value ("N/A") if no healthcheck
configured, "NONE" if configured but explicitly not created, and the healthcheck command
line otherwise
container.liveness_probe The container's liveness probe. Will be the null value ("N/A") if no liveness probe
configured, the liveness probe command line otherwise
container.readiness_probe The container's readiness probe. Will be the null value ("N/A") if no readiness probe
configured, the readiness probe command line otherwise
-------------------------------
Field Class: fd
Description: Every syscall that has a file descriptor in its arguments has these fields set with
information related to the file.
fd.num the unique number identifying the file descriptor.
fd.type type of FD. Can be 'file', 'directory', 'ipv4', 'ipv6', 'unix', 'pipe', 'event',
'signalfd', 'eventpoll', 'inotify' or 'signalfd'.
fd.typechar type of FD as a single character. Can be 'f' for file, 4 for IPv4 socket, 6 for IPv6
socket, 'u' for unix socket, p for pipe, 'e' for eventfd, 's' for signalfd, 'l' for
eventpoll, 'i' for inotify, 'o' for unknown.
fd.name FD full name. If the fd is a file, this field contains the full path. If the FD is a
socket, this field contain the connection tuple.
fd.directory If the fd is a file, the directory that contains it.
fd.filename If the fd is a file, the filename without the path.
fd.ip (FILTER ONLY) matches the ip address (client or server) of the fd.
fd.cip client IP address.
fd.sip server IP address.
fd.lip local IP address.
fd.rip remote IP address.
fd.port (FILTER ONLY) matches the port (either client or server) of the fd.
fd.cport for TCP/UDP FDs, the client port.
fd.sport for TCP/UDP FDs, server port.
fd.lport for TCP/UDP FDs, the local port.
fd.rport for TCP/UDP FDs, the remote port.
fd.l4proto the IP protocol of a socket. Can be 'tcp', 'udp', 'icmp' or 'raw'.
fd.sockfamily the socket family for socket events. Can be 'ip' or 'unix'.
fd.is_server 'true' if the process owning this FD is the server endpoint in the connection.
fd.uid a unique identifier for the FD, created by chaining the FD number and the thread ID.
fd.containername chaining of the container ID and the FD name. Useful when trying to identify which
container an FD belongs to.
fd.containerdirectory chaining of the container ID and the directory name. Useful when trying to identify which
container a directory belongs to.
fd.proto (FILTER ONLY) matches the protocol (either client or server) of the fd.
fd.cproto for TCP/UDP FDs, the client protocol.
fd.sproto for TCP/UDP FDs, server protocol.
fd.lproto for TCP/UDP FDs, the local protocol.
fd.rproto for TCP/UDP FDs, the remote protocol.
fd.net (FILTER ONLY) matches the IP network (client or server) of the fd.
fd.cnet (FILTER ONLY) matches the client IP network of the fd.
fd.snet (FILTER ONLY) matches the server IP network of the fd.
fd.lnet (FILTER ONLY) matches the local IP network of the fd.
fd.rnet (FILTER ONLY) matches the remote IP network of the fd.
fd.connected for TCP/UDP FDs, 'true' if the socket is connected.
fd.name_changed True when an event changes the name of an fd used by this event. This can occur in some
cases such as udp connections where the connection tuple changes.
fd.cip.name Domain name associated with the client IP address.
fd.sip.name Domain name associated with the server IP address.
fd.lip.name Domain name associated with the local IP address.
fd.rip.name Domain name associated with the remote IP address.
fd.dev device number (major/minor) containing the referenced file
fd.dev.major major device number containing the referenced file
fd.dev.minor minor device number containing the referenced file
-------------------------------
Field Class: syslog
Description: Content of Syslog messages.
syslog.facility.str facility as a string.
syslog.facility facility as a number (0-23).
syslog.severity.str severity as a string. Can have one of these values: emerg, alert, crit, err, warn,
notice, info, debug
syslog.severity severity as a number (0-7).
syslog.message message sent to syslog.
-------------------------------
Field Class: fdlist
Description: Poll event related fields.
fdlist.nums for poll events, this is a comma-separated list of the FD numbers in the 'fds' argument,
returned as a string.
fdlist.names for poll events, this is a comma-separated list of the FD names in the 'fds' argument,
returned as a string.
fdlist.cips for poll events, this is a comma-separated list of the client IP addresses in the 'fds'
argument, returned as a string.
fdlist.sips for poll events, this is a comma-separated list of the server IP addresses in the 'fds'
argument, returned as a string.
fdlist.cports for TCP/UDP FDs, for poll events, this is a comma-separated list of the client TCP/UDP
ports in the 'fds' argument, returned as a string.
fdlist.sports for poll events, this is a comma-separated list of the server TCP/UDP ports in the 'fds'
argument, returned as a string.
-------------------------------
Field Class: k8s
Description: Kubernetes related context. Available when configured to fetch k8s meta-data from API
Server.
k8s.pod.name Kubernetes pod name.
k8s.pod.id Kubernetes pod id.
k8s.pod.label Kubernetes pod label. E.g. 'k8s.pod.label.foo'.
k8s.pod.labels Kubernetes pod comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
k8s.rc.name Kubernetes replication controller name.
k8s.rc.id Kubernetes replication controller id.
k8s.rc.label Kubernetes replication controller label. E.g. 'k8s.rc.label.foo'.
k8s.rc.labels Kubernetes replication controller comma-separated key/value labels. E.g.
'foo1:bar1,foo2:bar2'.
k8s.svc.name Kubernetes service name (can return more than one value, concatenated).
k8s.svc.id Kubernetes service id (can return more than one value, concatenated).
k8s.svc.label Kubernetes service label. E.g. 'k8s.svc.label.foo' (can return more than one value,
concatenated).
k8s.svc.labels Kubernetes service comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
k8s.ns.name Kubernetes namespace name.
k8s.ns.id Kubernetes namespace id.
k8s.ns.label Kubernetes namespace label. E.g. 'k8s.ns.label.foo'.
k8s.ns.labels Kubernetes namespace comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
k8s.rs.name Kubernetes replica set name.
k8s.rs.id Kubernetes replica set id.
k8s.rs.label Kubernetes replica set label. E.g. 'k8s.rs.label.foo'.
k8s.rs.labels Kubernetes replica set comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
k8s.deployment.name Kubernetes deployment name.
k8s.deployment.id Kubernetes deployment id.
k8s.deployment.label Kubernetes deployment label. E.g. 'k8s.rs.label.foo'.
k8s.deployment.labels Kubernetes deployment comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
-------------------------------
Field Class: mesos
Description: Mesos related context.
mesos.task.name Mesos task name.
mesos.task.id Mesos task id.
mesos.task.label Mesos task label. E.g. 'mesos.task.label.foo'.
mesos.task.labels Mesos task comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
mesos.framework.name Mesos framework name.
mesos.framework.id Mesos framework id.
marathon.app.name Marathon app name.
marathon.app.id Marathon app id.
marathon.app.label Marathon app label. E.g. 'marathon.app.label.foo'.
marathon.app.labels Marathon app comma-separated key/value labels. E.g. 'foo1:bar1,foo2:bar2'.
marathon.group.name Marathon group name.
marathon.group.id Marathon group id.
-------------------------------
Field Class: span
Description: Fields used if information about distributed tracing is available.
span.id ID of the span. This is a unique identifier that is used to match the enter and exit
tracer events for this span. It can also be used to match different spans belonging to a
trace.
span.time time of the span's enter tracer as a human readable string that includes the nanosecond
part.
span.ntags number of tags that this span has.
span.nargs number of arguments that this span has.
span.tags dot-separated list of all of the span's tags.
span.tag one of the span's tags, specified by 0-based offset, e.g. 'span.tag[1]'. You can use a
negative offset to pick elements from the end of the tag list. For example,
'span.tag[-1]' returns the last tag.
span.args comma-separated list of the span's arguments.
span.arg one of the span arguments, specified by name or by 0-based offset. E.g. 'span.arg.xxx' or
'span.arg[1]'. You can use a negative offset to pick elements from the end of the tag
list. For example, 'span.arg[-1]' returns the last argument.
span.enterargs comma-separated list of the span's enter tracer event arguments. For enter tracers, this
is the same as evt.args. For exit tracers, this is the evt.args of the corresponding
enter tracer.
span.enterarg one of the span's enter arguments, specified by name or by 0-based offset. For enter
tracer events, this is the same as evt.arg. For exit tracer events, this is the evt.arg
of the corresponding enter event.
span.duration delta between this span's exit tracer event and the enter tracer event.
span.duration.human delta between this span's exit tracer event and the enter event, as a human readable
string (e.g. 10.3ms).
-------------------------------
Field Class: evtin
Description: Fields used if information about distributed tracing is available.
evtin.span.id accepts all the events that are between the enter and exit tracers of the spans with the
given ID and are generated by the same thread that generated the tracers.
evtin.span.ntags accepts all the events that are between the enter and exit tracers of the spans with the
given number of tags and are generated by the same thread that generated the tracers.
evtin.span.nargs accepts all the events that are between the enter and exit tracers of the spans with the
given number of arguments and are generated by the same thread that generated the
tracers.
evtin.span.tags accepts all the events that are between the enter and exit tracers of the spans with the
given tags and are generated by the same thread that generated the tracers.
evtin.span.tag accepts all the events that are between the enter and exit tracers of the spans with the
given tag and are generated by the same thread that generated the tracers. See the
description of span.tag for information about the syntax accepted by this field.
evtin.span.args accepts all the events that are between the enter and exit tracers of the spans with the
given arguments and are generated by the same thread that generated the tracers.
evtin.span.arg accepts all the events that are between the enter and exit tracers of the spans with the
given argument and are generated by the same thread that generated the tracers. See the
description of span.arg for information about the syntax accepted by this field.
evtin.span.p.id same as evtin.span.id, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.ntags same as evtin.span.ntags, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.nargs same as evtin.span.nargs, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.tags same as evtin.span.tags, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.tag same as evtin.span.tag, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.args same as evtin.span.args, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.p.arg same as evtin.span.arg, but also accepts events generated by other threads in the same
process that produced the span.
evtin.span.s.id same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.ntags same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.nargs same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.tags same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.tag same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.args same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.s.arg same as evtin.span.id, but also accepts events generated by the script that produced the
span, i.e. by the processes whose parent PID is the same as the one of the process
generating the span.
evtin.span.m.id same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.ntags same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.nargs same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.tags same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.tag same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.args same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
evtin.span.m.arg same as evtin.span.id, but accepts all the events generated on the machine during the
span, including other threads and other processes.
For example, to print only SSH related events;
sysdig proc.name=sshd
And press ENTER.
It is also possible to use comparison operators: =, !=, <, <=, >, >=, contains, icontains, in, exists. For example;
sysdig fd.name contains etc
Process must exist;
sysdig proc.name exists
You can also filter multiple events using the boolean operators: and, or, not;
sysdig "not (fd.name contains /proc or fd.name contains /dev)"
Sysdig Chisels
Sysdig also comes bundled with lua scripts that called chisels. These scripts can analyze the sysdig event stream to perform useful actions. For example, monitor user activity, monitor specific IP addresses etc.
To list Sysdig chisels;
sysdig -cl
Category: Application
---------------------
httplog HTTP requests log
httptop Top HTTP requests
memcachelog memcached requests log
Category: CPU Usage
-------------------
spectrogram Visualize OS latency in real time.
subsecoffset Visualize subsecond offset execution time.
topcontainers_cpu
Top containers by CPU usage
topprocs_cpu Top processes by CPU usage
Category: Errors
----------------
topcontainers_error
Top containers by number of errors
topfiles_errors Top files by number of errors
topprocs_errors top processes by number of errors
Category: I/O
-------------
echo_fds Print the data read and written by processes.
fdbytes_by I/O bytes, aggregated by an arbitrary filter field
fdcount_by FD count, aggregated by an arbitrary filter field
fdtime_by FD time group by
iobytes Sum of I/O bytes on any type of FD
iobytes_file Sum of file I/O bytes
spy_file Echo any read/write made by any process to all files. Optionall
y, you can provide the name of one file to only intercept reads
/writes to that file.
stderr Print stderr of processes
stdin Print stdin of processes
stdout Print stdout of processes
topcontainers_file
Top containers by R+W disk bytes
topfiles_bytes Top files by R+W bytes
topfiles_time Top files by time
topprocs_file Top processes by R+W disk bytes
udp_extract extract data from UDP streams to files.
Category: Logs
--------------
spy_logs Echo any write made by any process to a log file. Optionally, e
xport the events around each log message to file.
spy_syslog Print every message written to syslog. Optionally, export the e
vents around each syslog message to file.
Category: Misc
--------------
around Export to file the events around the time range where the given
filter matches.
Category: Net
-------------
iobytes_net Show total network I/O bytes
spy_ip Show the data exchanged with the given IP address
spy_port Show the data exchanged using the given IP port number
topconns Top network connections by total bytes
topcontainers_net
Top containers by network I/O
topports_server Top TCP/UDP server ports by R+W bytes
topprocs_net Top processes by network I/O
Category: Performance
---------------------
bottlenecks Slowest system calls
fileslower Trace slow file I/O
netlower Trace slow network I/0
proc_exec_time Show process execution time
scallslower Trace slow syscalls
topscalls Top system calls by number of calls
topscalls_time Top system calls by time
Category: Security
------------------
list_login_shells
List the login shell IDs
shellshock_detect
print shellshock attacks
spy_users Display interactive user activity
Category: System State
----------------------
lscontainers List the running containers
lsof List (and optionally filter) the open file descriptors.
netstat List (and optionally filter) network connections.
ps List (and optionally filter) the machine processes.
Category: Tracers
-----------------
tracers_2_statsd
Export spans duration as statds metrics.
Use the -i flag to get detailed information about a specific chisel
For example, to list running processes;
sysdig -c ps
To get top processes by CPU usage;
sysdig -c topprocs_cpu
Show the data exchanged with the given IP address;
sysdig -c spy_ip 192.168.100.1
Display interactive user activity;
sysdig -c spy_users
And many more.
Csysdig: Intuitive sysdig UI tool
Sysdig ships with an intuitive UI tool called Csysdig. It works in a similar way like top/htop command.
You can simply launch Csysdig from command line.
csysdig
Csysdig has different views. Press Fn+F12.
You can scroll up/down the views using arrow keys to select a specific view. Press Enter to display the view.
Sample spy_users view.
Awesome, isn’t it?
That brings us to a close of our tutorial on how to install Sysdig on Ubuntu 22.04.
We're passionate about sharing our knowledge and experiences with you through our blog. If you appreciate our efforts, consider buying us a virtual coffee. Your support keeps us motivated and enables us to continually improve, ensuring that we can provide you with the best content possible. Thank you for being a coffee-fueled champion of our work!
koromicha
I am the Co-founder of Kifarunix.com, Linux and the whole FOSS enthusiast, Linux System Admin and a Blue Teamer who loves to share technological tips and hacks with others as a way of sharing knowledge as:
"In vain have you acquired knowledge if you have not imparted it to others".
