In this guide, we are going to learn on installing Perf Performance Analysis Tool on CentOS 8. Perf aka perf_events aka perf tools is a powerful performance counters monitoring tool for Linux systems that gathers data about hardware events such as instructions executed, branches mispredicted or cache-misses suffered.
It also provides very low overhead profiling of applications to trace dynamic control flow and identify hotspots, per task, per CPU and per-workload counters, sampling and source code event annotation, dynamic creation of tracepoints using the kprobes and uprobes frameworks, for kernel and userspace dynamic tracing respectively.
Installing Perf Performance Analysis Tool on CentOS 8
Perf provide a userspace controlling utility called perf. You need to install this Perf userland utility in order to be able to use Perf on command line.
Perf is available on the default CentOS 8 repos through the perf package
dnf info perfAvailable Packages
Name : perf
Version : 4.18.0
Release : 147.5.1.el8_1
Architecture : x86_64
Size : 3.4 M
Source : kernel-4.18.0-147.5.1.el8_1.src.rpm
Repository : BaseOS
Summary : Performance monitoring for the Linux kernel
URL : http://www.kernel.org/
License : GPLv2
Description : This package contains the perf tool, which enables performance monitoring
: of the Linux kernel.Run system update.
dnf updateInstall Perf on CentOS 8
dnf install perfThe basic command line Syntax of perf utility is;
perf [--version] [--help] [OPTIONS] COMMAND [ARGS]The most common perf sub-commands include;
stat: runs a command and gathers performance counter statistics from it.top: generates and displays a performance counter profile in real time.record: runs a command and gathers a performance counter profile from it. The gathered data is saved intoperf.datawithout displaying anything.report: It readsperf.datato display the performance counter profile information recorded via perf record sub-command.list: displays the symbolic event types which can be selected in the various perf commands with the -e option.trace: Just like strace, perf trace sub-command shows various system calls being executed by a particular thread or process and the signals it receives.annotate: Perf annotate command displays a report file and an annotated version of the executed code.
If you just type perf command on the terminal, you should be able to see quite a number of subcommands;
perfperf
usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
The most commonly used perf commands are:
annotate Read perf.data (created by perf record) and display annotated code
archive Create archive with object files with build-ids found in perf.data file
bench General framework for benchmark suites
buildid-cache Manage build-id cache.
buildid-list List the buildids in a perf.data file
c2c Shared Data C2C/HITM Analyzer.
config Get and set variables in a configuration file.
data Data file related processing
diff Read perf.data files and display the differential profile
evlist List the event names in a perf.data file
ftrace simple wrapper for kernel's ftrace functionality
inject Filter to augment the events stream with additional information
kallsyms Searches running kernel for symbols
kmem Tool to trace/measure kernel memory properties
kvm Tool to trace/measure kvm guest os
list List all symbolic event types
lock Analyze lock events
mem Profile memory accesses
record Run a command and record its profile into perf.data
report Read perf.data (created by perf record) and display the profile
sched Tool to trace/measure scheduler properties (latencies)
script Read perf.data (created by perf record) and display trace output
stat Run a command and gather performance counter statistics
test Runs sanity tests.
timechart Tool to visualize total system behavior during a workload
top System profiling tool.
version display the version of perf binary
probe Define new dynamic tracepoints
trace strace inspired toolTo obtain help about a specific sub-command, just execute;
perf help SUB-COMMANDExample Usage of Perf on CentOS 8
Next, let us go over a few example usage of perf command on CentOS 8 and similar derivatives.
Gather Command Performance Counter statistics
The stat sub-command of perf utility enables you to run a command and gather its performance counter statistics
perf stat lscon file file1
Performance counter stats for 'ls':
1.72 msec task-clock # 0.827 CPUs utilized
0 context-switches # 0.000 K/sec
0 cpu-migrations # 0.000 K/sec
108 page-faults # 0.063 M/sec
<not supported> cycles
<not supported> instructions
<not supported> branches
<not supported> branch-misses
0.002075544 seconds time elapsed
0.002106000 seconds user
0.000000000 seconds sysRead more on man perf-stat.
Display Live Performance Counter Profile
perf-top is a system profiling tool used to generate and display a performance counter profile in real time. This can be achieved using the perf top command.
perf topSamples: 29 of event 'cpu-clock', 4000 Hz, Event count (approx.): 7250000 lost: 0/0 drop: 0/0
Overhead Shared Object Symbol
34.48% [kernel] [k] e1000_watchdog
13.79% [kernel] [k] _raw_spin_unlock_irqrestore
6.90% [kernel] [k] format_decode
6.90% [kernel] [k] vsnprintf
3.45% [kernel] [k] __x86_indirect_thunk_rax
3.45% [kernel] [k] __x86_indirect_thunk_rdx
3.45% [kernel] [k] kallsyms_expand_symbol.constprop.1
3.45% [kernel] [k] mem_cgroup_throttle_swaprate
3.45% [kernel] [k] memcpy
3.45% [kernel] [k] number
3.45% [kernel] [k] update_iter
3.45% libc-2.28.so [.] _IO_getdelim
3.45% libc-2.28.so [.] __memmove_avx_unaligned
3.45% perf [.] dso__next_symbol
3.45% perf [.] rb_next
...Consult man perf-top to see more usage.
List all symbolic event types
To list all symbolic event types using perf command;
perf list alignment-faults [Software event]
bpf-output [Software event]
context-switches OR cs [Software event]
cpu-clock [Software event]
...
msr/pperf/ [Kernel PMU event]
msr/smi/ [Kernel PMU event]
msr/tsc/ [Kernel PMU event]
...
rNNN [Raw hardware event descriptor]
cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor]
(see 'man perf-list' on how to encode it)
mem:<addr>[/len][:access] [Hardware breakpoint]
...Record Command Profile
To run a command and record its profile into perf.data;
perf record ls -ld /etcdrwxr-xr-x. 78 root root 8192 Mar 19 20:32 /etc
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.011 MB perf.data ]The basic syntax and a comprehensive description of using perf record command is stated on man perf-record.
perf record -e cpu-clock -a -g -- sleep 3[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.628 MB perf.data (4514 samples) ]Read perf.data and display the profile
To read the performance record, use the report sub-command. The command syntax is perf report [-i <file> | --input=file]
perf report -i perf.dataSamples: 4K of event 'cpu-clock', Event count (approx.): 1128500000
Children Self Command Shared Object Symbol
+ 99.87% 99.87% swapper [kernel.kallsyms] [k] mwait_idle
+ 99.87% 0.00% swapper [kernel.kallsyms] [k] secondary_startup_64
+ 99.87% 0.00% swapper [kernel.kallsyms] [k] start_kernel
+ 99.87% 0.00% swapper [kernel.kallsyms] [k] cpu_startup_entry
+ 99.87% 0.00% swapper [kernel.kallsyms] [k] do_idle
0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] ret_from_fork
0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] kthread
0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] worker_thread
0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] process_one_work
0.11% 0.09% kworker/0:3-eve [kernel.kallsyms] [k] e1000_watchdog
...Well, Perf tool is quite a comprehensive command and we can’t exhaust all if its usage in a simple introductory guide.
To read more about Perf Performance Analysis Tool, check;
Want to install Perf on Ubuntu 18.04?
Installing Perf Performance Analysis Tool on Ubuntu 18.04
Other Tutorials
How to Perform System Security Auditing with Lynis on Ubuntu 18.04
