In this guide, we are going to learn on installing Perf Performance Analysis Tool on CentOS 8. Perf aka
perf tools is a powerful performance counters monitoring tool for Linux systems that gathers data about hardware events such as instructions executed, branches mispredicted or cache-misses suffered.
It also provides very low overhead profiling of applications to trace dynamic control flow and identify hotspots, per task, per CPU and per-workload counters, sampling and source code event annotation, dynamic creation of tracepoints using the kprobes and uprobes frameworks, for kernel and userspace dynamic tracing respectively.
Installing Perf Performance Analysis Tool on CentOS 8
Perf provide a userspace controlling utility called
perf. You need to install this Perf userland utility in order to be able to use Perf on command line.
Perf is available on the default CentOS 8 repos through the
dnf info perf
Available Packages Name : perf Version : 4.18.0 Release : 147.5.1.el8_1 Architecture : x86_64 Size : 3.4 M Source : kernel-4.18.0-147.5.1.el8_1.src.rpm Repository : BaseOS Summary : Performance monitoring for the Linux kernel URL : http://www.kernel.org/ License : GPLv2 Description : This package contains the perf tool, which enables performance monitoring : of the Linux kernel.
Run system update.
Install Perf on CentOS 8
dnf install perf
The basic command line Syntax of perf utility is;
perf [--version] [--help] [OPTIONS] COMMAND [ARGS]
The most common perf sub-commands include;
stat: runs a command and gathers performance counter statistics from it.
top: generates and displays a performance counter profile in real time.
record: runs a command and gathers a performance counter profile from it. The gathered data is saved into
perf.datawithout displaying anything.
report: It reads
perf.datato display the performance counter profile information recorded via perf record sub-command.
list: displays the symbolic event types which can be selected in the various perf commands with the -e option.
trace: Just like strace, perf trace sub-command shows various system calls being executed by a particular thread or process and the signals it receives.
annotate: Perf annotate command displays a report file and an annotated version of the executed code.
If you just type perf command on the terminal, you should be able to see quite a number of subcommands;
perf usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS] The most commonly used perf commands are: annotate Read perf.data (created by perf record) and display annotated code archive Create archive with object files with build-ids found in perf.data file bench General framework for benchmark suites buildid-cache Manage build-id cache. buildid-list List the buildids in a perf.data file c2c Shared Data C2C/HITM Analyzer. config Get and set variables in a configuration file. data Data file related processing diff Read perf.data files and display the differential profile evlist List the event names in a perf.data file ftrace simple wrapper for kernel's ftrace functionality inject Filter to augment the events stream with additional information kallsyms Searches running kernel for symbols kmem Tool to trace/measure kernel memory properties kvm Tool to trace/measure kvm guest os list List all symbolic event types lock Analyze lock events mem Profile memory accesses record Run a command and record its profile into perf.data report Read perf.data (created by perf record) and display the profile sched Tool to trace/measure scheduler properties (latencies) script Read perf.data (created by perf record) and display trace output stat Run a command and gather performance counter statistics test Runs sanity tests. timechart Tool to visualize total system behavior during a workload top System profiling tool. version display the version of perf binary probe Define new dynamic tracepoints trace strace inspired tool
To obtain help about a specific sub-command, just execute;
perf help SUB-COMMAND
Example Usage of Perf on CentOS 8
Next, let us go over a few example usage of perf command on CentOS 8 and similar derivatives.
Gather Command Performance Counter statistics
The stat sub-command of perf utility enables you to run a command and gather its performance counter statistics
perf stat ls
con file file1 Performance counter stats for 'ls': 1.72 msec task-clock # 0.827 CPUs utilized 0 context-switches # 0.000 K/sec 0 cpu-migrations # 0.000 K/sec 108 page-faults # 0.063 M/sec <not supported> cycles <not supported> instructions <not supported> branches <not supported> branch-misses 0.002075544 seconds time elapsed 0.002106000 seconds user 0.000000000 seconds sys
Read more on
Display Live Performance Counter Profile
perf-top is a system profiling tool used to generate and display a performance counter profile in real time. This can be achieved using the
perf top command.
Samples: 29 of event 'cpu-clock', 4000 Hz, Event count (approx.): 7250000 lost: 0/0 drop: 0/0 Overhead Shared Object Symbol 34.48% [kernel] [k] e1000_watchdog 13.79% [kernel] [k] _raw_spin_unlock_irqrestore 6.90% [kernel] [k] format_decode 6.90% [kernel] [k] vsnprintf 3.45% [kernel] [k] __x86_indirect_thunk_rax 3.45% [kernel] [k] __x86_indirect_thunk_rdx 3.45% [kernel] [k] kallsyms_expand_symbol.constprop.1 3.45% [kernel] [k] mem_cgroup_throttle_swaprate 3.45% [kernel] [k] memcpy 3.45% [kernel] [k] number 3.45% [kernel] [k] update_iter 3.45% libc-2.28.so [.] _IO_getdelim 3.45% libc-2.28.so [.] __memmove_avx_unaligned 3.45% perf [.] dso__next_symbol 3.45% perf [.] rb_next ...
man perf-top to see more usage.
List all symbolic event types
To list all symbolic event types using perf command;
alignment-faults [Software event] bpf-output [Software event] context-switches OR cs [Software event] cpu-clock [Software event] ... msr/pperf/ [Kernel PMU event] msr/smi/ [Kernel PMU event] msr/tsc/ [Kernel PMU event] ... rNNN [Raw hardware event descriptor] cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] (see 'man perf-list' on how to encode it) mem:<addr>[/len][:access] [Hardware breakpoint] ...
Record Command Profile
To run a command and record its profile into perf.data;
perf record ls -ld /etc
drwxr-xr-x. 78 root root 8192 Mar 19 20:32 /etc [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.011 MB perf.data ]
The basic syntax and a comprehensive description of using perf record command is stated on
perf record -e cpu-clock -a -g -- sleep 3
[ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.628 MB perf.data (4514 samples) ]
Read perf.data and display the profile
To read the performance record, use the report sub-command. The command syntax is
perf report [-i <file> | --input=file]
perf report -i perf.data
Samples: 4K of event 'cpu-clock', Event count (approx.): 1128500000 Children Self Command Shared Object Symbol + 99.87% 99.87% swapper [kernel.kallsyms] [k] mwait_idle + 99.87% 0.00% swapper [kernel.kallsyms] [k] secondary_startup_64 + 99.87% 0.00% swapper [kernel.kallsyms] [k] start_kernel + 99.87% 0.00% swapper [kernel.kallsyms] [k] cpu_startup_entry + 99.87% 0.00% swapper [kernel.kallsyms] [k] do_idle 0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] ret_from_fork 0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] kthread 0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] worker_thread 0.13% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] process_one_work 0.11% 0.09% kworker/0:3-eve [kernel.kallsyms] [k] e1000_watchdog ...
Well, Perf tool is quite a comprehensive command and we can’t exhaust all if its usage in a simple introductory guide.
To read more about Perf Performance Analysis Tool, check;
Want to install Perf on Ubuntu 18.04?