Linux systems can be monitored using the strace tool. Strace is a command-line utility that records system calls made by the Linux kernel. This article will show you how to use strace to monitor system calls in your Linux system. ..
The Kernel and System Calls
As smart as they might be, computer programs can’t do everything for themselves. They need to make requests to have certain functions performed for them. These requests go to the Linux kernel. Typically, there’s a library or other software interface that the program calls, and the library then makes the appropriate request—called a system call—to the kernel.
Being able to see the system calls that a program has made and what the responses were can help you understand the inner workings of programs that interest you or that you have written. This is what strace does. It can help troubleshoot issues and look for bottlenecks.
This isn’t the same as debugging an application with a tool like gdb . A debugging program lets you investigate the internal operation of a program as it runs. It lets you step through the logic of your program and inspect memory and variable values. By comparison, what strace does is capture the system call information as the program is running. When the traced program terminates, strace lists the system call information to the terminal window.
System calls provide all sorts of low-level functionality, such as read and write actions on files, killing processes, and so on. There’s a list of hundreds of system calls on the syscalls man page.
RELATED: Debugging with GDB: Getting Started
Installing strace
If strace isn’t already installed on your computer, you can install it very easily.
On Ubuntu, use this command:
On Fedora, type this command:
On Manjaro, the command is:
First Steps with strace
We’ll use a small program to demonstrate strace. It doesn’t do much: It opens a file and writes a line of text to it, and it doesn’t have any error checking in it. It’s just a quick hack so that we have something to use with strace.
We saved this into a file called “file-io.c” and compiled it with gcc into an executable called stex, named for “strace example.”
We’ll call strace from the command line and pass the name of our new executable to it as the process that we want to have traced. We could just as easily trace any of the Linux commands or any other binary executable. We’re using our tiny program for two reasons.
The first reason is that strace is verbose. There can be a lot of output. That’s great when you’re using strace in anger, but it can be overwhelming at first. There’s limited strace output for our tiny program. The second reason is that our program has limited functionality, and the source code is short and straightforward. This makes it easier to identify which sections of the output refer to the different parts of the internal workings of the program.
We can clearly see the write system call sending the text “Write this to the file” to our opened file and the exit_group system call. This terminates all threads in the application and sends a return value back to the shell.
Filtering the Output
Even with our simple demonstration program, there’s quite a lot of output. We can use the -e (expression) option. We’ll pass in the name of the system call that we want to see.
You can report on multiple system calls by adding them as a comma-separated list. Don’t include any whitespace in the list of system calls.
Sending the Output to a File
The benefit of filtering the output is also the problem with filtering the output. You see what you’ve asked to see, but you don’t see anything else. And some of that other output might be more useful to you than the stuff you’ve asked to see.
Sometimes, it’s more convenient to capture everything and search and scroll through the entire set of results. That way, you won’t accidentally exclude anything important. The -o (output) option lets you send the output from a strace session to a text file.
You can then use the less command to scroll through the listing and search for system calls—or anything else—by name.
You can now use all of less‘s search capabilities to investigate the output.
RELATED: How to Use the less Command on Linux
Adding Timestamps
You can add several different timestamps to the output. The -r (relative timestamps) option adds timestamps that show the time difference between the start of each successive system call. Note that these time values will include the time spent in the previous system call and anything else that the program was doing before the next system call.
The timestamps are displayed at the start of each line of output.
To see the amount of time spent in each system call, use the -T (syscall-times) option. This shows the duration of time spent inside each system call.
The time durations are shown at the end of each system call line.
To see the time at which each system call was called, use the -tt (absolute timestamps) option. This shows the “wall clock” time, with a microsecond resolution.
The times are displayed at the start of each line.
Tracing a Running Process
If the process that you want to trace is already running, you can still attach strace to it. To do so, you need to know the process ID. You can use ps with grep to find this. We have Firefox running. To find out the ID of the firefox process, we can use ps and pipe it through grep.
We can see that the process ID is 8483. We’ll use the -p (process ID) option to tell strace which process to attach to. Note that you’ll need to use sudo :
You’ll see a notification that strace has attached itself to the process, and then the system trace calls will be displayed in the terminal window as usual.
Creating a Report
The -c (summary only) option causes strace to print a report. It generates a table for information about the system calls that were made by the traced program.
The columns are:
% time: The percentage of the execution time that was spent in each system call. seconds: The total time expressed in seconds and microseconds spent in each system call. usecs/call: The average time in microseconds spent in each system call. calls: The number of times that each system call was executed. errors: The number of failures for each system call. syscall: The name of the system call.
These values will show zeros for trivial programs that execute and terminate quickly. Real-world values are shown for programs that do something more meaningful than our demonstration application.
Deep Insights, Easily
The strace output can show you which system calls are being made, which ones are being made repeatedly, and how much execution time is being spent inside kernel-side code. That’s great information. Often, when you’re trying to understand what’s going on inside your code, it’s easy to forget that your binary is interacting almost nonstop with the kernel to perform many of its functions.
By using strace, you see the complete picture.
