Root Zanli
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
proc
/
thread-self
/
root
/
usr
/
share
/
doc
/
systemtap-sdt-dtrace
/
Filename :
NEWS
back
Copy
* What's new in version 5.4 - Type checking and autocast processing have been made more thorough, so elided variables are checked more and @defined() tests may be more complicated. Preexisting scripts that rely on elision for bypassing type violations may now get caught. No --compatible option exists to suppress this new behaviour. * What's new in version 5.3, 2025-05-02 - Updated to support drastic linux 6.13 kbuild changes, numerous other 6.14/6.15 linux API changes. - The first pass (-p1) now runs parse operations concurrently over of the hundreds of tapset .stp files, potentially saving seconds of time at startup. - The systemtap rpm package now invokes a few basic tests during %check, intended to confirm compatibility with the kernel. * What's new in version 5.2, 2024-11-08 - New /* pragma:suffix */ for file-level embedded C code blocks to be placed near the end of the emitted kernel module C file. - The systemtap-sdt-devel subrpm has been split into -dtrace and -devel subpackages. * What's new in version 5.1, 2024-04-26 - An experimental "--build-as=USER" flag to reduce privilege during script compilation. - Support DWARF "DW_OP_bra" location construct. - Dynamic namespace switching for processes targeted via "-x PID". - New probe-points for userland hardware breakpoints/watchpoints targeting processes (PR31180): probe process.data(ADDRESS).write probe process.data(ADDRESS).rw probe process.data(ADDRESS).length(LEN).write probe process.data(ADDRESS).length(LEN).rw - Support for stap --remote=bpf. - Improved searching heuristics for probing symbol names with/without symbol-version suffixes. - Improved message transport robustness for heavy traffic and for short runs. * What's new in version 5.0, 2023-11-04 - Performance improvements in uprobe registration and module startup. - More probe point process details are printed in "-L" list mode with more "-v". - For the case where newer kernels may break systemtap runtime APIs, better pass-4 failure diagnostics are printed. - Tapset function print_ubacktrace_fileline() now understands DWARF5. - The target(s) of process probes may be specified by path name globs, as located selected debuginfod servers. This requires the debuginfod servers to support "metadata" queries. The following probes all of the binaries matching /usr/*/curl known to the debuginfod servers, regardless of what's currently installed as the system /usr/bin/curl: # export DEBUGINFOD_URLS="URL1 URL2 ..." # stap -e 'probe debuginfod.process("/usr/*/curl").begin { log("hi") }' The archive glob may be given as an optional component to filter further: # stap -e 'probe debuginfod.archive("*.fc38*") .process("/usr/*/c*").function("main") { log("Hello World") }' - The kernel-user message transport system added framing codes, making the transport more reliable, but becoming incompatible across pre-5.0 versions. Use matching versions of stap and staprun. - RHEL6, kernel 2.6.32* as well as older releases have been deprecated from the codebase. RHEL7 and kernel version 3.10 are now the oldest supported versions for Systemtap. - The testsuite Makefile has been simplified to remove concurrency, so "parallel" and "resume" modes are gone. - New runtime macro STP_TIMING_NSECS is now supported for reporting probe timing stats in nsecs instead of cycles. This may become default later. - Add new runtime macro STP_FORCE_STDOUT_TTY to override STP_STDOUT_NOT_ATTY. * What's new in version 4.9, 2023-04-28 - ISystemtap: the new interactive systemtap jupyter kernel. This provides a simple GUI for writing/running Systemtap scripts in an easy, incremental way within Jupyterlab. Added 2 news scripts, stap-jupyter-install and stap-jupyter-container. See stap-jupyter(1). - The new "language-server" mode, specified by "stap --language-server", starts a LSP server which will communicate with the client via stdio. This server currently supports code completion suggestions. See language-server/README.md for more details and usage instructions. - Safety/liveness checking for $context variable assignments in probes of retpoline-compiled kernels is temporarily disabled. * What's new in version 4.8, 2022-11-03 - DWARF-related probes (.function, .statement) now merge DWARF and non-DWARF symbol-table based matches, rather than being either-or. - The python3 tapset was extended to support python3 3.9, 3.10, and 3.11. See stapprobes(3stap) for further details on probing python functions. - A template cve band-aid script is now included, which demonstrates how to use a new 'livepatch.stp' tapset to standardize activation, interactive control, and monitoring of systemtap cve band-aids. - The kernel runtime now uses much less memory when the number of "possible CPUs" are way more than the online ones. For example, VMWare guests usually have 128 "possible CPUs" while fewer CPUs are actually present or online in the guest system. - The memory allocation size is now irrelevant to the value of NR_CPUS of the current kernel. It is only subject to the number of "possible CPUs" or "online CPUs". - CPU hotplug is supported to the extent that there won't be any kernel panics or memory corruptions. - The bpf backend's embedded-code assembler has been improved to support more conventional assembly syntax with named opcodes. The opcode names are based on the iovisor bpf-docs documentation at https://github.com/iovisor/bpf-docs/blob/master/eBPF.md * What's new in version 4.7, 2022-05-02 - SystemTap now supports an additional method to sign modules on UEFI/SecureBoot systems. In addition to the existing method of using a trusted stap-server, the module can also be signed without using a server by specifying the "stap --sign-module" option. (Key enrollment still requires a one-time reboot and BIOS conversation.) https://sourceware.org/systemtap/wiki/SecureBoot - Includes new tool stap-profile-annotate, combining systemtap & debuginfod to collect system-wide profiling statistics, and produce annotated source files for all relevant programs/libraries. - target processes given with the -c/-x parameters are now always added to the -d list for possible symbol/unwind data extraction, for simplifying profiling invocations. Consider --ldd. - The "subbuf" size for kernel-to-user bulk message passing is fixed at the page size. Number of such subbufs is affected by the stap "-s" parameter and amount of free memory available. * What's new in version 4.6, 2021-11-15 - SystemTap has added support for the 64-bit RISC-V architecture. - stap-prep now tries to download the main kernel debuginfo file from a debuginfod server, if configured. - Updated syscall_any tapset mapping to include newer syscalls. - syscall_any tapset can be used by the bpf backend. - SystemTap now uses DynInst to perform a liveness analysis on target variables and warn when a guru-mode modification to a variable will have no effect. The liveness analysis is currently done on x86_64, PowerPC, and AArch64. - The kernel-user relayfs transport again sorts messages into a total time order across CPUs. High output-volume scripts may need a larger "-s BUF" parameter to reliably transfer. "-b" bulk mode is also available again as an alternative. - abort() tapset can be used by the bpf backend. - The bpf backend now supports foreach iteration in multi-key associative arrays. * What's new in version 4.5, 2021-05-07 - Java probing support has been updated to work with the latest versions of the JVM and byteman (PR27739). - An initial version of the bpf/uconversions.stp tapset provides tapset functions such as user_long_error() to access values in userspace. - Enabled the -c option to work with the bpf backend (PR25177). - Enum values can now be accessed as $context variables (PR25346). - Executables for which stap has execute but not read permissions (--x--x--x) can now be probed with build-id probes and debuginfod. This allows probing some setuid programs (PR27251). - Added VMA-tracking support to the stapdyn backend. - Several concurrency/locking fixes and improvements were made to the uprobes family of probes and to the transport subsystem. Large machines with hundreds of CPUs are better supported. - The kernel runtime now uses procfs as the default namespace for the relayfs pseudo-files. - Floating point variables may now be accessed directly as normal $context variables. 32-bit floats are automatically widened to doubles. (PR13838) - A wider variety of SecureBoot MOK keys are now recognized for more reliable triggering of stap-server module signing. Document some additional lockdown/secureboot administrative escape options. (PR26665) - On startup, stap explains the [man FOO] diagnostic syntax. * What's new in version 4.4, 2020-11-09 - Make syscall arguments writable again in non-DWARF probes on kernels that use syscall wrappers to pass arguments via pt_regs (currently x86_64 4.17+ and aarch64 4.19+). - Add new syntax for defining aliases with both a prologue and an epilogue: 'probe ALIAS = PROBE { <prologue> }, { <epilogue> }' - Add @probewrite predicate. @probewrite(var) returns 1 if var has been written to in the probe handler body and 0 otherwise. The check can only be used with probes that have an epilogue or prologue. - Implicit thread local storage variables can now be accessed on x86_64, ppc64le, and s390x. - Replaced spinlocks with RCU locks in vma map and utrace task's hash table lookups which reduces CPU time a lot when there are a lot of target processes and vma tracker or task finder is enabled. We also increased the default hash table sizes to reduce hash conflicts. - stap-prep now avoids downloading kernel debuginfo if a successful connection to debuginfod server is made. - The locks required to protect concurrent access to global variables has been optimized with a "pushdown" algorithm, so that they span the smallest possible critical region. Formerly, and with --compatible=4.3, locks always spanned the entire probe handler. Lock pushdown means much greater potential concurrency between probes running on different CPUs. - Systemtap now supports kernel-lockdown configurations that disable debugfs, by instead using procfs to carry relayfs transport files. - Systemtap now supports extracting 64-bit floating point and stored in long type. Also basic floating point arithmetic and comparison functions are provided in a tapset. More automated syntax coming soon. e.g.: probe process.function("foo") { fp = user_long(& $fp_variable) println (fp_to_string (fp_add (string_to_fp("3.14"), fp))) } * What's new in version 4.3, 2020-06-11 - tapset functions for reading CPU registers and primitives involved with the process memory maps work in 'probe process.begin'. - probe aliases starting with nfs.proc support IPv6 as server_ip and client_ip. Also, ip are represented with string valued now. - The target of process probes may be specified by hexadecimal buildid as an alternative to path names. This makes it possible to probe a variety of versions or aliases of a program, even if they are running inside containers under a different path name. Works best with a debuginfod server that publishes the executables / debuginfo. The following probes glibc.so 2.32-2.fc32.x86_64 from fedora running anywhere on your machine. # export DEBUGINFOD_URLS=https://debuginfod.elfutils.org/ # stap -e 'probe process("7ca24d4dc3de9d62d9ad6bb25e5b70a3e57a342f") .function("*system") { log("hi") }' - Functions can now be context-sensitive, meaning that they may make references to $context variables and similar constructs that could formerly appear only inside probe handlers. This is implemented by cloning the such functions for each probe. Only some probe point (dwarf-based user & kernel) types supported. function foo () { println ($$vars) } probe kernel.function("do_exit") { foo() } probe process("/bin/ls").function("main") { foo() } probe process("/lib*/libc.so.6").mark("*") { foo() } - Add new tapset function dump_stack() which prints the current kernel backtrace to the kernel trace buffer (as a thin wrapper around the kernel C API function dump_stack). - Almost all of the kmalloc() allocations exceeding 4KB have been replaced by vmalloc(). This helps stap's kernel runtime work properly on systems with serious fragmentation in physical memory address space. - More $variable resolution errors may be generated, especially for @var("") constructs that target global variables. These are duplicate-eliminated by default, but may be seen with verbosity>=2. - The stapbpf backend now supports try-catch statements, an improved error tapset and error probes. - The "Build-id mismatch" condition now becomes a warning, so while related probes are not inserted, the rest of the script may run. - The process(EXE).begin probe handlers are now always triggered for already-running target processes. - The proc_mem_rss() tapset function now includes the resident shared memory pages as expected. The old behavior can be restored by the --compatible=4.2 option on the command line. - Modules compiled with guru mode for a particular kernel version can now only be loaded on kernels with exactly matching version (vermagic string) instead of any kernel whose API matches according to the modversions mechanism. Use -B CONFIG_MODVERSIONS=y to restore the prior behaviour. * What's new in version 4.2, 2019-11-18 - Initial support for multi-dimensional supports has been added to the stapbpf backend. Note that these arrays cannot be iterated upon with a foreach loop. - The stapbpf backend now supports sorting by value in foreach loops. - The stapbpf backend now supports the concatenation operator for userspace probes. - The stapbpf backend now supports the target() function and -x option. - The gettimeofday_* functions are now provided for the stapbpf backend. - The values of an array can now be iterated over in foreach loops in the stapbpf backend. They are no longer defaulted to 0. - The stapbpf backend now supports order parameterization for begin and end probes. - When the -v option is set along with -L option, the output includes duplicate probe points which are distinguished by their PC address. - The stapbpf backend now supports stap-exporter extensions. - The stapbpf backend now supports procfs probes. The implementation uses FIFO special files in /var/tmp/systemtap-$EFFUSER/MODNAME instead of the proc filesystem files. - The eBPF backend now uses bpf raw tracepoints for kernel.trace("*") probes. These have target variable arguments that match the arguments available for the traditional linux kernel modules tracepoints. Support for the older bpf tracepoint arguments can be forced with a --compatible=4.1 option on the command line. - New backtracing functions print_[u]backtrace_fileline() have been added to the tapset. These functions behave similarly to print_[u]backtrace(), the only difference being that file names and line numbers are added to the backtrace. - Now it is possible to issue a backtrace using user specified pc, sp, and fp which can be used to generate backtraces of different contexts. This was introduced to get backtraces of user code from within the go runtime but it can also be used to do things like generating backtraces of user code from within signal handlers. - The compiler optimizes out probes with empty handlers. Previously, warnings were issued but, the probe was not internally removed. For example, this script now outputs a warning and an error as the only probe handler is empty: probe begin {} Additionally, probe handlers that evaluate to empty are also removed. For example, in this script, the begin probe is elided as $foo does not exist, however, an error won't be outputted because atleast one probe with a non-empty handler exists (probe begin): probe begin { print("Protected from elision") } probe end { if (@defined($foo)) { print("Evaluates to empty handler") } } - The automatic printing implementation now differentiates between pointer and integer types, which are printed as hex or decimal respectively. - The sys/sdt.h file changes the way i386 registers operands are sometimes named, due to an ambiguity. A comment block explains. * What's new in version 4.1, 2019-05-07 - Runtime/tapsets were ported to include up to kernel version 5.1-rc2 - The translator's pass-2 (elaboration) phase has been dramatically accelerated by eschewing processing of unreferenced parts of the tapset and embedded-C code. - New macros @this1, ..., @this8 have been added to the script language. The macros can be used to save values in entry probes and later retrieve them in return probes. This can be used in instances where @entry does not work. For example: probe tp_syscall.read { @this1 = buf_uaddr } probe tp_syscall.read.return { buf_uaddr = @this1 printf("%s", user_string(buf_uaddr)) } - Operator @var() no longer assumes @entry() in return probes. The old behavior can be restored by the '--compatible 4.0' option. - Where available (kernel 4.15+), the kernel-module runtime now uses the kernel-provided ktime_get_real_fast_ns() mechanism for timekeeping rather than the SystemTap runtime's own timekeeping machinery. - New stapbpf tapset task.stp with function task_current. - New stapbpf tapset functions kernel_string, kernel_string_n, execname, ktime_get_ns. - A new stapbpf transport layer has been implemented based on perf_events infrastructure. This transport layer removes some limitations on strings and printf() that were previously imposed by BPF's trace_printk() mechanism: - It is now possible to use format-width specifiers in printf(). - It is now possible to use more than three format specifiers in printf(). - It is now possible to use multiple '%s' format specifiers in printf(). - Using stapbpf should no longer trigger a trace_printk()-associated warning on dmesg. - In the stapbpf backend, foreach() now supports iteration of arrays with string keys. - A preview version of statistical aggregate functionality for stapbpf is now included. For now, in the stapbpf backend, aggregates only support @count(), @sum() and @avg() operations. - Added support for unhandled DWARF operator DW_OP_GNU_parameter_ref in location expressions for target symbols. * What's new in version 4.0, 2018-10-13 - Runtime/tapsets were ported to include up to kernel version 4.19-rc - A new network service, stap-exporter, is included. It glues systemtap and the web. It allows a prometheus (or compatible systems such as pcp) to consume metrics exported by systemtap scripts. Some tapset macros/functions are available to make it easier to write such scripts. See the stap-exporter(8) man page and the systemd service. - When a systemtap module is loaded, the name of the original stap script is now printed to dmesg by the kernel runtime. - On some Fedora kernels, the information necessary to automatically engage in SecureBoot module signing is hidden from systemtap. Setting the $SYSTEMTAP_SIGN environment variable forces it on. A running stap-server instance will also be needed. - Embedded-C functions marked /* guru */ may now be invoked from other tapset probes / functions, while still being invalid for normal call from an unprivileged user script. - The syscall tapset is now updated to work on kernel 4.17+. Additionally, the tapset now includes an automatic fallback alias to the sys_enter / sys_exit kernel tracepoints, if no other kprobe-based mechanism is found. These changes have brought unavoidable consequences. Raw $target variables for the syscall arguments and return probes (e.g. @entry($fd), $return, returnval()) may not longer be relied upon. Instead, use the variables defined by the tapset aliases. For example: % stap -L syscall.read syscall.read name:string fd:long buf_uaddr:long count:long argstr:string % stap -L syscall.read.return syscall.read.return name:string retval:long retstr:string to see the available variables for that syscall. See [man stapprobes] for further details. returnval() in particular is being deprecated soon; use retval in syscall.*.return probes instead. - New script language operators @kderef/@uderef and @kregister/@uregister were added. @kderef/@uderef (size,address) can be used to dereference integers and @kregister/@uregister (dwarf#) can be used to access register values. - A systemd service file has been added for systemtap.service (which runs a configurable set of scripts automatically on system startup). The existing /etc/init.d/systemtap init script has been moved to a new utility command 'systemtap-service' which preserves functionality such as configuring onboot systemtap scripts via dracut. See systemtap-service(8) for details. - The eBPF backend's string support has been improved. Strings can now be stored in variables, passed as function arguments, and stored as array keys and values. - The 3rd operand of the ternary operator '?:' in the script language now binds tighter than the binary assignment operators like '=' and '+=', just like the C language. The original operator precedence can be restored by the '--compatible 3.3' option. - The script language now supports the use of bare 'return' statements (without any return values) inside functions which do not return any values. A trailing semicolon is recommended for such return statements to avoid any potential ambiguity. The parser treats a following semicolon (';') or a closing curly bracket ('}') as a terminator for such bare return statements. - Parentheses after unary '&' with a target-symbol expression is now accepted in the script language. - Tapset functions register() and u_register() now support 8-bit x86 register names "ah", "al", "bh", "bl", "ch", "cl", "dh", and "dl" on both x86_64 and i386. And 16-bit x86 registers are now truly read as 16-bit integers instead of as 32-bit ones. - The experimental ftrace ring buffer mechanism (STP_USE_RING_BUFFER) has been deprecated and may be removed in future versions. * What's new in version 3.3, 2018-06-08 - A new "stap --example FOO.stp" mode searches the example scripts distributed with systemtap for a file named FOO.stp, so its whole path does not need to be typed in. - Systemtap's runtime has learned to deal with several of the collateral damage from kernel hardening after meltdown/spectre, including more pointer hiding and relocation. The kptr_restrict procfs flag is forced on if running on a new enough kernel. - The "stap --sysroot /PATH" option has received a revamp, so it works much better against cross-compiled environments. - The eBPF backend has learned to perform loops - at least in the userspace "begin/end" probe contexts, so one can iterate across BPF arrays for reporting. (The linux kernel eBPF interpreter precludes loops and string processing.) It can also handle much larger probe handler bodies, with a smarter register spiller/allocator. - The eBPF backend now supports uprobes, perf counter, timer, and tracepoint probes. - An rpm macro %_systemtap_tapsetdir is now defined, to make it easier for third party packages to add .stp files into the standard tapset. - Several low level locking-related fixes were added to the runtime that used uprobes/tracepoint apis, in order to work more reliably on rt kernels and on high-cpu-count machines. - Runtime/tapsets were ported to include up to kernel version 4.16. (The syscall tapsets are broken on kernel 4.17-rc, and will be fixed in a next release coming soon; PR23160.) - Add new built-in tapset function abort() which is similar to exit(), but it aborts the current probe handler (and any function calls in it) immediately instead of waiting for its completion. Probe handlers already running on *other* CPU cores, however, will still continue to their completion. Unlike error(), abort() cannot be caught by try {...} catch {...}. Similar to exit(), abort() yeilds the zero process exit code. It works with both the kernel and dyninst runtimes. This function can be disabled by the '--compatible 3.3' option. * What's new in version 3.2, 2017-10-18 - SystemTap now includes an extended Berkeley Packet Filter (eBPF) backend. This experimental backend does not use kernel modules and instead produces eBPF programs that are verified by the kernel and executed by an in-kernel virtual machine. Select this backend with the new stap option '--runtime=bpf'. For example: stap --runtime=bpf -e \ 'probe kernel.function("sys_open") { printf("hi from stapbpf!\n") }' Please see the stapbpf(8) man page for more information. - The regular expression engine now supports extraction of the matched string and subexpressions using the matched() tapset function: if ("regexculpicator" =~ "reg(ex.*p).*r") log(matched(1)) -> exculp - The translator produces better diagnostics for common/annoying case of missing debuginfo that blocks use of context $variables. - "stap -k" build trees in $TMPDIR now also include a preprocessed .i form of the generated module .c code, for problem diagnostics purposes. - The syscall.execve probes now provide a decoded env_str string vector, just like the argument vector. Because of this, the unused __count_envp() and __count_compat_evenp() functions have been deprecated. - The task_exe_file() Function has been deprecated and replaced by the current_exe_file() function. - A new probe alias input.char allows scripts to access input from stdin during runtime. * What's new in version 3.1, 2017-02-17 - Systemtap now needs C++11 to build. - Syscall and nd_syscall tapsets have been merged in a way that either dwarf-based, or non-dwarf probe gets automatically used based on debuginfo availability (e.g. probe syscall.open). To force use the dwarf based probe, a dw_syscall has been introduced (e.g. probe dw_syscall.open) and the non-dwarf syscall probes were left untouched (e.g. nd_syscall.open). - The syscall tapset files have been reorganized in a way that original big tapset files carrying many syscall probes were split into smaller 'sysc_' prefixed tapset files. This should reduce the syscall tapset maintenance burden. - The powerpc variant of syscall.compat_sysctl got deprecated on favor of syscall.sysctl32. This aligns the syscall to its respective nd_syscall and to ia64/s390/x86_64 variants too. - The syscall.compat_pselect7a (this was actually a typo, but still available for compatibility purposes with --compatible 1.3) has beed deprecated. - The 'description_auddr' convenience variable of syscall.add_key has been deprecated. - Support has been added for probing python 2 and 3 functions using a custom python helper module. Python function probes can target function entry, returns, or specific line numbers. probe python2.module("myscript").function("foo") { println($$parms) } To run with the custom python helper module, you'd use python's '-m' option like the following: stap myscript.stp -c "python -m HelperSDT myscript.py" - Java method probes now convert all types of java parameters to strings using the java toString() method before passing them to systemtap probes; new argN variables copy them into string variables. Previously, only numeric types were passed, and only by casting to integers. The previous behaviour is available with --compatible=3.0 . 3.1: probe java(...).class(...).method(...) { printf("%s", arg1) } 3.0: probe java(...).class(...).method(...) { printf("%d", $arg1) } - An older defensive measure to suppress kernel kprobes optimizations since the 3.x era has been disabled for recent kernels. This improves the performance of kernel function probes. In case of related problems, please report and work around with: # echo 0 > /proc/sys/debug/kprobes-optimization - Context variables in .return probes should be accessed with @entry($var) rather than $var, to make it clear that entry-time snapshots are being used. The latter construct now generates a warning. Availability testing with either @defined(@entry($var)) or @defined($var) works. - Tapsets containing process probes may now be placed in the special $prefix/share/systemtap/tapset/PATH/ directory to have their process parameter prefixed with the location of the tapset. For example, process("foo").function("NAME") expands to process("/usr/bin/foo").function("NAME") when placed in $prefix/share/systemtap/tapset/PATH/usr/bin/ This is intended to help write more reusable tapsets for userspace binaries. - The implementation of "var <<< X" for each aggregate variable is now specially compiled to compute only the script-requested @op(var) values, not all potential ones. This speeds up the <<< operations. - Systemtap now warns if script arguments given on the command line are unused, instead of mentioned by the script with $n/@n. - Netfilter tapsets now provide variables data_hex and data_str to display packet contents in hexadecimal and ASCII respectively. - Translator now accepts new @const() operator for convenient expressing constants in tapset code, or guru-mode scripts. See stap(1) for details. - New -T option allows the script to be terminated after a specified number of seconds. This is a shortcut for adding the probe, timer {exit()}. - New installcheck-parallel testsuite feature allows running the tests in parallel in order to save time. See testsuite/README for details. - New tapset functions set_user_string(), set_user_string_n(), set_user_long() set_user_int(), set_user_short(), set_user_char() and set_user_pointer() to write a value of specified type directly to a user space address. - New tapset functions user_buffer_quoted(), user_buffer_quoted_error(), kernel_buffer_quoted(), and kernel_buffer_quoted_error() to print a buffer of an exact length. These functions can handle '\0' characters as well. - New statistics @variance() operator using the Welford's online algorithm for per-cpu computation, and the Total Variance formula authored by Niranjan Kamat and Arnab Nandi from the Ohio State University for the cross-cpu aggregation. - New command within interactive mode, sample. Allows you to search through all included example scripts to load for further editing or running. Sample and example scripts have been moved to /usr/share/systemtap/examples. A symlink in the former location under $docdir links to it. * What's new in version 3.0, 2016-03-27 - The new experimental "interactive" mode, specified by "stap -i", drops you into a command-line prompt where you can build up a script, run it, edit it, run it again, etc. Type "help" for a list of commands. - New experimental --monitor[=INTERVAL] option similar to unix "top". This allows users to see statistics about the running module(uptime, module name, invoker uid, memory sizes, global variables, and the current probe list along with their statistics). An interface is also provided to allow control over the running module(resetting global variables, sorting the list of probes, deactivating and reactivating probes). - The performance of associative arrays have been dramatically improved, especially for densely filled tables and for multiple indexes. The hash tables behind these arrays is now sized as a function of the array maximum size with an optional MAPHASHBIAS space/time tradeoff knob. - Add macros @prints to print a scalar aggregate variable, @prints[1-9] to print an array aggregate (of given index-arity), formatted similarly to the automatic printing of written-only global variables. global a, b probe oneshot { a <<< 1; b[tid()] <<< 2 } probe end { @prints(a); @prints1(b) } - Functions may now be overloaded during module runtime using the "next" statement in script functions and STAP_NEXT macro for embedded-C functions. They may also be overloaded by number of parameters during compile time. For example, Runtime overloading: function f() { if (condition) next; print("first function") } function f() %{ STAP_NEXT; print("second function") %} function f() { print("third function") } For the given functions above, a functioncall f(), will execute the body of the third function if condition evaluates to true and print "third function". Note that the second function is unconditionally nexted. Parameter overloading: function g() { print("first function") } function g(x) { print("second function") } g() -> "first function" g(1) -> "second function" Note that runtime overloading does not occur in the above example as the number of parameters of the functions differ. The use of a next statement inside a function while no more overloads remain will trigger a runtime exception. The function candidates are selected at compile time and is determined by the number of arguments provided for the functioncall. - Add Czech version of manual pages. - The stap compile server will log the stap client's options that are passed to the server. The options that get logged on the server will include the script name or the -e script, depending on which is used by the client. - Embedded-C functions and blocks may now access script level global variables using the STAP_GLOBAL_GET_* and STAP_GLOBAL_SET_* macros. To read or write the script global var, the /* pragma:read:var */ or /* pragma:write:var */ marker must be placed in the embedded-C function or block. The written type must match the type inferred at script level. Scalars: STAP_GLOBAL_SET_var(STAP_GLOBAL_GET_var()+1) -> increments script global var by 1 STAP_GLOBAL_SET_var("hello") Associative arrays: STAP_GLOBAL_GET_var(index-1, ..., index-n) STAP_GLOBAL_SET_var(index-1, ..., index-n, new value) - Probe point brace expansion is now supported to improve brevity in specifying probe points. For example, process.{function("a"), function("b").{call,return}} => process.function("a"), process.function("b").call, process.function("b").return process.{function("*").callees,plt}? => process.function("*").callees?, process.plt? {kernel,module("nfs")}.function("nfs*")! => kernel.function("nfs*")!, module("nfs").function("nfs*")! - Profiling timers at arbitrary frequencies are now provided and perf probes now support a frequency field as an alternative to sampling counts. probe timer.profile.freq.hz(N) probe perf.type(N).config(M).hz(X) The specified frequency is only accurate up to around 100hz. You may need to provide a higher value to achieve the desired rate. - Added support for private global variables and private functions. The scope of these is limited to the tapset file they are defined in (PR19136). - New tapset function string_quoted() to quote and \-escape general strings. String $context variables that are pretty-printed are now processed with such a quotation engine, falling back to a 0x%x (hex pointer) on errors. - Functions get_mmap_args() and get_32mmap_args() got deprecated. * What's new in version 2.9, 2015-10-08 - SystemTap now uses symbols from /proc/kallsyms when kernel debuginfo is not available. - New --prologue-searching[=WHEN] option has been added to stap with '-P' being its short counterpart. Using --prologue-searching=never turns prologue searching deliberately off working around issue of int_arg() returning wrong value when a 32-bit userspace binary having debug info is being probed with active prologue searching (PR18649). - The powerpc variant of nd_syscall.compat_sysctl got deprecated on favor of nd_syscall.sysctl32. This aligns the nd_syscall to its respective syscall and to ia64/s390/x86_64 variants too. - New tapset function assert(expression, msg) has been added. - Embedded-C functions may now use the new STAP_PRINTF(fmt, ...) macro for output. - New tapset functions fullname_struct_path and fullname_struct_nameidata resolve full path names from internal kernel struct pointers. - New tapset functions arch_bytes() and uarch_bytes() to obtain address size for kernel and user space respectively. - New tapset function switch_file() allows control over rotation of output files. - The [nd_]syscall tapset got autodocumented. Related paragraph got added to PDF and HTML tapset reference. Also a new tapset::syscall 3stap man page got added. - Embedded-C functions with parameter arity-0 can now be marked with the /* stable */ /* pure */ pragmas, if (roughly speaking) the function is side-effect-free and idempotent. The translator may execute these speculatively and have their results memoized. This lets probes with multiple calls to such functions run faster. Context variable ($foo) getter functions (in non-guru mode), and numerous tapset functions are now marked as /* stable */ /* pure */. Several example scripts have been modified to eschew explicit memoization. - Callee probe points now support '.return' and '.call' suffix. For example, process("proc").function("foo").callee("bar").return will fire upon returning from bar when called by foo. process("proc").function("foo").callee("bar").call will only fire for non-inlined callees. - The following tapset variables and functions are deprecated in version 2.9: - The '__int32_compat' library macro got deprecated in favor of new '__compat_long' library macro. - The 'uargs' convenience variable of the 'seccomp' syscall probe got deprecated in favor of new 'uargs_uaddr' variable. - SystemTap has reduced its memory consumption by using interned_strings (a wrapper for boost::string_ref) in place of std::string instances. The change is to reduce the number of duplicate strings created by replacing them with interned_strings which act like pointers to existing strings. For the implementation of interned_string, see stringtable.h * What's new in version 2.8, 2015-06-17 - SystemTap has improved support for probing golang programs. Work has been done to be able to handle DWARF information, reporting file names, line numbers, and column numbers, and tolerance of odd characters in symbol names. - The function::*, probe::* and new macro::* man pages cross-references the enclosing tapset::* man page. For example: function::pn(3stap) mentions tapset::pn(3stap) in the SEE ALSO section - New stapref(1) man page provides a reference for the scripting language. The stapref page contains an overview of the features available in the language, such as keywords, data types, operators and more. - The @task macro performs the very common @cast to a task_struct. The embedded-C bodies of task_current() and pid2task() are now wrapped by @task, which gives them a debuginfo type on the return value. With autocast type propagation, this removes the need for any explicit @cast in many places. Other places which take untyped task pointers as parameters, for instance, now use @task as well to simplify their code. - New namespace-aware tapset functions [task_]ns_*() and ia new option --target-namespaces=PID to denote a target set of namespaces corresponding to the PID's namespaces. The namespace-aware tapsets will return values relative to the target namespaces if specified, or the stap process' namespaces. - Netfilter probes now attempt to decode Spanning Tree Protocol packets into local variables: probe netfilter.bridge.*, br_* variables, stp_dump.stp sample script. - Colorization of error string tokens is made more robust, especially in presence of $N/@N substitution. - The following tapset variables and functions are deprecated in version 2.8: - The 'hostname_uaddr' variable in the syscall.setdomainname and nd_syscall.setdomainname probe aliases have been deprecated in favor of the new 'domainname_uaddr' variable. - The 'fd' and 'fd_str' variables in the syscall.execveat and nd_syscall.execveat probe aliases have been deprecated in favor of the new 'dirfd' and 'dirfd_str' variables. * What's new in version 2.7, 2015-02-18 - Some systemtap sample scripts are now identified with the "_best" keyword, because they are generally useful or educational. They are now promoted within the generated index files. - Passing strings to and from functions has become faster due to optimization (passing some strings by reference instead of by value/copy). It may be disabled by using the unoptimize flag (-u). To make embedded-C functions eligible for the same optimization, use the pragma /* unmodified-fnargs */ to indicate that the function body will not modify the function arguments. Remember to use MAXSTRINGLEN for string length, rather than sizeof(string_arg) (which might now be a pointer). - SystemTap now allows .function probes to be specified by their full function name, file, and declaration line number. Use the .statement probe to probe a specific line number. - Tracepoint probes can now also be specified by the target subsystem. For example, the following are all supported: probe kernel.trace("sched:sched_switch") --> probe sched_switch found in the sched subsystem probe kernel.trace("sched:*") --> probe all tracepoints in sched subsystem As a result, tapset functions such as pn() will now return a different string than before. To retain the previous behaviour, use '--compatible=2.6'. - The following functions are deprecated in release 2.7: - _adjtx_mode_str(), _statfs_f_type_str(), _waitid_opt_str(), _internal_wait_opt_str(), and _epoll_events_str(). - New tapset functions [u]symfileline(), [u]symfile() and [u]symline() will return a string containing the specified portion of the filename:linenumber match from a given address. Using these functions may result in large generated modules from stored address->file:line information. * What's new in version 2.6, 2014-09-05 - SystemTap now supports on-the-fly arming/disarming of certain probe types: kprobes, uprobes, and timer.*s(NUM) probes. For example, this probe probe kernel.function("vfs_read") if (i > 4) { ... } will automatically register/unregister the associated kprobe on vfs_read whenever the value of the condition changes (as some probe handler modifies 'i'). This allows us to avoid probe overhead when we're not interested. If the arming capability is not relevant/useful, nest the condition in the normal probe handler: probe kernel.function("vfs_read") { if (i > 4) { ... } } - statement("*@file:NNN").nearest probes now available to let systemtap translate probe to nearest probe-able line to one given if necessary - process("PATH").library("PATH").plt("NAME").return probes are now supported. - SystemTap now supports SDT probes with operands that refer to symbols. - While in listing mode (-l/-L), probes printed are now more consistent and precise. - Statement probes now support enumerated linenos to probe discontiguous linenos using the form: process.statement("foo@file.c:3,5-7,9") - Statement counting is now suppressed in the generated c code for probes that are non-recursive and loop-free. Statement counting can be turned back on in unoptimize mode (-u). - SystemTap now asserts that the PID provided for a process probe corresponds to a running process. - DWARF process probes can be bound to a specific process using the form: process(PID).function("*") - SystemTap now accepts additional scripts through the new -E SCRIPT option. There still needs to be a main script specified through -e or file in order to provide an additional script. This makes it feasible to have scripts in the $HOME/.systemtap/rc file. For example: -E 'probe begin, end, error { log("systemtap script " . pn()) }' -E 'probe timer.s(30) { error ("timeout") } The -E SCRIPT option can also be used in listing mode (-l/-L), such that probe points for the additional scripts will not listed, but other parts of the script are still available, such as macros or aliases. - SystemTap now supports array slicing within foreach loop conditions, delete statements and membership tests. Wildcards are represented by "*". Examples of the expressions are: foreach ([a,b,c] in val[*,2,*]) delete val[*, 2, *] [*, 2, *] in val - Integer expressions which are derived from DWARF values, like context $vars, @cast, and @var, will now carry that type information into subsequent reads. Such expressions can now use "->" and "[]" operators, as can local variables which were assigned such values. foo = $param->foo; printf("x:%d y:%d\n", foo->x, foo->y) printf("my value is %d\n", ($type == 42 ? $foo : $bar)->value) printf("my parent pid is %d\n", task_parent(task_current())->tgid) * What's new in version 2.5, 2014-04-30 - Systemtap now supports backtracing through its own, invoking module. - Java probes now support backtracing using the print_java_backtrace() and sprint_java_backtrace() functions. - Statement probes (e.g. process.statement) are now faster to resolve, more precise, and work better with inlined functions. - New switches have been added to help inspect the contents of installed library files: stap --dump-functions --> list all library functions and their args stap --dump-probe-aliases --> list all library probe aliases - The heuristic algorithms used to search for function-prologue endings were improved, to cover more optimization (or lack-of-optimization, or incorrect-debuginfo) cases. These heuristics are necessary to find $context parameters for some function-call/entry probes. We recommend programs be built with CFLAGS+=-grecord-gcc-switches to feed information to the heuristics. - The stap --use-server option now more correctly supports address:port type parametrization, for manual use in the absence of avahi. - A new probe alias "oneshot" allows a single quick script fragment to run, then exit. - The argv tapset now merges translate-time and run-time positional arguments, so all of these work: stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' hello world stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' \ -G argv_1=hello -G argv_2=world staprun hello.ko argv_1=hello argv_2=world - SystemTap now falls back on the symbol table for probing functions in processes if the debuginfo is not available. - SystemTap now supports a %( guru_mode == 0 /* or 1 */ %) conditional for making dual-use scripts. - SystemTap now supports UEFI/SecureBoot systems, via machine-owner-keys maintained by a trusted stap-server on the network. (Key enrollment requires a one-time reboot and BIOS conversation.) https://sourceware.org/systemtap/wiki/SecureBoot - SystemTap now reports more accurate and succinct errors on type mismatches. - Embedded-C functions may use STAP_RETURN(value) instead of the more wordy STAP_RETVALUE assignment followed by a "goto out". The macro supports numeric or string values as appropriate. STAP_ERROR(...) is available to return with a (catchable) error. - Some struct-sockaddr fields are now individually decoded for socket-related syscalls: probe syscall.connect { println (uaddr_af, ":", uaddr_ip) } - The documentation for the SystemTap initscript service and the SystemTap compile-server service have been completely converted from README files to man pages (see systemtap(8) and stap-server(8)). - SystemTap is now capable of inserting modules early during the boot process on dracut-based systems. See the 'onboot' command in systemtap(8) for more information. - DWARF probes can now use the '.callee[s]' variants, which allow more precise function probing. For example, the probe point process("myproc").function("foo").callee("bar") will fire upon entering bar() from foo(). A '.callees' probe will instead place probes on all callees of foo(). Note that this also means that probe point wildcards should be used with more care. For example, use signal.*.return rather than signal.*.*, which would also match '.callees'. See stapprobes(3stap) for more info. This feature requires at least GCC 4.7. - A few new functions in the task_time tapsets, as well as a new tapset function task_ancestry(), which prints out the parentage of a process. - The kprocess.exec probe has been updated to use syscall.execve, which allows access to the new process' arguments (through the new 'argstr' or 'args' variables) as well as giving better support across kernel versions. Note also that the 'filename' variable now holds the filename (quoted), or the address (unquoted) if it couldn't be retrieved. - The [s]println() function can now be called without any arguments to simply print a newline. - Suggestions are now provided when markers could not be resolved. For example, process("stap").mark("benchmart") will suggest 'benchmark'. - SystemTap colors can now be turned off by simply setting SYSTEMTAP_COLORS to be empty, rather than having to make it invalid. - There is a new context tapset function, pnlabel(), which returns the name of the label which fired. - The following tapset variables and functions are deprecated in release 2.5: - The 'clone_flags', 'stack_start', 'stack_size', 'parent_tid_uaddr', and 'child_tid_uaddr' variables in the 'syscall.fork' and 'nd_syscall.fork' probe aliases. - The '_sendflags_str()' and '_recvflags_str()' functions have been deprecated in favor of the new '_msg_flags_str()' function. - The 'flags' and 'flags_str' variables in the 'syscall.accept' and 'nd_syscall.accept' probe alias. - The 'first', 'second', and 'uptr_uaddr' variables in the 'syscall.compat_sys_shmctl', and 'nd_syscall.compat_sys_shmctl' probe aliases have been deprecated in favor of the new 'shmid', 'cmd', and 'buf_uaddr' variables. * What's new in version 2.4, 2013-11-06 - Better suggestions are given in many of the semantic errors in which alternatives are provided. Additionally, suggestions are now provided when plt and trace probes could not be resolved. For example, kernel.trace("sched_siwtch") will suggest 'sched_switch'. - SystemTap is now smarter about error reporting. Errors from the same source are considered duplicates and suppressed. A message is displayed on exit if any errors/warnings were suppressed. - Statistics aggregate typed objects are now implemented locklessly, if the translator finds that they are only ever read (using the foreach / @count / etc. constructs) in a probe-begin/end/error. - SystemTap now supports probing inside virtual machines using the libvirt and unix schemes, e.g. stap -ve 'probe timer.s(1) { printf("hello!\n") }' \ --remote=libvirt://MyVirtualMachine Virtual machines managed by libvirt can be prepared using stapvirt. See stapvirt(1) and the --remote option in stap(1) for more details. - Systemtap now checks for and uses (when available) the .gnu_debugdata section which contains a subset of debuginfo, useful for backtraces and function probing - SystemTap map variables are now allocated with vmalloc() instead of with kmalloc(), which should cause memory to be less fragmented. - Although SystemTap itself requires elfutils 0.148+, staprun only requires elfutils 0.142+, which could be useful with the '--disable-translator' configure switch. - Under FIPS mode (/proc/sys/crypto/fips_enabled=1), staprun will refuse to load systemtap modules (since these are not normally signed with the kernel's build-time keys). This protection may be suppressed with the $STAP_FIPS_OVERRIDE environment variable. - The stap-server client & server code now enable all SSL/TLS ciphers rather than just the "export" subset. - For systems with in-kernel utrace, 'process.end' and 'thread.end' probes will hit before the target's parent process is notified of the target's death. This matches the behavior of newer kernels without in-kernel utrace. * What's new in version 2.3, 2013-07-25 - More context-accessing functions throw systemtap exceptions upon a failure, whereas in previous versions they might return non-error sentinel values like "" or "<unknown>". Use try { } / catch { } around these, or new wrapper functions such as user_string_{n_}quoted() that internally absorb exceptions. - java("org.my.MyApp") probes are now restricted to pre-existing jvm pid's with a listing in jps -l output to avoid recursive calls - The tapset [nd_]syscall.semop parameter tsops_uaddr is renamed sops_uaddr for consistency with [nd_]syscall.semtimedop. - The udp.stp tapset adds some ip-address/port variables. - A new guru-mode-only tapset function raise() is available to send signals to the current task. - Support for the standard Posix ERE named character classes has been added to the regexp engine, e.g. [:digit:], [:alpha:], ... - A substantial internal overhaul of the regexp engine has resulted in correct behaviour on further obscure edge cases. The regexp engine now implements the ERE standard and correctly passes the testsuite for the glibc regexp engine (minus portions corresponding to unimplemented features -- i.e. subexpression capture and reuse). - Alternative functions are now suggested when function probes could not be resolved. For example, kernel.function("vfs_reads") will suggest vfs_read. Other probes for which suggestions are made are module.function, process.function, and process.library.function. - Has life been a bit bland lately? Want to spice things up? Why not write a few faulty probes and feast your eyes upon the myriad of colours adorning your terminal as SystemTap softly whispers in your ear... 'parse error'. Search for '--color' in 'man stap' for more info. - The following tapset functions are deprecated in release 2.3: 'stap_NFS_CLIENT', '__getfh_inode', '_success_check', '_sock_prot_num', '_sock_fam_num', '_sock_state_num', '_sock_type_num', and '_sock_flags_num'. * What's new in version 2.2.1, 2013-05-16 * What's new in version 2.2, 2013-05-14 - Experimental support has been added for probing Java methods using Byteman 2.0 as a backend. Java method probes can target method entries, returns, or specific statements in the method as specified by line number. probe java("org.my.MyApp").class("^java.lang.Object").method("foo(int)") { println($$parms) } See java/README for information on how to set up Java/Byteman functionality. Set env STAPBM_VERBOSE=yes for more tracing. - The stap -l output and pn() tapset function's return value may be slightly different for complicated web of wildcarded/aliased probes. - The dyninst backend has improved in several aspects: - Setting custom values for global variables is now supported, both with -G when compiling a script, and from the stapdyn command line when loading a precompiled module. - A high-performance shared-memory-based transport is used for trace data. - A systemd service file and tmpfile have been added to allow systemtap-server to be managed natively by systemd. - Due to the removal of register_timer_hook in recent kernels, the behaviour of timer.profile has been changed slightly. This probe is now an alias which uses the old mechanism where possible, but falls back to perf.sw.cpu_clock or another mechanism when the kernel timer hook is not available. To require the kernel timer hook mechanism in your script, use timer.profile.tick instead of timer.profile. - The following tapset variables are deprecated in release 2.2: - The 'origin' variables in the 'generic.fop.llseek', 'generic.fop.llseek.return', and 'nfs.fop.llseek' probes. The 'origin' variable has been replaced by the 'whence' variable. - The 'page_index' variable in the 'vfs.block_sync_page' and 'vfs.buffer_migrate_page' probe aliases. - The 'write_from' and 'write_upto' variables in the '_vfs.block_prepare_write' and '_vfs.block_prepare_write.return' probe aliases. - The 'regs' variable in the 'syscall.sigaltstack', 'nd_syscall.sigaltstack', 'syscall.fork', and 'nd_syscall.fork' probe aliases. - The 'first', 'second', 'third', and 'uptr_uaddr' variables in the 'syscall.compat_sys_shmat' and 'nd_syscall.compat_sys_shmat' probe aliases. - The following tapset functions are deprecated in release 2.2: 'ppos_pos', '_dev_minor', and '_dev_major' - The folowing tapset functions used to return error strings instead of raising an error. The original behavior is deprecated in release 2.2. 'ctime', 'probemod', 'modname' * What's new in version 2.1, 2013-02-13 - EMACS and VIM editor modes for systemtap source files are included / updated. - The translator now eliminates duplicate tapset files between its preferred directory (as configured during the build with --prefix=/ or specified with the -I /path option), and files it may find under $XDG_DATA_DIRS. This should eliminate a class of conflicts between parallel system- and hand-built systemtap installations. - The translator accepts a --suppress-time-limits option, which defeats time-related constraints, to allows probe handlers to run for indefinite periods. It requires the guru mode (-g) flag to work. Add the earlier --suppress-handler-errors flag for a gung-ho "just-keep-going" attitude. - Perf event probes may now be read on demand. The counter probe is defined using the counter-name part: probe perf.type(0).config(0).counter("NAME"). The counter is read in a user space probe using @perf("NAME"), e.g. process("PROCESS").statement("func@file") {stat <<< @perf("NAME")} - Perf event probes may now be bound to a specific task using the process-name part: probe perf.type(0).config(0).process("NAME") { } If the probed process name is not specified, then it is inferred from the -c CMD argument. - Some error messages and warnings now refer to additional information that is found in man pages. These are generally named error::FOO or warning::BAR (in the 7stap man page section) and may be read via % man error::FOO % man warning::BAR - The dyninst backend has improved in several aspects: - The runtime now allows much more concurrency when probing multithreaded processes, and will also follow probes across forks. - Several new probe types are now supported, including timers, function return, and process.begin/end and process.thread.begin/end. - Semaphores for SDT probes are now set properly. - Attaching to existing processes with -x PID now works. - The foreach looping construct can now sort aggregate arrays by the user's choice of aggregating function. Previously, @count was implied. e.g.: foreach ([x,y] in array @sum +) { println(@sum(array[x,y])) } - Proof of concept support for regular expression matching has been added: if ("aqqqqqb" =~ "q*b") { ... } if ("abc" !~ "q*b") { ... } The eventual aim is to support roughly the same functionality as the POSIX Extended Regular Expressions implemented by glibc. Currently missing features include extraction of the matched string and subexpressions, and named character classes ([:alpha:], [:digit:], &c). Special thanks go to the re2c project, whose public domain code this functionality has been based on. For more info on re2c, see: http://sourceforge.net/projects/re2c/ - The folowing tapset variables are deprecated in release 2.1 and will be removed in release 2.2: - The 'send2queue' variable in the 'signal.send' probe. - The 'oldset_addr' and 'regs' variables in the 'signal.handle' probe. - The following tapset probes are deprecated in release 2.1 and will be removed in release 2.2: - signal.send.return - signal.handle.return * What's new in version 2.0, 2012-10-09 - Systemtap includes a new prototype backend, which uses Dyninst to instrument a user's own processes at runtime. This backend does not use kernel modules, and does not require root privileges, but is restricted with respect to the kinds of probes and other constructs that a script may use. Users from source should configure --with-dyninst and install a fresh dyninst snapshot such as that in Fedora rawhide. It may be necessary to disable conflicting selinux checks; systemtap will advise. Select this new backend with the new stap option --runtime=dyninst and a -c target process, along with normal options. (-x target processes are not supported in this prototype version.) For example: stap --runtime=dyninst -c 'stap -l begin' \ -e 'probe process.function("main") { println("hi from dyninst!") }' - To aid diagnosis, when a kernel panic occurs systemtap now uses the panic_notifier_list facility to dump a summary of its trace buffers to the serial console. - The systemtap preprocessor now has a simple macro facility as follows: @define add(a,b) %( ((@a)+(@b)) %) @define probegin(x) %( probe begin { @x } %) @probegin( foo = @add(40, 2); print(foo) ) Macros defined in the user script and regular tapset .stp files are local to the file. To get around this, the tapset library can define globally visible 'library macros' inside .stpm files. (A .stpm file must contain a series of @define directives and nothing else.) The status of the feature is experimental; semantics of macroexpansion may change (unlikely) or expand in the future. - Systemtap probe aliases may be used with additional suffixes attached. The suffixes are passed on to the underlying probe point(s) as shown below: probe foo = bar, baz { } probe foo.subfoo.option("gronk") { } // expands to: bar.subfoo.option("gronk"), baz.subfoo.option("gronk") In practical terms, this allows us to specify additional options to certain tapset probe aliases, by writing e.g. probe syscall.open.return.maxactive(5) { ... } - To support the possibility of separate kernel and dyninst backends, the tapsets have been reorganized into separate folders according to backend. Thus kernel-specific tapsets are located under linux/, the dyninst-specific ones under dyninst/ - The backtrace/unwind tapsets have been expanded to allow random access to individual elements of the backtrace. (A caching mechanism ensures that the backtrace computation run at most once for each time a probe fires, regardless of how many times or what order the query functions are called in.) New tapset functions are: stack/ustack - return n'th element of backtrace callers/ucallers - return first n elements of backtrace print_syms/print_usyms - print full information on a list of symbols sprint_syms/sprint_usyms - as above, but return info as a string The following existing functions have been superseded by print_syms() et al.; new scripts are recommended to avoid using them: print_stack() print_ustack() sprint_stack() sprint_ustack() - The probefunc() tapset function is now myproc-unprivileged, and can now be used in unprivileged scripts for such things as profiling in userspace programs. For instance, try running systemtap.examples/general/para-callgraph.stp in unprivileged mode with a stapusr-permitted probe. The previous implementation of probefunc() is available with "stap --compatible=1.8". - Preprocessor conditional to vary code based on script privilege level: unprivileged -- %( systemtap_privilege == "stapusr" %? ... %) privileged -- %( systemtap_privilege != "stapusr" %? ... %) or, alternately %( systemtap_privilege == "stapsys" || systemtap_privilege == "stapdev" %? ... %) - To ease migration to the embedded-C locals syntax introduced in 1.8 (namely, STAP_ARG_* and STAP_RETVALUE), the old syntax can now be re-enabled on a per-function basis using the /* unmangled */ pragma: function add_foo:long(a:long, b:long) %{ /* unmangled */ THIS->__retvalue = THIS->a + STAP_ARG_b; %} Note that both the old and the new syntax may be used in an /* unmangled */ function. Functions not marked /* unmangled */ can only use the new syntax. - Adjacent string literals are now glued together irrespective of intervening whitespace or comments: "foo " "bar" --> "foo bar" "foo " /* comment */ "bar" --> "foo bar" Previously, the first pair of literals would be glued correctly, while the second would cause a syntax error. * What's new in version 1.8, 2012-06-17 - staprun accepts a -T timeout option to allow less frequent wake-ups to poll for low-throughput output from scripts. - When invoked by systemtap, the kbuild $PATH environment is sanitized (prefixed with /usr/bin:/bin:) in an attempt to exclude compilers other than the one the kernel was presumed built with. - Printf formats can now use "%#c" to escape non-printing characters. - Pretty-printed bitfields use integers and chars use escaped formatting for printing. - The systemtap compile-server and client now support IPv6 networks. - IPv6 addresses may now be specified on the --use-server option and will be displayed by --list-servers, if the avahi-daemon service is running and has IPv6 enabled. - Automatic server selection will automatically choose IPv4 or IPv6 servers according to the normal server selection criteria when avahi-daemon is running. One is not preferred over the other. - The compile-server will automatically listen on IPv6 addresses, if available. - To enable IPv6 in avahi-daemon, ensure that /etc/avahi/avahi-daemon.conf contains an active "use-ipv6=yes" line. After adding this line run "service avahi-daemon restart" to activate IPv6 support. - See man stap(1) for details on how to use IPv6 addresses with the --use-server option. - Support for DWARF4 .debug_types sections (for executables and shared libraries compiled with recent GCC's -gdwarf-4 / -fdebug-types-section). PR12997. SystemTap now requires elfutils 0.148+, full .debug_types support depends on elfutils 0.154+. - Systemtap modules are somewhat smaller & faster to compile. Their debuginfo is now suppressed by default; use -B CONFIG_DEBUG_INFO=y to re-enable. - @var now an alternative language syntax for accessing DWARF variables in uprobe and kprobe handlers (process, kernel, module). @var("somevar") can be used where $somevar can be used. The @var syntax also makes it possible to access non-local, global compile unit (CU) variables by specifying the CU source file as follows @var("somevar@some/src/file.c"). This will provide the target variable value of global "somevar" as defined in the source file "some/src/file.c". The @var syntax combines with all normal features of DWARF target variables like @defined(), @entry(), [N] array indexing, field access through ->, taking the address with the & prefix and shallow or deep pretty printing with a $ or $$ suffix. - Stap now has resource limit options: --rlimit-as=NUM --rlimit-cpu=NUM --rlimit-nproc=NUM --rlimit-stack=NUM --rlimit-fsize=NUM All resource limiting has been moved from the compile server to stap itself. When running the server as "stap-server", default resource limit values are specified in ~stap-server/.systemtap/rc. - Bug CVE-2012-0875 (kernel panic when processing malformed DWARF unwind data) is fixed. - The systemtap compile-server now supports multiple concurrent connections. Specify the desired maximum number of concurrent connections with the new stap-server/stap-serverd --max-threads option. Specify a value of '0' to tell the server not to spawn any new threads (handle all connections serially in the main thread). The default value is the number of processor cores on the host. - The following tapset functions are deprecated in release 1.8 and will be removed in release 1.9: daddr_to_string() - SystemTap now mangles local variables to avoid collisions with C headers included by tapsets. This required a change in how embedded-C functions access local parameters and the return value slot. Instead of THIS->foo in an embedded-C function, please use the newly defined macro STAP_ARG_foo (substitute the actual name of the argument for 'foo'); instead of THIS->__retvalue, use the newly defined STAP_RETVALUE. All of the tapsets and test cases have been adapted to use this new notation. If you need to run code which uses the old THIS-> notation, run stap with the --compatible=1.7 option. - There is updated support for user-space probing against kernels >= 3.5, which have no utrace but do have the newer inode-uprobes work by Srikar Dronamraju and colleagues. For kernels < 3.5, the following 3 sets of kernel patches would need to be backported to your kernel to use this preliminary user-space probing support: - inode-uprobes patches: - 2b144498350860b6ee9dc57ff27a93ad488de5dc - 7b2d81d48a2d8e37efb6ce7b4d5ef58822b30d89 - a5f4374a9610fd7286c2164d4e680436727eff71 - 04a3d984d32e47983770d314cdb4e4d8f38fccb7 - 96379f60075c75b261328aa7830ef8aa158247ac - 3ff54efdfaace9e9b2b7c1959a865be6b91de96c - 35aa621b5ab9d08767f7bc8d209b696df281d715 - 900771a483ef28915a48066d7895d8252315607a - e3343e6a2819ff5d0dfc4bb5c9fb7f9a4d04da73 - exec tracepoint kernel patch: - 4ff16c25e2cc48cbe6956e356c38a25ac063a64d - task_work_add kernel patches: - e73f8959af0439d114847eab5a8a5ce48f1217c4 - 4d1d61a6b203d957777d73fcebf19d90b038b5b2 - 413cd3d9abeaef590e5ce00564f7a443165db238 - dea649b8ac1861107c5d91e1a71121434fc64193 - f23ca335462e3c84f13270b9e65f83936068ec2c * What's new in version 1.7, 2012-02-01 - Map inserting and deleting is now significantly faster due to improved hashing and larger hash tables. The hashes are also now randomized to provide better protection against deliberate collision attacks. - Formatted printing is faster by compiling the formatting directives to C code rather than interpreting at run time. - Systemtap loads extra command line options from $SYSTEMTAP_DIR/rc ($HOME/.systemtap/rc by default) before the normal argc/argv. This may be useful to activate site options such as --use-server or --download-debuginfo or --modinfo. - The stap-server has seen many improvements, and is no longer considered experimental. - The stap-server service (initscript) now supports four new options: -D MACRO[=VALUE] --log LOGFILE --port PORT-NUMBER --SSL CERT-DATABASE These allow the specification of macro definitions to be passed to stap by the server, the location of the log file, network port number and NSS certificate database location respectively. These options are also supported within individual server configuration files. See stap-server and initscript/README.stap-server for details. The stap-server is no longer activated by default. - process("PATH").[library("PATH")].function("NAME").exported probes are now supported to filter function() to only exported instances. - The translator supports a new --suppress-handler-errors option, which causes most runtime errors to be turned into quiet skipped probes. This also disables the MAXERRORS and MAXSKIPPED limits. - Translator warnings have been standardized and controlled by the -w / -W flags. - The translator supports a new --modinfo NAME=VALUE option to emit additional MODULE_INFO(n,v) macros into the generated code. - There is no more fixed maximum number of VMA pages that will be tracked at runtime. This reduces memory use for those scripts that don't need any, or only limited target process VMA tracking and allows easier system wide probes inspecting shared library variables and/or user backtraces. stap will now silently ignore -DTASK_FINDER_VMA_ENTRY_ITEMS. - The tapset functions remote_id() and remote_uri() identify the member of a swarm of "stap --remote FOO --remote BAR baz.stp" concurrent executions. - Systemtap now supports a new privilege level and group, "stapsys", which is equivalent to the privilege afforded by membership in the group "stapdev", except that guru mode (-g) functionality may not be used. To support this, a new option, --privilege=[stapusr|stapsys|stapdev] has been added. --privilege=stapusr is equivalent to specifying the existing --unprivileged option. --privilege=stapdev is the default. See man stap(1) for details. - Scripts that use kernel.trace("...") probes compile much faster. - The systemtap module cache is cleaned less frequently, governed by the number of seconds in the $SYSTEMTAP_DIR/cache/cache_clean_interval_s file. - SDT can now define up to 12 arguments in a probe point. - Parse errors no longer generate a cascade of false errors. Instead, a parse error skips the rest of the current probe or function, and resumes at the next one. This should generate fewer and better messages. - Global array wrapping is now supported for both associative and statistics typed arrays using the '%' character to signify a wrapped array. For example, 'global foo%[100]' would allow the array 'foo' to be wrapped if more than 100 elements are inserted. - process("PATH").library("PATH").plt("NAME") probes are now supported. Wildcards are supported in the plt-name part, to refer to any function in the program linkage table which matches the glob pattern and the rest of the probe point. - A new option, --dump-probe-types, will dump a list of supported probe types. If --unprivileged is also specified, the list will be limited to probe types which are available to unprivileged users. - Systemtap can now automatically download the required debuginfo using abrt. The --download-debuginfo[=OPTION] can be used to control this feature. Possible values are: 'yes', 'no', 'ask', and a positive number representing the timeout desired. The default behavior is to not automatically download the debuginfo. - The translator has better support for probing C++ applications by better undertanding of compilation units, nested types, templates, as used in probe point and @cast constructs. - On 2.6.29+ kernels, systemtap can now probe kernel modules that arrive and/or depart during the run-time of a session. This allows probing of device driver initialization functions, which had formerly been blocklisted. - New tapset functions for cpu_clock and local_clock access were added. - There is some limited preliminary support for user-space probing against kernels such as linux-next, which have no utrace but do have the newer inode-uprobes work by Srikar Dronamraju and colleagues. - The following probe types are deprecated in release 1.7 and will be removed in release 1.8: kernel.function(number).inline module(string).function(number).inline process.function(number).inline process.library(string).function(number).inline process(string).function(number).inline process(string).library(string).function(number).inline - The systemtap-grapher is deprecated in release 1.7 and will be removed in release 1.8. - The task_backtrace() tapset function was deprecated in 1.6 and has been removed in 1.7. - MAXBACKTRACE did work in earlier releases, but has now been documented in the stap 1 manual page. - New tapset function probe_type(). Returns a short string describing the low level probe handler type for the current probe point. - Both unwind and symbol data is now only collected and emitted for scripts actually using backtracing or function/data symbols. Tapset functions are marked with /* pragma:symbols */ or /* pragma:unwind */ to indicate they need the specific data. - Kernel backtraces can now be generated for non-pt_regs probe context if the kernel support dump_trace(). This enables backtraces from certain timer probes and tracepoints. - ubacktrace() should now also work for some kernel probes on x86 which can use the dwarf unwinder to recover the user registers to provide more accurate user backtraces. - For s390x the systemtap runtime now properly splits kernel and user addresses (which are in separate address spaces on that architecture) which enable user space introspection. - ppc and s390x now supports user backtraces through the DWARF unwinder. - ppc now handles function descriptors as symbol names correctly. - arm support kernel backtraces through the DWARF unwinder. - arm now have a uprobes port which enables user probes. This still requires some kernel patches (user_regsets and tracehook support for arm). - Starting in release 1.7, these old variables will be deprecated: - The 'pid' variable in the 'kprocess.release' probe has been deprecated in favor of the new 'released_pid' variable. - The 'args' variable in the '_sunrpc.clnt.create_client.rpc_new_client_inline' probe has been deprecated in favor of the new internal-only '__args' variable. - Experimental support for recent kernels without utrace has been added for the following probe types: process(PID).begin process("PATH").begin process.begin process(PID).thread.begin process("PATH").thread.begin process.thread.begin process(PID).end process("PATH").end process.end process(PID).thread.end process("PATH").thread.end process.thread.end process(PID).syscall process("PATH").syscall process.syscall process(PID).syscall.return process("PATH").syscall.return process.syscall.return - staprun disables kprobe-optimizations in recent kernels, as problems were found. (PR13193) * What's new in version 1.6, 2011-07-25 - Security fixes for CVE-2011-2503: read instead of mmap to load modules, CVE-2011-2502: Don't allow path-based auth for uprobes - The systemtap compile-server no longer uses the -k option when calling the translator (stap). As a result, the server will now take advantage of the module cache when compiling the same script more than once. You may observe an improvement in the performance of the server in this situation. - The systemtap compile-server and client now each check the version of the other, allowing both to adapt when communicating with a down-level counterpart. As a result, all version of the client can communicate with all versions of the server and vice-versa. Client will prefer newer servers when selecting a server automatically. - SystemTap has improved support for the ARM architecture. The kread() and kwrite() operations for ARM were corrected allowing many of the tapsets probes and function to work properly on the ARM architecture. - Staprun can now rename the module to a unique name with the '-R' option before inserting it. Systemtap itself will also call staprun with '-R' by default. This allows the same module to be inserted more than once, without conflicting duplicate names. - Systemtap error messages now provide feedback when staprun or any other process fails to launch. This also specifically covers when the user doesn't have the proper permissions to launch staprun. - Systemtap will now map - to _ in module names. Previously, stap -L 'module("i2c-core").function("*")' would be empty. It now returns a list had stap -L 'module("i2c_core").function("*") been specified. - Systemtap now fills in missing process names to probe points, to avoid having to name them twice twice: % stap -e 'probe process("a.out").function("*") {}' -c 'a.out ...' Now the probed process name is inferred from the -c CMD argument. % stap -e 'probe process.function("*") {}' -c 'a.out ...' - stap -L 'process("PATH").syscall' will now list context variables - Depends on elfutils 0.142+. - Deprecated task_backtrace:string (task:long). This function will go away after 1.6. Please run your scripts with stap --check-version. * What's new in version 1.5, 2011-05-23 - Security fixes for CVE-2011-1781, CVE-2011-1769: correct DW_OP_{mod,div} division-by-zero bug - The compile server and its related tools (stap-gen-ert, stap-authorize-cert, stap-sign-module) have been re-implemented in C++. Previously, these components were a mix of bash scripts and C code. These changes should be transparent to the end user with the exception of NSS certificate database password prompting (see below). The old implementation would prompt more than once for the same password in some situations. - eventcount.stp now allows for event counting in the format of 'stap eventcount.stp process.end syscall.* ...', and also reports corresponding event tid's. - Systemtap checks that the build-id of the module being probed matches the build-id saved in the systemtap module. Invoking systemtap with -DSTP_NO_BUILDID_CHECK will bypass this build-id runtime verification. See man ld(1) for info on --build-id. - stapio will now report if a child process has an abnormal exit along with the associated status or signal. - Compiler optimization may sometimes result in systemtap not being able to access a user-space probe argument. Compiling the application with -DSTAP_SDT_ARG_CONSTRAINT=nr will force the argument to be an immediate or register value which should enable systemtap to access the argument. - GNU Gettext has now been intergrated with systemtap. Our translation page can be found at http://www.transifex.net/projects/p/systemtap/ . "make update-po" will generate the necessary files to use translated messages. Please refer to the po/README file for more info and please consider contributing to this I18N effort! - The new addr() function returns the probe's instruction pointer. - process("...").library("...") probes are now supported. Wildcards are supported in the library-name part, to refer to any shared library that is required by process-name, which matches the glob pattern and the rest of the probe point. - The "--remote USER@HOST" functionality can now be specified multiple times to fan out on multiple targets. If the targets have distinct kernel and architecture configurations, stap will automatically build the script appropriately for each one. This option is also no longer considered experimental. - The NSS certificate database generated for use by the compile server is now generated with no password. Previously, a random password was generated and used to access the database. This change should be transparent to most users. However, if you are prompted for a password when using systemtap, then running $libexecdir/stap-gen-cert should correct the problem. - The timestamp tapset includes jiffies() and HZ() for lightweight approximate timekeeping. - A powerful new command line option --version has been added. - process.mark now supports $$parms for reading probe parameters. - A new command line option, --use-server-on-error[=yes|no] is available for stap. It instructs stap to retry compilation of a script using a compile server if it fails on the local host. The default setting is 'no'. - The following deprecated tools have been removed: stap-client stap-authorize-server-cert stap-authorize-signing-cert stap-find-or-start-server stap-find-servers Use the --use-server, --trust-server and --list-servers options of stap instead. * What's new in version 1.4, 2011-01-17 - Security fixes for CVE-2010-4170, CVE-2010-4171: staprun module loading/unloading - A new /* myproc-unprivileged */ marker is now available for embedded C code and and expressions. Like the /* unprivileged */ marker, it makes the code or expression available for use in unprivileged mode (see --unprivileged). However, it also automatically adds a call to assert_is_myproc() to the code or expression, thus, making it available to the unprivileged user only if the target of the current probe is within the user's own process. - The experimental "--remote USER@HOST" option will run pass 5 on a given ssh host, after building locally (or with --use-server) for that target. - Warning messages from the script may now be suppressed with the stap and/or staprun -w option. By default, duplicate warning messages are suppressed (up to a certain limit). With stap --vp 00002 and above, the duplicate elimination is defeated. - The print_ubacktrace and usym* functions attempt to print the full path of the user-space binaries' paths, instead of just the basename. The maximum saved path length is set by -DTASK_FINDER_VMA_ENTRY_PATHLEN, default 64. Warning messages are produced if unwinding fails due to a missing 'stap -d MODULE' option, providing preloaded unwind data. - The new tz_ctime() tapset function prints times in the local time zone. - More kernel tracepoints are accessible to the kernel.trace("...") mechanism, if kernel source trees or debuginfo are available. These formerly "hidden" tracepoints are those that are declared somewhere other than the usual include/linux/trace/ headers, such as xfs and kvm. - debuginfo-based process("...").function/.statement/.mark probes support wildcards in the process-name part, to refer to any executable files that match the glob pattern and the rest of the probe point. - The -t option now displays information per probe-point rather than a summary for each probe. It also now shows the derivation chain for each probe-point. - A rewrite of the sys/sdt.h header file provides zero-cost startup (few or no ELF relocations) for the debuginfo-less near-zero-cost runtime probes. Binaries compiled with earlier sdt.h versions remain supported. The stap -L (listing) option now lists parameters for sys/sdt.h markers. - The implementation of the integrated compile-server client has been extended. o --use-server now accepts an argument representing a particular server and may be specified more than once. o --list-servers now accepts an expanded range of arguments. o a new --trust-servers option has been added to stap to replace several old certificate-management scripts. o The following tools are now deprecated and will be removed in release 1.5: stap-client stap-authorize-server-cert stap-authorize-signing-cert stap-find-or-start-server stap-find-servers See man stap(1) for complete details. - The compile-server now returns the uprobes.ko to the client when it is required by the script being compiled. The integrated compile-server client now makes it available to be loaded by staprun. The old (deprecated) stap-client does not do this. - process probes with scripts as the target are recognized by stap and the interpreter would be selected for probing. - Starting in release 1.5, these old variables/functions will be deprecated and will only be available when the '--compatible=1.4' flag is used: - In the 'syscall.add_key' probe, the 'description_auddr' variable has been deprecated in favor of the new 'description_uaddr' variable. - In the 'syscall.fgetxattr', 'syscall.fsetxattr', 'syscall.getxattr', 'syscall.lgetxattr', and 'syscall.lremovexattr' probes, the 'name2' variable has been deprecated in favor of the new 'name_str' variable. - In the 'nd_syscall.accept' probe the 'flag_str' variable has been deprecated in favor of the new 'flags_str' variable. - In the 'nd_syscall.dup' probe the 'old_fd' variable has been deprecated in favor of the new 'oldfd' variable. - In the 'nd_syscall.fgetxattr', 'nd_syscall.fremovexattr', 'nd_syscall.fsetxattr', 'nd_syscall.getxattr', and 'nd_syscall.lremovexattr' probes, the 'name2' variable has been deprecated in favor of the new 'name_str' variable. - The tapset alias 'nd_syscall.compat_pselect7a' was misnamed. It should have been 'nd_syscall.compat_pselect7' (without the trailing 'a'). - The tapset function 'cpuid' is deprecated in favor of the better known 'cpu'. - In the i386 'syscall.sigaltstack' probe, the 'ussp' variable has been deprecated in favor of the new 'uss_uaddr' variable. - In the ia64 'syscall.sigaltstack' probe, the 'ss_uaddr' and 'oss_uaddr' variables have been deprecated in favor of the new 'uss_uaddr' and 'uoss_uaddr' variables. - The powerpc tapset alias 'syscall.compat_sysctl' was deprecated and renamed 'syscall.sysctl32'. - In the x86_64 'syscall.sigaltstack' probe, the 'regs_uaddr' variable has been deprecated in favor of the new 'regs' variable. * What's new in version 1.3, 2010-07-21 - The uprobes kernel module now has about half the overhead when probing NOPs, which is particularly relevant for sdt.h markers. - New stap option -G VAR=VALUE allows overriding global variables by passing the settings to staprun as module options. - The tapset alias 'syscall.compat_pselect7a' was misnamed. It should have been 'syscall.compat_pselect7' (without the trailing 'a'). Starting in release 1.4, the old name will be deprecated and will only be available when the '--compatible=1.3' flag is used. - A new procfs parameter .umask(UMASK) which provides modification of file permissions using the proper umask value. Default file permissions for a read probe are 0400, 0200 for a write probe, and 0600 for a file with a read and write probe. - It is now possible in some situations to use print_ubacktrace() to get a user space stack trace from a kernel probe point. e.g. for user backtraces when there is a pagefault: $ stap -d /bin/sort --ldd -e 'probe vm.pagefault { if (pid() == target()) { printf("pagefault @0x%x\n", address); print_ubacktrace(); } }' -c /bin/sort [...] pagefault @0x7fea0595fa70 0x000000384f07f958 : __GI_strcmp+0x12b8/0x1440 [libc-2.12.so] 0x000000384f02824e : __gconv_lookup_cache+0xee/0x5a0 [libc-2.12.so] 0x000000384f021092 : __gconv_find_transform+0x92/0x2cf [libc-2.12.so] 0x000000384f094896 : __wcsmbs_load_conv+0x106/0x2b0 [libc-2.12.so] 0x000000384f08bd90 : mbrtowc+0x1b0/0x1c0 [libc-2.12.so] 0x0000000000404199 : ismbblank+0x39/0x90 [sort] 0x0000000000404a4f : inittables_mb+0xef/0x290 [sort] 0x0000000000406934 : main+0x174/0x2510 [sort] 0x000000384f01ec5d : __libc_start_main+0xfd/0x1d0 [libc-2.12.so] 0x0000000000402509 : _start+0x29/0x2c [sort] [...] - New tapset functions to get a string representation of a stack trace: sprint_[u]backtrace() and sprint_[u]stack(). - New tapset function to get the module (shared library) name for a user space address umodname:string(long). The module name will now also be in the output of usymdata() and in backtrace addresses even when they were not given with -d at the command line. - Kernel backtraces are now much faster (replaced a linear search with a binary search). - A new integrated compile-server client is now available as part of stap. o 'stap --use-server ...' is equivalent to 'stap-client ...' o 'stap --list-servers' is equivalent to 'stap-find-servers' o 'stap --list-servers=online' is equivalent to 'stap-find-servers --all' o stap-client and its related tools will soon be deprecated. o the nss-devel and avahi-devel packages are required for building stap with the integrated client (checked during configuration). o nss and avahi are required to run the integrated client. - A new operator @entry is available for automatically saving an expression at entry time for use in a .return probe. probe foo.return { println(get_cycles() - @entry(get_cycles())) } - Probe $target variables and @cast() can now use a suffix to print complex data types as strings. Use a single '$' for a shallow view, or '$$' for a deeper view that includes nested types. For example, with fs_struct: $fs$ : "{.users=%i, .lock={...}, .umask=%i, .in_exec=%i, .root={...}, .pwd={...}}" $fs$$ : "{.users=%i, .lock={.raw_lock={.lock=%u}}, .umask=%i, .in_exec=%i, .root={.mnt=%p, .dentry=%p}, .pwd={.mnt=%p, .dentry=%p}}" - The <sys/sdt.h> user-space markers no longer default to an implicit MARKER_NAME_ENABLED() semaphore check for each marker. To check for enabled markers use a .d declaration file, then: if (MARKER_NAME_ENABLED()) MARKER_NAME() - Hyphenated <sys/sdt.h> marker names such as process(...).mark("foo-bar") are now accepted in scripts. They are mapped to the double-underscore form ("foo__bar"). - More robust <sys/sdt.h> user-space markers support is included. For some platforms (x86*, ppc*), this can let systemtap probe the markers without debuginfo. This implementation also supports preserving the "provider" name associated with a marker: probe process("foo").provider("bar").mark("baz") to match STAP_PROBE<n>(bar, baz <...>) (Compile with -DSTAP_SDT_V1 to revert to the previous implementation. Systemtap supports pre-existing or new binaries using them.) - Embedded-C may be used within expressions as values, when in guru mode: num = %{ LINUX_VERSION_CODE %} // int64_t name = %{ /* string */ THIS_MODULE->name %} // const char* printf ("%s %x\n", name, num) The usual /* pure */, /* unprivileged */, and /* guru */ markers may be used as with embedded-C functions. - By default the systemtap-runtime RPM builds now include a shared library, staplog.so, that allows crash to extract systemtap data from a vmcore image. - Iterating with "foreach" can now explicitly save the value for the loop. foreach(v = [i,j] in array) printf("array[%d,%s] = %d\n", i, j, v /* array[i,j] */) - The new "--ldd" option automatically adds any additional shared libraries needed by probed or -d-listed userspace binaries to the -d list, to enable symbolic backtracing through them. Similarly, the new "--all-modules" option automatically adds any currently loaded kernel modules (listed in /proc/modules) to the -d list. - A new family of set_kernel_* functions make it easier for gurus to write new values at arbitrary memory addresses. - Probe wildcards can now use '**' to cross the '.' separator. $ stap -l 'sys**open' syscall.mq_open syscall.open - Backward compatibility flags (--compatible=VERSION, and matching script preprocessing predicate %( systemtap_v CMP "version" %) and a deprecation policy are being introduced, in case future tapset/language changes break valid scripts. * What's new in version 1.2, 2010-03-22 - Prototype support for "perf events", where the kernel supports the 2.6.33 in-kernel API. Probe points may refer to low-level perf_event_attr type/config numbers, or to a number of aliases defined in the new perf.stp tapset: probe perf.sw.cpu_clock, perf.type(0).config(4) { } - Type-casting can now use multiple headers to resolve codependencies. @cast(task, "task_struct", "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask - Tapset-related man pages have been renamed. 'man -k 3stap' should show the installed list, which due to prefixing should no longer collide over ordinary system functions. - User space marker arguments no longer use volatile if the version of gcc, which must be at least 4.5.0, supports richer DWARF debuginfo. Use cflags -DSTAP_SDT_VOLATILE=volatile or -DSTAP_SDT_VOLATILE= when building the sys/sdt.h application to override this one way or another. - A new construct for error handling is available. It is similar to c++ exception catching, using try and catch as new keywords. Within a handler or function, the following is valid and may be nested: try { /* arbitrary statements */ } catch (er) { /* e.g. println("caught error ", er) */ } - A new command line flag '-W' forces systemtap to abort translation of a script if any warnings are produced. It is similar to gcc's -Werror. (If '-w' is also supplied to suppress warnings, it wins.) - A new predicate @defined is available for testing whether a particular $variable/expression is resolvable at translate time: probe foo { if (@defined($bar)) log ("$bar is available here") } - Adjacent string literals are glued together, making this construct valid: probe process("/usr" @1 "/bin").function("*") { ... } - In order to limit potential impact from future security problems, the stap-server process does not permit its being launched as root. - On recent kernels, for some architectures/configurations, hardware breakpoint probes are supported. The probe point syntax is: probe kernel.data(ADDRESS).write probe kernel.data(ADDRESS).length(LEN).write probe kernel.data("SYMBOL_NAME").write * What's new in version 1.1, 2010-01-15 - New tracepoint based tapset for memory subsystem. - The loading of signed modules by staprun is no longer allowed for ordinary, unprivileged users. This means that only root, members of the group 'stapdev' and members of the group 'stapusr' can load systemtap modules using staprun, stap or stap-client. The minimum privilege required to run arbitrary --unprivileged scripts is now 'stapusr' membership. - The stap-server initscript is available. This initscript allows you to start systemtap compile servers as a system service and to manage these servers as a group or individually. The stap-server initscript is installed by the systemtap-server rpm. The build directory for the uprobes module (/usr/share/systemtap/runtime/uprobes) is made writable by the 'stap-server' group. All of the files generated when building the uprobes module, including the digital signature, are also writable by members of stap-server. See initscript/README.stap-server for details. - Some of the compile server client, server and certificate management tools have been moved from $bindir to $libexecdir/systemtap. You should use the new stap-server script or the stap-server initscript for server management where possible. The stap-server script provides the same functionality as the stap-server initscript except that the servers are run by the invoking user by default as opposed to servers started by the stap-server initscript which are run by the user stap-server by default. See stap-server(8) for more information. You may continue to use these tools by adding $libexecdir/systemtap to your path. You would need to do this, for example, if you are not root, you want to start a compile server and you are not running systemtap from a private installation. In this case you still need to use stap-start-server. - Any diagnostic output line that starts with "ERROR", as in error("foo"), will promote a "Pass 5: run failed", and the return code is 1. - Systemtap now warns about global variables being referenced from other script files. This aims to protect against unintended local-vs-global namespace collisions such as: % cat some_tapset.stp probe baz.one = bar { foo = $foo; bar = $bar } % cat end_user_script.stp global foo # intended to be private variable probe timer.s(1) { foo ++ } probe baz.* { println(foo, pp()) } % stap end_user_script.stp WARNING: cross-file global variable reference to foo from some_tapset.stp - Preprocessor conditional for kernel configuration testing: %( CONFIG_foo == "y" %? ... %) - ftrace(msg:string) tapset function to send strings to the system-wide ftrace ring-buffer (if any). - Better support for richer DWARF debuginfo output from GCC 4.5 (variable tracking assignments). Kernel modules are now always resolved against all their dependencies to find any info referring to missing symbols. DW_AT_const_value is now supported when no DW_AT_location is available. * What's new in verson 1.0, 2009-09-22 - process().mark() probes now use an enabling semaphore to reduce the computation overhead of dormant probes. - The function spec for dwarf probes now supports C++ scopes, so you can limit the probes to specific namespaces or classes. Multiple scopes can be specified, and they will be matched progressively outward. probe process("foo").function("std::vector<*>::*") { } probe process("foo").function("::global_function") { } - It is now possible to cross-compile systemtap scripts for foreign architectures, using the new '-a ARCH' and '-B OPT=VALUE' flags. For example, put arm-linux-gcc etc. into your $PATH, and point systemtap at the target kernel build tree with: stap -a arm -B CROSS_COMPILE=arm-linux- -r /build/tree [...] The -B option is passed to kbuild make. -r identifies the already configured/built kernel tree and -a its architecture (kbuild ARCH=...). Systemtap will infer -p4. - Cross compilation using the systemtap client and server - stap-start-server now accepts the -r, -R, -I, -B and -a options in order to start a cross compiling server. The server will correctly advertise itself with respect to the kernel release and architecture that it compiles for. - When specified on stap-client, the -r and -a options will be considered when searching for a suitable server. - When using the systemtap client and server udp port 5353 must be open in your firewall in order for the client to find servers using avahi-browse. Also the systemtap server will choose a random port in the range 1024-63999 for accepting ssl connections. - Support for unprivileged users: *********************************************************************** * WARNING!!!!!!!!!! * * This feature is EXPERIMENTAL at this time and should be used with * * care. This feature allows systemtap kernel modules to be loaded by * * unprivileged users. The user interface and restrictions will change * * as this feature evolves. * *********************************************************************** - Systemtap modules generated from scripts which use a restricted subset of the features available may be loaded by staprun for unprivileged users. Previously, staprun would load modules only for root or for members of the groups stapdev and stapusr. - Using the --unprivileged option on stap enables translation-time checking for use by unprivileged users (see restrictions below). - All modules deemed suitable for use by unprivileged users will be signed by the systemtap server when --unprivileged is specified on stap-client. See module signing in release 0.9.8 and stap-server in release 0.9 below. - Modules signed by trusted signers (servers) and verified by staprun will be loaded by staprun regardless of the user's privilege level. - The system administrator asserts the trustworthiness of a signer (server) by running stap-authorize-signing-cert <cert-file> as root, where the <cert-file> can be found in ~<user>/.systemtap/ssl/server/stap.cert for servers started by ordinary users and in $sysconfdir/systemtap/ssl/server/stap.cert for servers started by root. - Restrictions are intentionally strict at this time and may be relaxed in the future: - probe points are restricted to: begin, begin(n), end, end(n), error, error(n), never, timer.{jiffies,s,sec,ms,msec,us,usec,ns,nsec}(n)*, timer.hz(n), process.* (for processes owned by the user). - use of embedded C code is not allowed. - use of tapset functions is restricted. - some tapset functions may not be used at all. A message will be generated at module compilation time. - some actions by allowed tapset functions may only be performed in the context of the user's own process. A runtime fault will occur in these situations, for example, direct memory access. - The is_myproc() tapset function has been provided so that tapset writers for unprivileged users can check that the context is of the users own process before attempting these actions. - accessing the kernel memory space is not allowed. - The following command line options may not be used by stap-client -g, -I, -D, -R, -B - The following environment variables are ignored by stap-client: SYSTEMTAP_RUNTIME, SYSTEMTAP_TAPSET, SYSTEMTAP_DEBUGINFO_PATH - nss and nss-tools are required to use this feature. - Support output file switching by SIGUSR2. Users can command running stapio to switch output file by sending SIGUSR2. - Memory consumption for scripts involving many uprobes has been dramatically reduced. - The preprocessor now supports || and && in the conditions. e.g. %( arch == "x86_64" || arch == "ia64" %: ... %) - The systemtap notion of "architecture" now matches the kernel's, rather than that of "uname -m". This means that 32-bit i386 family are all known as "i386" rather than "i386" or "i686"; "ppc64" as "powerpc"; "s390x" as "s390", and so on. This is consistent between the new "-a ARCH" flag and the script-level %( arch ... %) conditional. - It is now possible to define multiple probe aliases with the same name. A probe will expand to all matching aliases. probe foo = bar { } probe foo = baz { } probe foo { } # expands twice, once to bar and once to baz - A new experimental transport mechanism, using ftrace's ring_buffer, has been added. This may become the default transport mechanism in future versions of systemtap. To test this new transport mechanism, define 'STP_USE_RING_BUFFER'. - Support for recognizing DW_OP_{stack,implicit}_value DWARF expressions as emitted by GCC 4.5. * What's new in version 0.9.9, 2009-08-04 - Systemwide kernel .function.return (kretprobe) maxactive defaults may be overridden with the -DKRETACTIVE=nnn parameter. - Translation pass 2 is significantly faster by avoiding unnecessary searching through a kernel build/module directory tree. - When compiled against elfutils 0.142 systemtap now handles the new DW_OP_call_frame_CFA generated by by GCC. - uprobes and ustack() are more robust when used on applications that depend on prelinked/separate debuginfo shared libraries. - User space PROBE marks are not always found with or without separate debuginfo. The .probes section itself is now always put in the main elf file and marked as allocated. When building pic code the section is marked writable. The selinux memory check problems seen with programs using STAP_PROBES is fixed. - statement() probes can now override "address not at start of statement" errors in guru mode. They also provide alternative addresses to use in non-guru mode. - The stapgraph application can generate graphs of data and events emitted by systemtap scripts in real time. Run "stapgraph testsuite/systemtap.examples/general/grapher.stp" for an example of graphing the system load average and keyboard events. - Dwarf probes now show parameters and local variables in the verbose listing mode (-L). - Symbol aliases are now resolved to their canonical dwarf names. For example, probing "malloc" in libc resolves to "__libc_malloc". - The syntax for dereferencing $target variables and @cast() gained new capabilities: - Array indexes can now be arbitrary numeric expressions. - Array subscripts are now supported on pointer types. - An '&' operator before a @cast or $target returns the address of the final component, especially useful for nested structures. - For reading all probe variables, kernel.mark now supports $$vars and $$parms, and process.syscall now supports $$vars. - The SNMP tapset provides probes and functions for many network statistics. See stapprobes.snmp(3stap) for more details. - The dentry tapset provides functions to map kernel VFS directory entries to file or full path names: d_path(), d_name() and reverse_path_walk(). - SystemTap now has userspace markers in its own binaries, and the stap tapset provides the available probepoints and local variables. - Miscellaneous new tapset functions: - pgrp() returns the process group ID of the current process - str_replace() performs string replacement * What's new in version 0.9.8, 2009-06-11 - Miscellaneous new tapset functions: - sid() returns the session ID of the current process - stringat() indexes a single character from a string. - Using %M in print formats for hex dumps can now print entire buffers, instead of just small numbers. - Dwarfless syscalls: The nd_syscalls tapset is now available to probe system calls without requiring kernel debugging information. All of the same probepoints in the normal syscalls tapset are available with an "nd_" prefix, e.g. syscall.open becomes nd_syscall.open. Most syscall arguments are also available by name in nd_syscalls. - Module signing: If the appropriate nss libraries are available on your system, stap-server will sign each compiled module using a self-generated certificate. This is the first step toward extending authority to load certain modules to unprivileged users. For now, if the system administrator adds a certificate to a database of trusted signers (stap-authorize-signing-cert), modules signed using that certificate will be verified by staprun against tampering. Otherwise, you should notice no difference in the operation of stap or staprun. * What's new in version 0.9.7, 2009-04-23 - @cast can now determine its type information using an explicit header specification. For example: @cast(tv, "timeval", "<sys/time.h>")->tv_sec @cast(task, "task_struct", "kernel<linux/sched.h>")->tgid - The overlapping process.* tapsets are now separated. Those probe points documented in stapprobes(3stap) remain the same. Those that were formerly in stapprobes.process(3stap) have been renamed to kprocess, to reflect their kernel perspective on processes. - The --skip-badvars option now also suppresses run-time error messages that would otherwise result from erroneous memory accesses. Such accesses can originate from $context expressions fueled by erroneous debug data, or by kernel_{long,string,...}() tapset calls. - New probes kprobe.function(FUNCTION) and kprobe.function(FUNCTION).return for dwarfless probing. These postpone function address resolution to run-time and use the kprobe symbol-resolution mechanism. Probing of absolute statements can be done using the kprobe.statement(ADDRESS).absolute construct. - EXPERIMENTAL support for user process unwinding. A new collection of tapset functions have been added to handle user space backtraces from probe points that support them (currently process and timer probes - for timer probes test whether or not in user space first with the already existing user_mode() function). The new tapset functions are: uaddr - User space address of current running task. usymname - Return the symbol of an address in the current task. usymdata - Return the symbol and module offset of an address. print_ustack - Print out stack for the current task from string. print_ubacktrace - Print stack back trace for current task. ubacktrace - Hex backtrace of current task stack. Please read http://sourceware.org/ml/systemtap/2009-q2/msg00364.html on the current restrictions and possible changes in the future and give feedback if you want to influence future developments. * What's new in version 0.9.5, 2009-03-27 - New probes process().insn and process().insn.block that allows inspection of the process after each instruction or block of instructions executed. So to count the total number of instructions a process executes during a run do something like: $ stap -e 'global steps; probe process("/bin/ls").insn {steps++} probe end {printf("Total instructions: %d\n", steps);}' \ -c /bin/ls This feature can slow down execution of a process somewhat. - Systemtap probes and function man pages extracted from the tapsets are now available under 3stap. To show the page for probe vm.pagefault or the stap function pexecname do: $ man 3stap vm.pagefault $ man 3stap pexecname - Kernel tracepoints are now supported for probing predefined kernel events without any debuginfo. Tracepoints incur less overhead than kprobes, and context parameters are available with full type information. Any kernel 2.6.28 and later should have defined tracepoints. Try the following to see what's available: $ stap -L 'kernel.trace("*")' - Typecasting with @cast now supports modules search paths, which is useful in case there are multiple places where the type definition may be found. For example: @cast(sdev, "scsi_device", "kernel:scsi_mod")->sdev_state - On-file flight recorder is supported. It allows stap to record huge trace log on the disk and to run in background. Passing -F option with -o option runs stap in background mode. In this mode, staprun is detached from console, and stap itself shows staprun's pid and exits. Specifying the max size and the max number of log files are also available by passing -S option. This option has one or two arguments seperated by a comma. The first argument is the max size of a log file in MB. If the size of a log file exceeds it, stap switches to the next log file automatically. The second is how many files are kept on the disk. If the number of log files exceeds it, the oldest log file is removed automatically. The second argument can be omitted. For example, this will record output on log files each of them is smaller than 1024MB and keep last 3 logs, in background. % stap -F -o /tmp/staplog -S 1024,3 script.stp - In guru mode (-g), the kernel probing blocklist is disabled, leaving only a subset - the kernel's own internal kprobe blocklist - to attempt to filter out areas unsafe to probe. The differences may be enough to probe more interrupt handlers. - Variables unavailable in current context may be skipped by setting a session level flag with command line option --skip-badvars now available. This replaces any dwarf $variable expressions that could not be resolved with literal numeric zeros, along with a warning message. - Both kernel markers and kernel tracepoint support argument listing through stap -L 'kernel.mark("*")' or stap -L 'kernel.trace("*")' - Users can use -DINTERRUPTIBLE=0 to prevent interrupt reentrancy in their script, at the cost of a bit more overhead to toggle the interrupt mask. - Added reentrancy debugging. If stap is run with the arguments "-t -DDEBUG_REENTRANCY", additional warnings will be printed for every reentrancy event, including the probe points of the resident and interloper probes. - Default to --disable-pie for configure. Use --enable-pie to turn it back on. - Improved sdt.h compatibility and test suite for static dtrace compatible user space markers. - Some architectures now use syscall wrappers (HAVE_SYSCALL_WRAPPERS). The syscall tapset has been enhanced to take care of the syscall wrappers in this release. - Security fix for CVE-2009-0784: stapusr module-path checking race. * What's new in version 0.9, 2009-02-19 - Typecasting is now supported using the @cast operator. A script can define a pointer type for a "long" value, and then access type members using the same syntax as with $target variables. For example, this will retrieve the parent pid from a kernel task_struct: @cast(pointer, "task_struct", "kernel")->parent->pid - process().mark() probes are now possible to trace static user space markers put in programs with the STAP_PROBE macro using the new sys/sdt.h include file. This also provides dtrace compatible markers through DTRACE_PROBE and an associated python 'dtrace' script that can be used in builds based on dtrace that need dtrace -h or -G functionality. - For those that really want to run stap from the build tree there is now the 'run-stap' script in the top-level build directory that sets up the SYSTEMTAP_TAPSET, SYSTEMTAP_RUNTIME, SYSTEMTAP_STAPRUN, and SYSTEMTAP_STAPIO environment variables (installing systemtap, in a local prefix, is still recommended for common use). - Systemtap now comes with a new Beginners Guide that walks the user through their first steps setting up stap, understanding how it all works, introduces some useful scripts and describes some common pitfalls. It isn't created by default since it needs a Publican setup, but full build instructions can be found in the wiki: http://sourceware.org/systemtap/wiki/PublicanQuikHowto An online version can be found at: http://sourceware.org/systemtap/SystemTap_Beginners_Guide.pdf - Standard tapsets included with Systemtap were modified to include extractable documentation information based on the kernel-doc infrastructure. When configured --enabled-docs a HTML and PDF version of the Tapset Reference Manual is produced explaining probes defined in each tapset. - The systemtap client and compile server are now available. These allow you to compile a systemtap module on a host other than the one which it will be run, providing the client and server are compatible. Other than using a server for passes 1 through 4, the client behaves like the 'stap' front end itself. This means, among other things, that the client will automatically load the resulting module on the local host unless -p[1234] was specified. See stap-server(8) for more details. The client/server now use SSL for network connection security and for signing. The systemtap client and server are prototypes only. Interfaces, options and usage may change at any time. - function("func").label("label") probes are now supported to allow matching the label of a function. - Systemtap initscript is available. This initscript allows you to run systemtap scripts as system services (in flight recorder mode) and control those scripts individually. See README.systemtap for details. - The stap "-r DIR" option may be used to identify a hand-made kernel build directory. The tool determines the appropriate release string automatically from the directory. - Serious problems associated with user-space probing in shared libraries were corrected, making it now possible to experiment with probe shared libraries. Assuming dwarf debugging information is installed, use this twist on the normal syntax: probe process("/lib64/libc-2.8.so").function("....") { ... } This would probe all threads that call into that library. Running "stap -c CMD" or "stap -x PID" naturally restricts this to the target command+descendants only. $$vars etc. may be used. - For scripts that sometimes terminate with excessive "skipped" probes, rerunning the script with "-t" (timing) will print more details about the skippage reasons. - Symbol tables and unwind (backtracing) data support were formerly compiled in for all probed modules as identified by the script (kernel; module("name"); process("file")) plus those listed by the stap "-d BINARY" option. Now, this data is included only if the systemtap script uses tapset functions like probefunc() or backtrace() that require such information. This shrinks the probe modules considerably for the rest. - Per-pass verbosity control is available with the new "--vp {N}+" option. "stap --vp 040" adds 4 units of -v verbosity only to pass 2. This is useful for diagnosing errors from one pass without excessive verbosity from others. - Most probe handlers now run with interrupts enabled, for improved system responsiveness and less probing overhead. This may result in more skipped probes, for example if a reentrant probe handler is attempted from within an interrupt handler. It may also make the systemtap overload detection facility more likely to be triggered, as interrupt handlers' run time would be included in the self-assessed overhead of running probe handlers. * What's new in version 0.8, 2008-11-13 - Cache limiting is now available. If the compiled module cache size is over a limit specified in the $SYSTEMTAP_DIR/cache/cache_mb_limit file, some old cache entries will be unlinked. See man stap(1) for more. - Error and warning messages are now followed by source context displaying the erroneous line/s and a handy '^' in the following line pointing to the appropriate column. - A bug reporting tool "stap-report" is now available which will quickly retrieve much of the information requested here: http://sourceware.org/systemtap/wiki/HowToReportBugs - The translator can resolve members of anonymous structs / unions: given struct { int foo; struct { int bar; }; } *p; this now works: $p->bar - The stap "-F" flag activates "flight recorder" mode, which consists of translating the given script as usual, but implicitly launching it into the background with staprun's existing "-L" (launch) option. A user can later reattach to the module with "staprun -A MODULENAME". - Additional context variables are available on user-space syscall probes. - $argN ($arg1, $arg2, ... $arg6) in process(PATH_OR_PID).syscall gives you the argument of the system call. - $return in process(PATH_OR_PID).syscall.return gives you the return value of the system call. - Target process mode (stap -c CMD or -x PID) now implicitly restricts all "process.*" probes to the given child process. (It does not affect kernel.* or other probe types.) The CMD string is normally run directly, rather than via a /bin/sh -c subshell, since then utrace/uprobe probes receive a fairly "clean" event stream. If metacharacters like redirection operators were present in CMD, then "sh -c CMD" is still used, and utrace/uprobe probes will receive events from the shell. % stap -e 'probe process.syscall, process.end { printf("%s %d %s\n", execname(), pid(), pp())}'\ -c ls ls 2323 process.syscall ls 2323 process.syscall ls 2323 process.end - Probe listing mode is improved: "-L" lists available script-level variables % stap -L 'syscall.*open*' syscall.mq_open name:string name_uaddr:long filename:string mode:long u_attr_uaddr:long oflag:long argstr:string syscall.open name:string filename:string flags:long mode:long argstr:string syscall.openat name:string filename:string flags:long mode:long argstr:string - All user-space-related probes support $PATH-resolved executable names, so probe process("ls").syscall {} probe process("./a.out").syscall {} work now, instead of just probe process("/bin/ls").syscall {} probe process("/my/directory/a.out").syscall {} - Prototype symbolic user-space probing support: # stap -e 'probe process("ls").function("*").call { log (probefunc()." ".$$parms) }' \ -c 'ls -l' This requires: - debugging information for the named program - a version of utrace in the kernel that is compatible with the "uprobes" kernel module prototype. This includes RHEL5 and older Fedora, but not yet current lkml-track utrace; a "pass 4a"-time build failure means your system cannot use this yet. - Global variables which are written to but never read are now automatically displayed when the session does a shutdown. For example: global running_tasks probe timer.profile {running_tasks[pid(),tid()] = execname()} probe timer.ms(8000) {exit()} - A formatted string representation of the variables, parameters, or local variables at a probe point is now supported via the special $$vars, $$parms, and $$locals context variables, which expand to a string containing a list "var1=0xdead var2=0xbeef var3=?". (Here, var3 exists but is for some reason unavailable.) In return probes only, $$return expands to an empty string for a void function, or "return=0xf00". * What's new in version 0.7, 2008-07-15 - .statement("func@file:*") and .statement("func@file:M-N") probes are now supported to allow matching a range of lines in a function. This allows tracing the execution of a function. - Scripts relying on probe point wildcards like "syscall.*" that expand to distinct kprobes are processed significantly faster than before. - The vector of script command line arguments is available in a tapset-provided global array argv[]. It is indexed 1 ... argc, another global. This can substitute for of preprocessor directives @NNN that fail at parse time if there are not enough arguments. printf("argv: %s %s %s", argv[1], argv[2], argv[3]) - .statement("func@file+line") probes are now supported to allow a match relative to the entry of the function incremented by line number. This allows using the same systemtap script if the rest of the file.c source only changes slightly. - A probe listing mode is available. % stap -l vm.* vm.brk vm.mmap vm.munmap vm.oom_kill vm.pagefault vm.write_shared - More user-space probe types are added: probe process(PID).begin { } probe process("PATH").begin { } probe process(PID).thread.begin { } probe process("PATH").thread.begin { } probe process(PID).end { } probe process("PATH").end { } probe process(PID).thread.end { } probe process("PATH").thread.end { } probe process(PID).syscall { } probe process("PATH").syscall { } probe process(PID).syscall.return { } probe process("PATH").syscall.return { } - Globals now accept ; terminators global odds, evens; global little[10], big[5]; * What's new in version 0.6, 2007-12-15 - A copy of the systemtap tutorial and language reference guide are now included. - There is a new format specifier, %m, for the printf family of functions. It functions like %s, except that it does not stop when a nul ('\0') byte is encountered. The number of bytes output is determined by the precision specifier. The default precision is 1. For example: printf ("%m", "My String") // prints one character: M printf ("%.5", myString) // prints 5 bytes beginning at the start // of myString - The %b format specifier for the printf family of functions has been enhanced as follows: 1) When the width and precision are both unspecified, the default is %8.8b. 2) When only one of the width or precision is specified, the other defaults to the same value. For example, %4b == %.4b == %4.4b 3) Nul ('\0') bytes are used for field width padding. For example, printf ("%b", 0x1111deadbeef2222) // prints all eight bytes printf ("%4.2b", 0xdeadbeef) // prints \0\0\xbe\xef - Dynamic width and precision are now supported for all printf family format specifiers. For example: four = 4 two = 2 printf ("%*.*b", four, two, 0xdeadbbeef) // prints \0\0\xbe\xef printf ("%*d", four, two) // prints <space><space><space>2 - Preprocessor conditional expressions can now include wildcard style matches on kernel versions. %( kernel_vr != "*xen" %? foo %: bar %) - Prototype support for user-space probing is showing some progress. No symbolic notations are supported yet (so no probing by function names, file names, process names, and no access to $context variables), but at least it's something: probe process(PID).statement(ADDRESS).absolute { } This will set a uprobe on the given process-id and given virtual address. The proble handler runs in kernel-space as usual, and can generally use existing tapset functions. - Crash utility can retrieve systemtap's relay buffer from a kernel dump image by using staplog which is a crash extension module. To use this feature, type commands as below from crash(8)'s command line: crash> extend staplog.so crash> help systemtaplog Then, you can see more precise help message. - You can share a relay buffer amoung several scripts and merge outputs from several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options. For example: # run a host script % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST & # wait until starting the host. % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST Then, you'll see "hello world" in merged.out. - You can add a conditional statement for each probe point or aliase, which is evaluated when the probe point is hit. If the condition is false, the whole probe body(including aliases) is skipped. For example: global switch = 0; probe syscall.* if (switch) { ... } probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */ - Systemtap will warn you if your script contains unused variables or functions. This is helpful in case of misspelled variables. If it doth protest too much, turn it off with "stap -w ...". - You can add error-handling probes to a script, which are run if a script was stopped due to errors. In such a case, "end" probes are not run, but "error" ones are. probe error { println ("oops, errors encountered; here's a report anyway") foreach (coin in mint) { println (coin) } } - In a related twist, one may list probe points in order of preference, and mark any of them as "sufficient" beyond just "optional". Probe point sequence expansion stops if a sufficient-marked probe point has a hit. This is useful for probes on functions that may be in a module (CONFIG_FOO=m) or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know which. Instead of probe module("sd").function("sd_init_command") ? , kernel.function("sd_init_command") ? { ... } which might match neither, now one can write this: probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */ kernel.function("sd_init_command") { ... } - New security model. To install a systemtap kernel module, a user must be one of the following: the root user; a member of the 'stapdev' group; or a member of the 'stapusr' group. Members of the stapusr group can only use modules located in the /lib/modules/VERSION/systemtap directory (where VERSION is the output of "uname -r"). - .statement("...@file:line") probes now apply heuristics to allow an approximate match for the line number. This works similarly to gdb, where a breakpoint placed on an empty source line is automatically moved to the next statement. A silly bug that made many $target variables inaccessible to .statement() probes was also fixed. - LKET has been retired. Please let us know on <systemtap@sourceware.org> if you have been a user of the tapset/tools, so we can help you find another way. - New families of printing functions println() and printd() have been added. println() is like print() but adds a newline at the end; printd() is like a sequence of print()s, with a specified field delimiter. * What's new since version 0.5.14?, 2007-07-03 - The way in which command line arguments for scripts are substituted has changed. Previously, $1 etc. would interpret the corresponding command line argument as an numeric literal, and @1 as a string literal. Now, the command line arguments are pasted uninterpreted wherever $1 etc. appears at the beginning of a token. @1 is similar, but is quoted as a string. This change does not modify old scripts, but has the effect of permitting substitution of arbitrary token sequences. # This worked before, and still does: % stap -e 'probe timer.s($1) {}' 5 # Now this also works: % stap -e 'probe syscall.$1 {log(@1)}' open # This won't crash, just signal a recursion error: % stap -e '$1' '$1' # As before, $1... is recognized only at the beginning of a token % stap -e 'probe begin {foo$1=5}' * What's new since version 0.5.13?, 2007-03-26 - The way in which systemtap resolves function/inline probes has changed: .function(...) - now refers to all functions, inlined or not .inline(...) - is deprecated, use instead: .function(...).inline - filters function() to only inlined instances .function(...).call - filters function() to only non-inlined instances .function(...).return - as before, but now pairs best with .function().call .statement() is unchanged. * What's new since version 0.5.12?, 2007-01-01 - When running in -p4 (compile-only) mode, the compiled .ko file name is printed on standard output. - An array element with a null value such as zero or an empty string is now preserved, and will show up in a "foreach" loop or "in" test. To delete such an element, the scripts needs to use an explicit "delete array[idx]" statement rather than something like "array[idx]=0". - The new "-P" option controls whether prologue searching heuristics will be activated for function probes. This was needed to get correct debugging information (dwarf location list) data for $target variables. Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is no longer default. A new configure flag (--enable-prologues) restores it as a default setting, and is appropriate for older compilers (gcc 3.*). - Each systemtap module prints a one-line message to the kernel informational log when it starts. This line identifies the translator version, base address of the probe module, a broken-down memory consumption estimate, and the total number of probes. This is meant as a debugging / auditing aid. - Begin/end probes are run with interrupts enabled (but with preemption disabled). This will allow begin/end probes to be longer, to support generating longer reports. - The numeric forms of kernel.statement() and kernel.function() probe points are now interpreted as relocatable values - treated as relative to the _stext symbol in that kernel binary. Since some modern kernel images are relocated to a different virtual address at startup, such addresses may shift up or down when actually inserted into a running kernel. kernel.statement(0xdeadbeef): validated, interpreted relative to _stext, may map to 0xceadbeef at run time. In order to specify unrelocated addresses, use the new ".absolute" probe point suffix for such numeric addresses. These are only allowed in guru mode, and provide access to no $target variables. They don't use debugging information at all, actually. kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only * What's new since version 0.5.10?, 2006-10-19 - Offline processing of debugging information, enabling general cross-compilation of probe scripts to remote hosts, without requiring identical module/memory layout. This slows down compilation/translation somewhat. - Kernel symbol table data is loaded by staprun at startup time rather than compiled into the module. - Support the "limit" keyword for foreach iterations: foreach ([x,y] in ary limit 5) { ... } This implicitly exits after the fifth iteration. It also enables more efficient key/value sorting. - Support the "maxactive" keyword for return probes: probe kernel.function("sdfsdf").maxactive(848) { ... } This allows up to 848 concurrently outstanding entries to the sdfsdf function before one returns. The default maxactive number is smaller, and can result in missed return probes. - Support accessing of saved function arguments from within return probes. These values are saved by a synthesized function-entry probe. - Add substantial version/architecture checking in compiled probes to assert correct installation of debugging information and correct execution on a compatible kernel. - Add probe-time checking for sufficient free stack space when probe handlers are invoked, as a safety improvement. - Add an optional numeric parameter for begin/end probe specifications, to order their execution. probe begin(10) { } /* comes after */ probe begin(-10) {} - Add an optional array size declaration, which is handy for very small or very large ones. global little[5], big[20000] - Include some example scripts along with the documentation. - Change the start-time allocation of probe memory to avoid causing OOM situations, and to abort cleanly if free kernel memory is short. - Automatically use the kernel DWARF unwinder, if present, for stack tracebacks. - Many minor bug fixes, performance, tapset, and error message improvements.