Demonstrations of compactstall, the Linux eBPF/bcc version.
compactsnoop traces the compact zone system-wide, and print various details.
Example output (manual trigger by echo 1 > /proc/sys/vm/compact_memory):
# ./compactsnoop
COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS
zsh 23685 0 ZONE_DMA -1 SYNC 0.025 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 3.925 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 113.975 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 81.57 complete
zsh 23685 0 ZONE_DMA -1 SYNC 0.02 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 4.631 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 113.975 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 80.647 complete
zsh 23685 0 ZONE_DMA -1 SYNC 0.020 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 3.367 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 115.18 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 81.766 complete
zsh 23685 0 ZONE_DMA -1 SYNC 0.025 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 4.346 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 114.570 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 80.820 complete
zsh 23685 0 ZONE_DMA -1 SYNC 0.026 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 4.611 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 113.993 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 80.928 complete
zsh 23685 0 ZONE_DMA -1 SYNC 0.02 complete
zsh 23685 0 ZONE_DMA32 -1 SYNC 3.889 complete
zsh 23685 0 ZONE_NORMAL -1 SYNC 113.776 complete
zsh 23685 1 ZONE_NORMAL -1 SYNC 80.727 complete
^C
While tracing, the processes alloc pages due to memory fragmentation is too
serious to meet contiguous memory requirements in the system, compact zone
events happened, which will increase the waiting delay of the processes.
compactsnoop can be useful for discovering when compact_stall(/proc/vmstat)
continues to increase, whether it is caused by some critical processes or not.
The STATUS include (CentOS 7.6's kernel)
compact_status = {
# COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable
0: "skipped",
# COMPACT_CONTINUE: compaction should continue to another pageblock
1: "continue",
# COMPACT_PARTIAL: direct compaction partially compacted a zone and there are suitable pages
2: "partial",
# COMPACT_COMPLETE: The full zone was compacted
3: "complete",
}
or (kernel 4.7 and above)
compact_status = {
# COMPACT_NOT_SUITABLE_ZONE: For more detailed tracepoint output - internal to compaction
0: "not_suitable_zone",
# COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable
1: "skipped",
# COMPACT_DEFERRED: compaction didn't start as it was deferred due to past failures
2: "deferred",
# COMPACT_NOT_SUITABLE_PAGE: For more detailed tracepoint output - internal to compaction
3: "no_suitable_page",
# COMPACT_CONTINUE: compaction should continue to another pageblock
4: "continue",
# COMPACT_COMPLETE: The full zone was compacted scanned but wasn't successful to compact suitable pages.
5: "complete",
# COMPACT_PARTIAL_SKIPPED: direct compaction has scanned part of the zone but wasn't successful to compact suitable pages.
6: "partial_skipped",
# COMPACT_CONTENDED: compaction terminated prematurely due to lock contentions
7: "contended",
# COMPACT_SUCCESS: direct compaction terminated after concluding that the allocation should now succeed
8: "success",
}
The -p option can be used to filter on a PID, which is filtered in-kernel. Here
I've used it with -T to print timestamps:
# ./compactsnoop -Tp 24376
TIME(s) COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS
101.364115000 zsh 24376 0 ZONE_DMA -1 SYNC 0.025 complete
101.364555000 zsh 24376 0 ZONE_DMA32 -1 SYNC 3.925 complete
^C
This shows the zsh process allocs pages, and compact zone events happening,
and the delays are not affected much.
A maximum tracing duration can be set with the -d option. For example, to trace
for 2 seconds:
# ./compactsnoop -d 2
COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS
zsh 26385 0 ZONE_DMA -1 SYNC 0.025444 complete
^C
The -e option prints out extra columns
# ./compactsnoop -e
COMM PID NODE ZONE ORDER MODE FRAGIDX MIN LOW HIGH FREE LAT(ms) STATUS
summ 28276 1 ZONE_NORMAL 3 ASYNC 0.728 11284 14105 16926 14193 3.58 partial
summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 14479 0.0 complete
summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 14785 0.019 complete
summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 15199 0.006 partial
summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 17360 0.030 complete
summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 15443 0.024 complete
summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 15634 0.018 complete
summ 28276 1 ZONE_NORMAL 3 ASYNC 0.832 11284 14105 16926 15301 0.006 partial
summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 14774 0.005 partial
summ 28276 1 ZONE_NORMAL 3 ASYNC 0.733 11284 14105 16926 19888 0.012 partial
^C
The FRAGIDX is short for fragmentation index, which only makes sense if an
allocation of a requested size would fail. If that is true, the fragmentation
index indicates whether external fragmentation or a lack of memory was the
problem. The value can be used to determine if page reclaim or compaction
should be used.
Index is between 0 and 1 so return within 3 decimal places
0 => allocation would fail due to lack of memory
1 => allocation would fail due to fragmentation
We can see the whole buddy's fragmentation index from /sys/kernel/debug/extfrag/extfrag_index
The MIN/LOW/HIGH shows the watermarks of the zone, which can also get from
/proc/zoneinfo, and FREE means nr_free_pages (can be found in /proc/zoneinfo too).
The -K option prints out kernel stack
# ./compactsnoop -K -e
summ 28276 0 ZONE_NORMAL 3 ASYNC 0.528 11043 13803 16564 22654 13.258 partial
kretprobe_trampoline+0x0
try_to_compact_pages+0x121
__alloc_pages_direct_compact+0xac
__alloc_pages_slowpath+0x3e9
__alloc_pages_nodemask+0x404
alloc_pages_current+0x98
new_slab+0x2c5
___slab_alloc+0x3ac
__slab_alloc+0x40
kmem_cache_alloc_node+0x8b
copy_process+0x18e
do_fork+0x91
sys_clone+0x16
stub_clone+0x44
summ 28276 1 ZONE_NORMAL 3 ASYNC -1.000 11284 14105 16926 22074 0.008 partial
kretprobe_trampoline+0x0
try_to_compact_pages+0x121
__alloc_pages_direct_compact+0xac
__alloc_pages_slowpath+0x3e9
__alloc_pages_nodemask+0x404
alloc_pages_current+0x98
new_slab+0x2c5
___slab_alloc+0x3ac
__slab_alloc+0x40
kmem_cache_alloc_node+0x8b
copy_process+0x18e
do_fork+0x91
sys_clone+0x16
stub_clone+0x44
summ 28276 0 ZONE_NORMAL 3 ASYNC 0.527 11043 13803 16564 25653 9.812 partial
kretprobe_trampoline+0x0
try_to_compact_pages+0x121
__alloc_pages_direct_compact+0xac
__alloc_pages_slowpath+0x3e9
__alloc_pages_nodemask+0x404
alloc_pages_current+0x98
new_slab+0x2c5
___slab_alloc+0x3ac
__slab_alloc+0x40
kmem_cache_alloc_node+0x8b
copy_process+0x18e
do_fork+0x91
sys_clone+0x16
stub_clone+0x44
# ./compactsnoop -h
usage: compactsnoop.py [-h] [-T] [-p PID] [-d DURATION] [-K] [-e]
Trace compact zone
optional arguments:
-h, --help show this help message and exit
-T, --timestamp include timestamp on output
-p PID, --pid PID trace this PID only
-d DURATION, --duration DURATION
total duration of trace in seconds
-K, --kernel-stack output kernel stack trace
-e, --extended_fields
show system memory state
examples:
./compactsnoop # trace all compact stall
./compactsnoop -T # include timestamps
./compactsnoop -d 10 # trace for 10 seconds only
./compactsnoop -K # output kernel stack trace
./compactsnoop -e # show extended fields
