On 2016/10/25 at 11:27, Xunlei Pang wrote:
On 2016/10/10 at 15:43, Xunlei Pang wrote:
> The current method for kdump memory debug is to use dracut
"rd.memdebug=[0-3]",
> it is not enough for debugging kernel modules. For example, when we want to find
> out which kernel module consumes a large amount of memory, "rd.memdebug"
won't
> help too much.
>
> A better way is needed to achieve this requirement, this is very useful for kdump
> OOM debugging.
>
> The principle of this patch series is to use kernel trace to track slab and buddy
> allocation calls during kernel module loading(module_init), thus we can analyze
> all the trace data and get the total memory consumption.
>
> The trace events include memory calls under /sys/kernel/debug/tracing/events:
> kmem/mm_page_alloc
> kmem/mm_page_free
> kmem/kmalloc
> kmem/kmalloc_node
> kmem/kmem_cache_alloc
> kmem/kmem_cache_alloc_node
>
> We also inpect the following events to detect the module loading
> module/module_load
> module/module_put
>
> We can get the module name and task pid from "module_load" event which
> also mark the beginning of the loading, and module_put called by the
> same task pid implies the end of the loading. So the memory events
> recorded in between by the same task pid are consumed by this module
> during loading(i.e. modprobe or module_init()).
>
> With these information, we can record approximately the total memory
> consumption involved by each kernel module loading.
>
> One major flaw of this method is that the trace ring buffer consumes a lot
> of memory. If it is too small, old records maybe be overwritten by subsequent
For some large systems, the amount of memory consumption is appreciable.
Does anyone have a good idea to reduce the memory size consumed by tracing?
I previously tried to use trace filter to track specific process doing modprobe, but
according to the test results, normally the process name with "systemd-udevd"
and "modprobe" will insert the ko, but sometimes I found some unknown process
will also insert the ko. So it's unreliable to do so.
The "module_load" event trace data output will be like:
modprobe-1887 [001] .... 13412.516997: module_load: nfs
But sometimes, "module_load" event trace data output will be like:
<...>-1558 [000] .... 12961.529111: module_load: fscache
So the task name(the first column) lost.
After some digging I found it's due to the insufficient trace cmdline buffer,
there are two files related:
tracing/saved_cmdlines
tracing/saved_cmdlines_size
I finally found the fact that if the cmdlines buffer runs out, trace output will miss the
task comm-pid information.
So actually we are able to monitor the module loading process using the filter.
There are three kinds of known applications for loading kernel modules:
"systemd-udevd", "modprobe" and "insmod".
Using them as the filter considerably reduces trace memory consumption, it only
needs tens of lines in trace buffer normally, thus the default size suffices.
Regards,
Xunlei
I am currently thinking to monitor buddy event only, as all the large slab allocation
will actually fall back to buddy to request pages. This will considerably reduce the
requirement of trace buffer size.
Regards,
Xunlei
> records. The trace ring buffer is set to be 10MB by default, but it can be
> overridden by users via the standard kernel boot parameter
"trace_buf_size".
>
> Users should increase the crash kernel memory reservation as needed after
> setting large trace ring buffer size, in case oom happens during debugging.
>
> Usage:
> 1)Pass "rd.memdebug" to kdump kernel cmdline using
"KDUMP_COMMANDLINE_APPEND" in /etc/sysconfig/kdump.
> 2)Pass the extra "trace_buf_size=nn[KMG]" to specify trace ring buffer
size(per cpu) as needed.
>
> Xunlei Pang (2):
> memdebug-ko: add dracut-memdebug-ko.sh to debug kernel module memory
> consumption
> module-setup: apply kernel module memory debug support
>
> dracut-kdump.sh | 11 ++++
> dracut-memdebug-ko.sh | 144 +++++++++++++++++++++++++++++++++++++++++++++++++
> dracut-module-setup.sh | 12 +++++
> kdumpctl | 14 +++++
> kexec-tools.spec | 2 +
> 5 files changed, 183 insertions(+)
> create mode 100755 dracut-memdebug-ko.sh
>