[Bug 1064929] New: Kernel container scalability issue in docker devicemapper backend

List overview All Threads
Download

newer

older

[Bug 1069718] New: Kernel...

[Bug 1064012] New: improve docker...

Red Hat Bugzilla

13 Feb 2014 13 Feb '14

4:05 p.m.

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Bug ID: 1064929 Summary: Kernel container scalability issue in docker devicemapper backend Product: Red Hat Enterprise Linux 7 Version: 7.0 Component: kernel Severity: high Assignee: kernel-mgr@redhat.com Reporter: sct@redhat.com QA Contact: kernel-qe@redhat.com CC: agk@redhat.com, alexl@redhat.com, golang@lists.fedoraproject.org, jeder@redhat.com, lsm5@redhat.com, mattdm@redhat.com, mgoldman@redhat.com, skottler@redhat.com, vbatts@redhat.com Depends On: 1061359 Blocks: 1064012 Group: private

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1061359 [Bug 1061359] improve docker devicemapper backend scalability https://bugzilla.redhat.com/show_bug.cgi?id=1064012 [Bug 1064012] improve docker devicemapper backend scalability

-- You are receiving this mail because: You are on the CC list for the bug.

Show replies by date

Red Hat Bugzilla

13 Feb 13 Feb

4:56 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker devicemapper backend

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks|1064012 |

--- Comment #2 from Stephen Tweedie sct@redhat.com --- *** Bug 1064012 has been marked as a duplicate of this bug. ***

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1064012 [Bug 1064012] improve docker devicemapper backend scalability

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

14 Feb 14 Feb

2:56 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker devicemapper backend

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Jeremy Eder jeder@redhat.com changed:

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

3:33 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker devicemapper backend

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Jeremy Eder jeder@redhat.com changed:

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

17 Feb 17 Feb

6:56 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker devicemapper backend

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Linda Wang lwang@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-mgr@redhat.com |aviro@redhat.com

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

19 Feb 19 Feb

6:39 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Ric Wheeler rwheeler@redhat.com changed:

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

20 Feb 20 Feb

1:06 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Ian Kent ikent@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |ikent@redhat.com

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

1:37 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

--- Comment #12 from Ian Kent ikent@redhat.com --- Created attachment 865491 --> https://bugzilla.redhat.com/attachment.cgi?id=865491&action=edit kernel config diff

I was looking mostly at the "Lock Debugging (spinlocks, mutexes, etc...)" and the "RCU Debugging" sections but didn't see what I thought were similar differences between the rhel7 and ubuntu configs. on the test machine.

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

4:42 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

--- Comment #17 from Stephen Tweedie sct@redhat.com --- Created attachment 865585 --> https://bugzilla.redhat.com/attachment.cgi?id=865585&action=edit Disable CONFIG_SLUB_DEBUG

Patch against dist-git kernel rhel-7.0 branch to disable CONFIG_SLUB_DEBUG.

The -BuildRequires: rpm-build >= 4.9.0-1, elfutils >= elfutils-0.153-1 +BuildRequires: rpm-build >= 4.9.0-1, elfutils >= 0.153-1 is needed to satisfy mock (elfutils-0.153-1 has two "-" separators and is rejected as an invalid VR.)

Build with $ rhpkg srpm $ mock -r rhel-7-x86_64 --without=debug --without=debuginfo $PATH_TO_SRPM

NB -- the patch updates ONLY the kernel-x86_64 config, not kernel-debug nor other architectures.

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

25 Feb 25 Feb

3:50 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1069718

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1069718 [Bug 1069718] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

6:17 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Bhavna Sarathy bsarathy@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1069814

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

6:34 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

--- Comment #23 from Stephen Tweedie sct@redhat.com --- Created attachment 867566 --> https://bugzilla.redhat.com/attachment.cgi?id=867566&action=edit Debug info from docker containers

Tarball containing:

-rw-r--r-- sct/sct 8428 2014-02-24 19:34 mountinfo.1.txt

/proc/1/mountinfo root namespace mount info (after 3 docker containers created)

-rw-r--r-- sct/sct 1485 2014-02-24 19:34 mountinfo.d1.txt -rw-r--r-- sct/sct 1485 2014-02-24 19:34 mountinfo.d2.txt -rw-r--r-- sct/sct 1485 2014-02-24 19:34 mountinfo.d3.txt

/proc/$pid/mountinfo from those 3 containers; and

-rw-r--r-- sct/sct 332960 2014-02-24 19:41 docker.strace

strace -f of the docker daemon creating a single container, including all the various mount and unshare calls.

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

4 Mar 4 Mar

4:32 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1072446

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

4:38 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1072451

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1072451 [Bug 1072451] Kernel container scalability issue in docker - [2/4] Expand size of mount hash table

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

4:45 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1072457

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

4:50 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Stephen Tweedie sct@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1072461

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

20 Mar 20 Mar

8:05 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Paul W. Frields pfrields@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Tracking Status|NEW |MODIFIED Blocks|1072446, 1072451, 1072457, | |1072461 | Depends On| |1072446, 1072451, 1072457, | |1072461

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

21 Mar 21 Mar

9:35 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Josh Poimboeuf jpoimboe@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC|jpoimboe@redhat.com |

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

24 Mar 24 Mar

3:19 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Linqing Lu lilu@redhat.com changed:

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

3:36 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Eryu Guan eguan@redhat.com changed:

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

14 Apr 14 Apr

10:29 a.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Eryu Guan eguan@redhat.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Status|MODIFIED |VERIFIED

--- Comment #29 from Eryu Guan eguan@redhat.com --- Reproduced with kernel-3.10.0-106.el7, following steps in comment 5

Samples: 931K of event 'cycles', Event count (approx.): 39097077164 37.59% [kernel] [k] pcpu_alloc_area 17.67% [kernel] [k] pcpu_free_area 7.97% [kernel] [k] lg_global_lock 6.90% [kernel] [k] seq_list_start 3.57% [kernel] [k] memset 3.52% [kernel] [k] find_next_bit 3.02% [kernel] [k] lg_global_unlock 1.69% libc-2.17.so [.] _IO_vfscanf

Verified with kernel-3.10.0-121.el7

Samples: 253K of event 'cycles', Event count (approx.): 70681567890 13.92% libc-2.17.so [.] _IO_vfscanf 6.75% systemd [.] 0x0000000000080922 5.26% libc-2.17.so [.] malloc_consolidate 5.14% libc-2.17.so [.] _int_malloc 4.77% [kernel] [k] strchr 2.72% libc-2.17.so [.] _int_free 2.50% [kernel] [k] memcpy 2.25% [kernel] [k] format_decode 2.05% libmount.so.1.1.0 [.] 0x00000000000149b1 1.87% [kernel] [k] seq_escape 1.80% [kernel] [k] vsnprintf 1.74% libc-2.17.so [.] __strcmp_sse42

Also autofs regression/nfs regression tests passed without hitting new issues.

(Actually there's one regression in autofs tests, see bug 1083815, but that's probably caused by bug 1031475)

QE also have done three rounds testings since Snapshot 12 (on snapshot 13, RC-1 and RC-2), no other severe issues found.

So set to VERIFIED.

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

26 Jun 26 Jun

1:36 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929

Ludek Smid lsmid@redhat.com changed:

--- Comment #30 from Ludek Smid lsmid@redhat.com --- This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

-- You are receiving this mail because: You are on the CC list for the bug.

Red Hat Bugzilla

15 Sep 15 Sep

10:12 p.m.

New subject: [Bug 1064929] Kernel container scalability issue in docker - VFS and mount path consuming too much CPU

https://bugzilla.redhat.com/show_bug.cgi?id=1064929 Bug 1064929 depends on bug 1061359, which changed state.

Bug 1061359 Summary: improve docker devicemapper backend scalability https://bugzilla.redhat.com/show_bug.cgi?id=1061359

-- You are receiving this mail because: You are on the CC list for the bug.

3649

Age (days ago)

3863

Last active (days ago)

golang@lists.fedoraproject.org

22 comments

1 participants

tags (0)

participants (1)

Red Hat Bugzilla