Change in vdsm[ovirt-3.5]: profiling: add support for memory profiling

fromani at redhat.com fromani at redhat.com
Tue Feb 16 10:40:43 UTC 2016


Hello Dan Kenigsberg,

I'd like you to do a code review.  Please visit

    https://gerrit.ovirt.org/47594

to review the following change.

Change subject: profiling: add support for memory profiling
......................................................................

profiling: add support for memory profiling

There are many python memory profilers, but most of them are
tailored for interactive usage, so additional integration work
is needed to use them in a daemon, like VDSM.

This patch adds support for dowser:
<http://www.aminus.net/wiki/Dowser>
<https://pypi.python.org/pypi/dowser/0.2>

which has the following benefits which makes it in a better position
to be used with VDSM:
+ self contained, includes WEB UI
+ allows live monitoring
+ tailored for cherrypy usage, so friendly towards server applications.
+ easily portable (pure package)

It must be noted that dowser has some drawbacks as well
- significat dependencies, both in number and in size (with respect
  to VDSM standards): cherrypy, PIL
- based on gc module

As per the cpu profile already added, this code is meant to be
used only as debug aid or in development environments.
The feature is controlled by a config tunable and disabled by default;
the feature disable itself if any dependency is missed.

Change-Id: Ib56b65513e0118b68cd43791bf655c928d6a26e2
Signed-off-by: Francesco Romani <fromani at redhat.com>
Reviewed-on: https://gerrit.ovirt.org/32019
Reviewed-by: Dan Kenigsberg <danken at redhat.com>
---
M debian/vdsm-python.install
M lib/vdsm/config.py.in
M lib/vdsm/profiling/Makefile.am
A lib/vdsm/profiling/memory.py
M lib/vdsm/profiling/profile.py
M vdsm.spec.in
6 files changed, 124 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/94/47594/2

diff --git a/debian/vdsm-python.install b/debian/vdsm-python.install
index 030714d..e65281d 100644
--- a/debian/vdsm-python.install
+++ b/debian/vdsm-python.install
@@ -17,6 +17,7 @@
 ./usr/lib/python2.7/dist-packages/vdsm/profiling/__init__.py
 ./usr/lib/python2.7/dist-packages/vdsm/profiling/cpu.py
 ./usr/lib/python2.7/dist-packages/vdsm/profiling/errors.py
+./usr/lib/python2.7/dist-packages/vdsm/profiling/memory.py
 ./usr/lib/python2.7/dist-packages/vdsm/profiling/profile.py
 ./usr/lib/python2.7/dist-packages/vdsm/qemuimg.py
 ./usr/lib/python2.7/dist-packages/vdsm/sslutils.py
diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in
index f327231..92bf5a7 100644
--- a/lib/vdsm/config.py.in
+++ b/lib/vdsm/config.py.in
@@ -42,6 +42,12 @@
         ('cpu_profile_clock', 'cpu',
             'Sets the underlying clock type (cpu, wall)'),
 
+        ('memory_profile_enable', 'false',
+            'Enable whole process profiling (requires dowser profiler).'),
+
+        ('memory_profile_port', '9090',
+            'Port on which the dowser Web UI will be reachable.'),
+
         ('host_mem_reserve', '256',
             'Reserves memory for the host to prevent VMs from using all the '
             'physical pages. The values are in Mbytes.'),
diff --git a/lib/vdsm/profiling/Makefile.am b/lib/vdsm/profiling/Makefile.am
index 6bfd7d3..0d3cc01 100644
--- a/lib/vdsm/profiling/Makefile.am
+++ b/lib/vdsm/profiling/Makefile.am
@@ -24,5 +24,6 @@
 	__init__.py \
 	cpu.py \
 	errors.py \
+	memory.py \
 	profile.py \
 	$(NULL)
diff --git a/lib/vdsm/profiling/memory.py b/lib/vdsm/profiling/memory.py
new file mode 100755
index 0000000..c15d1aa
--- /dev/null
+++ b/lib/vdsm/profiling/memory.py
@@ -0,0 +1,102 @@
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+"""
+This module provides memory profiling.
+"""
+
+import logging
+import threading
+
+from vdsm.config import config
+from vdsm.utils import traceback
+
+from .errors import UsageError
+
+# Import modules lazily when profile is started
+dowser = None
+cherrypy = None
+
+_lock = threading.Lock()
+_thread = None
+
+
+def start():
+    """ Starts application memory profiling """
+    if is_enabled():
+        _start_profiling()
+
+
+def stop():
+    """ Stops application memory profiling """
+    if is_enabled():
+        _stop_profiling()
+
+
+def is_enabled():
+    return config.getboolean('vars', 'memory_profile_enable')
+
+
+def is_running():
+    return _thread is not None
+
+
+ at traceback()
+def _memory_viewer():
+    cherrypy.tree.mount(dowser.Root())
+
+    cherrypy.config.update({
+        'server.socket_host': '0.0.0.0',
+        'server.socket_port': config.getint('vars', 'memory_profile_port')})
+
+    cherrypy.engine.start()
+
+
+def _start_profiling():
+    global cherrypy
+    global dowser
+    global _thread
+
+    logging.debug("Starting memory profiling")
+
+    import cherrypy
+    import dowser
+    # this nonsense makes pyflakes happy
+    cherrypy
+    dowser
+
+    with _lock:
+        if is_running():
+            raise UsageError('Memory profiler is already running')
+        _thread = threading.Thread(name='memprofile',
+                                   target=_memory_viewer)
+        _thread.daemon = True
+        _thread.start()
+
+
+def _stop_profiling():
+    global _thread
+    logging.debug("Stopping memory profiling")
+    with _lock:
+        if is_running():
+            cherrypy.engine.exit()
+            cherrypy.engine.block()
+            _thread.join()
+            _thread = None
diff --git a/lib/vdsm/profiling/profile.py b/lib/vdsm/profiling/profile.py
index d14b431..1bfe326 100644
--- a/lib/vdsm/profiling/profile.py
+++ b/lib/vdsm/profiling/profile.py
@@ -23,11 +23,24 @@
 """
 
 from . import cpu
+from . import memory
 
 
 def start():
     cpu.start()
+    memory.start()
 
 
 def stop():
     cpu.stop()
+    memory.stop()
+
+
+def status():
+    res = {}
+    for profiler in (cpu, memory):
+        res[profiler.__name__] = {
+            "enabled": profiler.is_enabled(),
+            "running": profiler.is_running()
+        }
+    return res
diff --git a/vdsm.spec.in b/vdsm.spec.in
index bda574e..b1bf547 100644
--- a/vdsm.spec.in
+++ b/vdsm.spec.in
@@ -1269,6 +1269,7 @@
 %{python_sitelib}/%{vdsm_name}/profiling/__init__.py*
 %{python_sitelib}/%{vdsm_name}/profiling/cpu.py*
 %{python_sitelib}/%{vdsm_name}/profiling/errors.py*
+%{python_sitelib}/%{vdsm_name}/profiling/memory.py*
 %{python_sitelib}/%{vdsm_name}/profiling/profile.py*
 %{python_sitelib}/%{vdsm_name}/qemuimg.py*
 %{python_sitelib}/%{vdsm_name}/SecureXMLRPCServer.py*


-- 
To view, visit https://gerrit.ovirt.org/47594
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib56b65513e0118b68cd43791bf655c928d6a26e2
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.5
Gerrit-Owner: Francesco Romani <fromani at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation at ovirt.org>


More information about the vdsm-patches mailing list