Change in vdsm[master]: repoplot: Optimize log parsing

danken at redhat.com danken at redhat.com
Fri Feb 26 15:59:35 UTC 2016


Dan Kenigsberg has submitted this change and it was merged.

Change subject: repoplot: Optimize log parsing
......................................................................


repoplot: Optimize log parsing

Profiling show that most time is spent matching regular expressions.
Switch to string.find for matching the relevant lines, and extract the
timestamp using split from relevant lines.

Profiling
=========

Without this patch
------------------

Wed Feb 24 09:37:39 2016    repoplot-old.prof

         1607042 function calls (1587959 primitive calls) in 8.397 seconds

   Ordered by: internal time
   List reduced from 4142 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    48515    6.375    0.000    6.375    0.000 {method 'search' of '_sre.SRE_Pattern' objects}
      179    0.166    0.001    0.171    0.001 {eval}
    85278    0.158    0.000    0.270    0.000 backend_pdf.py:141(pdfRepr)
      424    0.122    0.000    0.286    0.001 backend_pdf.py:1372(pathOperations)
      146    0.120    0.001    0.124    0.001 {built-in method draw_path}
    26718    0.097    0.000    0.136    0.000 path.py:353(iter_segments)
    93406    0.039    0.000    0.040    0.000 {hasattr}
        1    0.039    0.039    6.640    6.640 repoplot:82(parse)
    29985    0.034    0.000    0.035    0.000 {method 'extend' of 'list' objects}
    56796    0.032    0.000    0.032    0.000 {method 'encode' of 'unicode' objects}

With this patch
---------------

Wed Feb 24 09:47:15 2016    repoplot-new.prof

         1606310 function calls (1587243 primitive calls) in 2.037 seconds

   Ordered by: internal time
   List reduced from 4142 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    85278    0.171    0.000    0.282    0.000 backend_pdf.py:141(pdfRepr)
      179    0.160    0.001    0.165    0.001 {eval}
      424    0.125    0.000    0.293    0.001 backend_pdf.py:1372(pathOperations)
      146    0.125    0.001    0.130    0.001 {built-in method draw_path}
    26718    0.099    0.000    0.139    0.000 path.py:353(iter_segments)
    93406    0.040    0.000    0.040    0.000 {hasattr}
    29985    0.033    0.000    0.034    0.000 {method 'extend' of 'list' objects}
        1    0.032    0.032    0.262    0.262 repoplot:80(parse)
    56796    0.031    0.000    0.031    0.000 {method 'encode' of 'unicode' objects}
     3064    0.026    0.000    0.026    0.000 {built-in method compress}

Before this patch, parsing 20 MiB log took 6.64 seconds, with this patch it
takes only 0.26 seconds (25X faster).

Timing
======

Processing 20 MiB log without this patch
----------------------------------------

$ time repoplot vdsm.log

real    0m8.029s
user    0m7.934s
sys     0m0.100s

Processing 20 MiB log with this patch
-------------------------------------

$ time repoplot vdsm.log

real    0m1.725s
user    0m1.650s
sys     0m0.077s

Processing multiple logs (289 MiB total)
----------------------------------------

$ time repoplot -n vdsm.log vdsm.log.9 vdsm.log.8 vdsm.log.7 \
    vdsm.log.6 vdsm.log.5 vdsm.log.4 vdsm.log.3 vdsm.log.2 \
    vdsm.log.1 vdsm.log

real    0m6.682s
user    0m6.542s
sys     0m0.145s

Change-Id: Iac2b2e675d1c4397a5659e89a34689ce7e4004a6
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
Reviewed-on: https://gerrit.ovirt.org/53949
Reviewed-by: Francesco Romani <fromani at redhat.com>
Continuous-Integration: Jenkins CI
Reviewed-by: Dan Kenigsberg <danken at redhat.com>
---
M contrib/repoplot
1 file changed, 5 insertions(+), 7 deletions(-)

Approvals:
  Nir Soffer: Verified
  Jenkins CI: Passed CI tests
  Dan Kenigsberg: Looks good to me, approved
  Francesco Romani: Looks good to me, but someone else must approve



-- 
To view, visit https://gerrit.ovirt.org/53949
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iac2b2e675d1c4397a5659e89a34689ce7e4004a6
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Adam Litke <alitke at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani at redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Yaniv Kaul <ykaul at redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation at ovirt.org>


More information about the vdsm-patches mailing list