Change in vdsm[master]: repoplot: Optimize log parsing
danken at redhat.com
danken at redhat.com
Fri Feb 26 15:59:35 UTC 2016
Dan Kenigsberg has submitted this change and it was merged.
Change subject: repoplot: Optimize log parsing
......................................................................
repoplot: Optimize log parsing
Profiling show that most time is spent matching regular expressions.
Switch to string.find for matching the relevant lines, and extract the
timestamp using split from relevant lines.
Profiling
=========
Without this patch
------------------
Wed Feb 24 09:37:39 2016 repoplot-old.prof
1607042 function calls (1587959 primitive calls) in 8.397 seconds
Ordered by: internal time
List reduced from 4142 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
48515 6.375 0.000 6.375 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
179 0.166 0.001 0.171 0.001 {eval}
85278 0.158 0.000 0.270 0.000 backend_pdf.py:141(pdfRepr)
424 0.122 0.000 0.286 0.001 backend_pdf.py:1372(pathOperations)
146 0.120 0.001 0.124 0.001 {built-in method draw_path}
26718 0.097 0.000 0.136 0.000 path.py:353(iter_segments)
93406 0.039 0.000 0.040 0.000 {hasattr}
1 0.039 0.039 6.640 6.640 repoplot:82(parse)
29985 0.034 0.000 0.035 0.000 {method 'extend' of 'list' objects}
56796 0.032 0.000 0.032 0.000 {method 'encode' of 'unicode' objects}
With this patch
---------------
Wed Feb 24 09:47:15 2016 repoplot-new.prof
1606310 function calls (1587243 primitive calls) in 2.037 seconds
Ordered by: internal time
List reduced from 4142 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
85278 0.171 0.000 0.282 0.000 backend_pdf.py:141(pdfRepr)
179 0.160 0.001 0.165 0.001 {eval}
424 0.125 0.000 0.293 0.001 backend_pdf.py:1372(pathOperations)
146 0.125 0.001 0.130 0.001 {built-in method draw_path}
26718 0.099 0.000 0.139 0.000 path.py:353(iter_segments)
93406 0.040 0.000 0.040 0.000 {hasattr}
29985 0.033 0.000 0.034 0.000 {method 'extend' of 'list' objects}
1 0.032 0.032 0.262 0.262 repoplot:80(parse)
56796 0.031 0.000 0.031 0.000 {method 'encode' of 'unicode' objects}
3064 0.026 0.000 0.026 0.000 {built-in method compress}
Before this patch, parsing 20 MiB log took 6.64 seconds, with this patch it
takes only 0.26 seconds (25X faster).
Timing
======
Processing 20 MiB log without this patch
----------------------------------------
$ time repoplot vdsm.log
real 0m8.029s
user 0m7.934s
sys 0m0.100s
Processing 20 MiB log with this patch
-------------------------------------
$ time repoplot vdsm.log
real 0m1.725s
user 0m1.650s
sys 0m0.077s
Processing multiple logs (289 MiB total)
----------------------------------------
$ time repoplot -n vdsm.log vdsm.log.9 vdsm.log.8 vdsm.log.7 \
vdsm.log.6 vdsm.log.5 vdsm.log.4 vdsm.log.3 vdsm.log.2 \
vdsm.log.1 vdsm.log
real 0m6.682s
user 0m6.542s
sys 0m0.145s
Change-Id: Iac2b2e675d1c4397a5659e89a34689ce7e4004a6
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
Reviewed-on: https://gerrit.ovirt.org/53949
Reviewed-by: Francesco Romani <fromani at redhat.com>
Continuous-Integration: Jenkins CI
Reviewed-by: Dan Kenigsberg <danken at redhat.com>
---
M contrib/repoplot
1 file changed, 5 insertions(+), 7 deletions(-)
Approvals:
Nir Soffer: Verified
Jenkins CI: Passed CI tests
Dan Kenigsberg: Looks good to me, approved
Francesco Romani: Looks good to me, but someone else must approve
--
To view, visit https://gerrit.ovirt.org/53949
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iac2b2e675d1c4397a5659e89a34689ce7e4004a6
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Adam Litke <alitke at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani at redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Yaniv Kaul <ykaul at redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation at ovirt.org>
More information about the vdsm-patches
mailing list