Change in vdsm[master]: compat: Use simplejson instead of json

nsoffer at redhat.com nsoffer at redhat.com
Fri Nov 13 21:23:47 UTC 2015


Nir Soffer has uploaded a new change for review.

Change subject: compat: Use simplejson instead of json
......................................................................

compat: Use simplejson instead of json

We use to require simplejson when using Python 2.6 since the built-in
json library was missing the C extension, causing performance issue.
When we moved to Python 2.7, we drop this requirement, since Python 2.7
includes the C extension.

However json and simplejson are not equivalent. The built-in json
library likes to convert plain strings to unicode:

>>> import json
>>> json.loads(json.dumps({"key": unicode("\xd7\x90", 'utf8')}))
{u'key': u'\u05d0'}

While simplejson keeps plain strings as strings:

>>> json.loads(json.dumps({"key": unicode("\xd7\x90", 'utf8')}))
{'key': u'\u05d0'}

While "key" and u"key" are equal, they have different behavior when
combining them with non-ascii strings:

>>> "ascii" + "\xd7\x90"
'ascii\xd7\x90'

>>> u"ascii" + "\xd7\x90"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

Since u"ascii" is unicode string, Python try to convert "\xd7\x90" assuming
the default encoding (ascii), and fails.

Another example is logging:

>>> import logging
>> logging.error("ascii=%s utf8=%s", u"ascii", "\xd7\x90")
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 859, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 732, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 471, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

Again, works fine with using string object:

>>> logging.error("ascii=%s utf8=%s", "ascii", "\xd7\x90")
ERROR:root:ascii=ascii utf8=א

We require now the simplejson library to avoid this issue.

Change-Id: I2f0fb2ac45da86a344be556e6474667650bca966
Bug-Url: https://bugzilla.redhat.com/1281940
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
---
M lib/vdsm/compat.py
M vdsm.spec.in
2 files changed, 7 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/65/48565/1

diff --git a/lib/vdsm/compat.py b/lib/vdsm/compat.py
index 8ac5201..519943e 100644
--- a/lib/vdsm/compat.py
+++ b/lib/vdsm/compat.py
@@ -26,15 +26,8 @@
     import pickle
     pickle  # yep, this is needed twice.
 
-try:
-    # on RHEL/Centos 6.x, the JSON module in the python standard
-    # library does not include significant speedups:
-    # stdlib is based on simplejson 1.9, speedups were added on 2.0.9.
-    # In general, speedups are first found on the
-    # simplejson package.
-    import simplejson as json
-    json  # make pyflakes happy
-except ImportError:
-    # no big deal, fallback to standard libary
-    import json
-    json  # yep, this is needed twice.
+# We want to work only with simplejson, since the builtin json library
+# likes to convert plain ascii strings to unicode, causing failures when
+# mixing non-ascii string and "unicode" values.
+# See https://bugzilla.redhat.com/1281940
+import simplejson as json
diff --git a/vdsm.spec.in b/vdsm.spec.in
index 4a8860d..16dc1e9 100644
--- a/vdsm.spec.in
+++ b/vdsm.spec.in
@@ -75,6 +75,7 @@
 BuildRequires: python-netaddr
 BuildRequires: python-nose
 BuildRequires: python-six
+BuildRequires: python-simplejson
 BuildRequires: rpm-build
 
 # BuildRequires needed by the tests during the build
@@ -118,6 +119,7 @@
 Requires: xz
 Requires: ntp
 Requires: iproute >= 3.10.0
+Requires: python-simplejson
 Requires: python-netaddr
 Requires: python-inotify
 Requires: python-argparse


-- 
To view, visit https://gerrit.ovirt.org/48565
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2f0fb2ac45da86a344be556e6474667650bca966
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer at redhat.com>


More information about the vdsm-patches mailing list