Change in vdsm[master]: startup: Change system default encoding to utf8

danken at redhat.com danken at redhat.com
Mon Dec 21 14:24:43 UTC 2015


Dan Kenigsberg has submitted this change and it was merged.

Change subject: startup: Change system default encoding to utf8
......................................................................


startup: Change system default encoding to utf8

In Python 2, the system default encoding is 'ascii'. This causes mixing
of unicode and non-ascii strings (e.g. utf8 encoded) to fail with
UnicodeDecodeError or UnicodeEncodeError. The trigger for this failures
is starting using the built-in json library, that returns all values as
unicode strings, even if the value is ascii.

For example:

>>> u'ascii' + '\xd7\x90'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0:
ordinal not in range(128)

Python tries to decode the second value implicitly, and fails since this
is a utf8 encoded string.

To avoid such issues, the entire application must be changed to use only
strings or only unicode internally, and never mix these types.  In the
distant future, when we run on Python 3, this will be true. For now, the
only way to prevent these issues systematically is to change the system
default encoding to 'utf8'.

Changing the default encoding is done with sys.setdefaultencoding(), but
this function exists only during startup, and can be called only from
sitecustomize module.

We keep now sitecustomize.py module, and change PYTHONPATH so it will be
loaded on startup.

This change is not needed in Python 3 since the default encoding is
already utf8, and Python does not do any implicit decoding or encoding
when mixing 'str' and 'bytes' types.

Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Bug-Url: https://bugzilla.redhat.com/1281940
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
Reviewed-on: https://gerrit.ovirt.org/48661
Continuous-Integration: Jenkins CI
Reviewed-by: Milan Zamazal <mzamazal at redhat.com>
Reviewed-by: Francesco Romani <fromani at redhat.com>
Reviewed-by: Piotr Kliczewski <piotr.kliczewski at gmail.com>
---
M debian/vdsm.install
M init/daemonAdapter
M tests/Makefile.am
A tests/unicode_test.py
M vdsm.spec.in
M vdsm/Makefile.am
A vdsm/sitecustomize.py
7 files changed, 84 insertions(+), 0 deletions(-)

Approvals:
  Piotr Kliczewski: Looks good to me, approved
  Nir Soffer: Verified; Passed CI tests
  Jenkins CI: Passed CI tests
  Francesco Romani: Looks good to me, but someone else must approve
  Milan Zamazal: Looks good to me, but someone else must approve



-- 
To view, visit https://gerrit.ovirt.org/48661
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Adam Litke <alitke at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani at redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal at redhat.com>
Gerrit-Reviewer: Nir Soffer <nsoffer at redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski at gmail.com>
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei at redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation at ovirt.org>


More information about the vdsm-patches mailing list