[blivet master] Handle unicode strings in Size spec parsing.

David Shea dshea at redhat.com
Wed Oct 22 14:03:51 UTC 2014


On 10/22/2014 08:48 AM, Anne Mulhern wrote:
>
>
>
> ----- Original Message -----
>> From: "David Shea" <dshea at redhat.com>
>> To: anaconda-patches at lists.fedorahosted.org
>> Sent: Tuesday, October 21, 2014 5:08:43 PM
>> Subject: [blivet master] Handle unicode strings in Size spec parsing.
>>
>> Since _() and friends now return unicode objects, most of size.py no
>> longer needs to decode strings into unicode objects. Change
>> humanReadable to return unicode objects that will be converted back to
>> str by __str__ if necessary. Handle unicode inputs to the Size
>> constructor.
>> ---
>>   blivet/size.py | 24 ++++++++++++------------
>>   1 file changed, 12 insertions(+), 12 deletions(-)
>>
>> diff --git a/blivet/size.py b/blivet/size.py
>> index ec09bf0..6dafdcc 100644
>> --- a/blivet/size.py
>> +++ b/blivet/size.py
>> @@ -75,12 +75,11 @@ _BYTES = [N_(b'B'), N_(b'b'), N_(b'byte'), N_(b'bytes')]
>>   _PREFIXES = _BINARY_PREFIXES + _DECIMAL_PREFIXES
>>   
>>   # Translated versions of the byte and prefix arrays as lazy comprehensions
>> -# All strings are decoded as utf-8 so that locale-specific upper/lower
>> functions work
>>   def _xlated_bytes():
>> -    return (_(b).decode("utf-8") for b in _BYTES)
>> +    return (_(b) for b in _BYTES)
>>   
>>   def _xlated_prefix(p):
>> -    return _Prefix(p.factor, _(p.prefix).decode("utf-8"),
>> _(p.abbr).decode("utf-8"))
>> +    return _Prefix(p.factor, _(p.prefix), _(p.abbr))
>>   
>>   def _xlated_binary_prefixes():
>>       return (_xlated_prefix(p) for p in _BINARY_PREFIXES)
>> @@ -102,15 +101,15 @@ def _makeSpecs(prefix, abbr, xlate):
>>   
>>       if prefix:
>>           if xlate:
>> -            specs.append(prefix.lower() + _(b"byte").decode("utf-8"))
>> -            specs.append(prefix.lower() + _(b"bytes").decode("utf-8"))
>> +            specs.append(prefix.lower() + _(b"byte"))
>> +            specs.append(prefix.lower() + _(b"bytes"))
>>           else:
>>               specs.append(_lowerASCII(prefix) + "byte")
>>               specs.append(_lowerASCII(prefix) + "bytes")
>>   
>>       if abbr:
>>           if xlate:
>> -            specs.append(abbr.lower() + _(b"b").decode("utf-8"))
>> +            specs.append(abbr.lower() + _(b"b"))
>>               specs.append(abbr.lower())
>>           else:
>>               specs.append(_lowerASCII(abbr) + "b")
>> @@ -172,10 +171,13 @@ def _parseSpec(spec):
>>               if spec_ascii in check:
>>                   return size * factor
>>   
>> -    # No English match found, try localized size specs. Accept any utf-8
>> -    # character and leave the result as a (unicode object.
>> +    # No English match found, try localized size specs. Convert the
>> +    # specifier to a unicode string if it's not one already.
>>       if six.PY2:
>> -        spec_local = specifier.decode("utf-8")
>> +        if isinstance(specifier, unicode):
>> +            spec_local = specifier
>> +        else:
>> +            spec_local = specifier.decode("utf-8")
>>       else:
>>           # str = unicode in Python3
>>           spec_local = specifier
>> @@ -357,7 +359,5 @@ class Size(Decimal):
>>           if radix != '.':
>>               retval_str = retval_str.replace('.', radix)
>>   
>> -        # Convert unicode objects to str before concatenating so that the
>> -        # resulting expression is a str.
>>           # pylint: disable=undefined-loop-variable
>> -        return retval_str + " " + abbr.encode("utf-8") + _("B")
>> +        return retval_str + " " + abbr + _("B")
>> --
>> 2.1.0
>>
>> _______________________________________________
>> anaconda-patches mailing list
>> anaconda-patches at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/anaconda-patches
>>
> If try block under:
>
>   # Only attempt to parse as English if all characters are ASCII
>
> comment still make sense in current situation, Ack.
>
> - mulhern

It's still valid in the sense that it still works, since python2 
unicode.decode('ascii') can be used as a test for non-ascii characters 
in the same way as str.decode('ascii')

 >>> u'asdf'.decode('ascii')
u'asdf'
 >>> 'asdf'.decode('ascii')
u'asdf'
 >>> u'ásdf'.decode('ascii')
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in 
position 0: ordinal not in range

The part where it gets converted back to str may not be the cleanest or 
clearest, but it makes string.maketrans work for the _lowerASCII method. 
I'll change the "Convert back to a str type to match the _BYTES and 
_PREFIXES arrays" comment to mention that, instead.

  In Python 3 there are greater problems than the str being converted to 
bytes, such as string.maketrans not existing at all.


More information about the anaconda-patches mailing list