[blivet master] Handle unicode strings in Size spec parsing.

Anne Mulhern amulhern at redhat.com
Wed Oct 22 15:02:18 UTC 2014





----- Original Message -----
> From: "David Shea" <dshea at redhat.com>
> To: anaconda-patches at lists.fedorahosted.org
> Sent: Wednesday, October 22, 2014 10:03:51 AM
> Subject: Re: [blivet master] Handle unicode strings in Size spec parsing.
> 
> On 10/22/2014 08:48 AM, Anne Mulhern wrote:
> >
> >
> >
> > ----- Original Message -----
> >> From: "David Shea" <dshea at redhat.com>
> >> To: anaconda-patches at lists.fedorahosted.org
> >> Sent: Tuesday, October 21, 2014 5:08:43 PM
> >> Subject: [blivet master] Handle unicode strings in Size spec parsing.
> >>
> >> Since _() and friends now return unicode objects, most of size.py no
> >> longer needs to decode strings into unicode objects. Change
> >> humanReadable to return unicode objects that will be converted back to
> >> str by __str__ if necessary. Handle unicode inputs to the Size
> >> constructor.
> >> ---
> >>   blivet/size.py | 24 ++++++++++++------------
> >>   1 file changed, 12 insertions(+), 12 deletions(-)
> >>
> >> diff --git a/blivet/size.py b/blivet/size.py
> >> index ec09bf0..6dafdcc 100644
> >> --- a/blivet/size.py
> >> +++ b/blivet/size.py
> >> @@ -75,12 +75,11 @@ _BYTES = [N_(b'B'), N_(b'b'), N_(b'byte'),
> >> N_(b'bytes')]
> >>   _PREFIXES = _BINARY_PREFIXES + _DECIMAL_PREFIXES
> >>   
> >>   # Translated versions of the byte and prefix arrays as lazy
> >>   comprehensions
> >> -# All strings are decoded as utf-8 so that locale-specific upper/lower
> >> functions work
> >>   def _xlated_bytes():
> >> -    return (_(b).decode("utf-8") for b in _BYTES)
> >> +    return (_(b) for b in _BYTES)
> >>   
> >>   def _xlated_prefix(p):
> >> -    return _Prefix(p.factor, _(p.prefix).decode("utf-8"),
> >> _(p.abbr).decode("utf-8"))
> >> +    return _Prefix(p.factor, _(p.prefix), _(p.abbr))
> >>   
> >>   def _xlated_binary_prefixes():
> >>       return (_xlated_prefix(p) for p in _BINARY_PREFIXES)
> >> @@ -102,15 +101,15 @@ def _makeSpecs(prefix, abbr, xlate):
> >>   
> >>       if prefix:
> >>           if xlate:
> >> -            specs.append(prefix.lower() + _(b"byte").decode("utf-8"))
> >> -            specs.append(prefix.lower() + _(b"bytes").decode("utf-8"))
> >> +            specs.append(prefix.lower() + _(b"byte"))
> >> +            specs.append(prefix.lower() + _(b"bytes"))
> >>           else:
> >>               specs.append(_lowerASCII(prefix) + "byte")
> >>               specs.append(_lowerASCII(prefix) + "bytes")
> >>   
> >>       if abbr:
> >>           if xlate:
> >> -            specs.append(abbr.lower() + _(b"b").decode("utf-8"))
> >> +            specs.append(abbr.lower() + _(b"b"))
> >>               specs.append(abbr.lower())
> >>           else:
> >>               specs.append(_lowerASCII(abbr) + "b")
> >> @@ -172,10 +171,13 @@ def _parseSpec(spec):
> >>               if spec_ascii in check:
> >>                   return size * factor
> >>   
> >> -    # No English match found, try localized size specs. Accept any utf-8
> >> -    # character and leave the result as a (unicode object.
> >> +    # No English match found, try localized size specs. Convert the
> >> +    # specifier to a unicode string if it's not one already.
> >>       if six.PY2:
> >> -        spec_local = specifier.decode("utf-8")
> >> +        if isinstance(specifier, unicode):
> >> +            spec_local = specifier
> >> +        else:
> >> +            spec_local = specifier.decode("utf-8")
> >>       else:
> >>           # str = unicode in Python3
> >>           spec_local = specifier
> >> @@ -357,7 +359,5 @@ class Size(Decimal):
> >>           if radix != '.':
> >>               retval_str = retval_str.replace('.', radix)
> >>   
> >> -        # Convert unicode objects to str before concatenating so that the
> >> -        # resulting expression is a str.
> >>           # pylint: disable=undefined-loop-variable
> >> -        return retval_str + " " + abbr.encode("utf-8") + _("B")
> >> +        return retval_str + " " + abbr + _("B")
> >> --
> >> 2.1.0
> >>
> >> _______________________________________________
> >> anaconda-patches mailing list
> >> anaconda-patches at lists.fedorahosted.org
> >> https://lists.fedorahosted.org/mailman/listinfo/anaconda-patches
> >>
> > If try block under:
> >
> >   # Only attempt to parse as English if all characters are ASCII
> >
> > comment still make sense in current situation, Ack.
> >
> > - mulhern
> 
> It's still valid in the sense that it still works, since python2
> unicode.decode('ascii') can be used as a test for non-ascii characters
> in the same way as str.decode('ascii')
> 
>  >>> u'asdf'.decode('ascii')
> u'asdf'
>  >>> 'asdf'.decode('ascii')
> u'asdf'
>  >>> u'ásdf'.decode('ascii')
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in
> position 0: ordinal not in range
> 
> The part where it gets converted back to str may not be the cleanest or
> clearest, but it makes string.maketrans work for the _lowerASCII method.
> I'll change the "Convert back to a str type to match the _BYTES and
> _PREFIXES arrays" comment to mention that, instead.

OK, thanks.
 
>   In Python 3 there are greater problems than the str being converted to
> bytes, such as string.maketrans not existing at all.
> 

Hmmm, yeah, I see we're not so Python 3 compatible yet.

_______________________________________________
> anaconda-patches mailing list
> anaconda-patches at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/anaconda-patches
> 

- mulhern


More information about the anaconda-patches mailing list