On Wed, 2010-08-25 at 01:10 -0400, Toshio Kuratomi wrote:
On Tue, Aug 24, 2010 at 08:14:11PM -0400, David Malcolm wrote:
> On Tue, 2010-08-24 at 18:28 -0400, David Malcolm wrote:
> > Profiling of a Fedora 14 "yum install *" showed it spending 80
seconds
> > out of 274 within its implementation of "str_eq" (24 million calls),
so
> > optimizing this seems worthwhile.
>
> (replying to self, sorry)
>
> I added some instrumentation to my copy of yum/i18n.py:
>
> str_eq_calls = {}
> def str_eq(a, b):
> """ convert between unicode and not and compare them, w/o warning
or
> being annoying"""
> typesig = (type(a), type(b))
> if typesig in str_eq_calls:
> str_eq_calls[typesig] += 1
> else:
> str_eq_calls[typesig] = 1
> (rest of function follows)
>
>
> to capture the type signature of all of these calls, adding:
> from yum.i18n import str_eq_calls
> from pprint import pprint
> pprint(str_eq_calls)
> to yummain.py, to dump the stats on output.
>
> The output indicates that in this case at least [1], all of the calls
> are for the str vs str case:
>
> {(<type 'str'>, <type 'str'>): 24274562}
>
Hah! So there's a few things that we should think about here:
* We're dealing almost exclusively with byte strings here.
* This function exists because we must occassionally get unicode strings
* I count exactly two places in the yum code where this function is called
There's multiple ways that we could approach this issue. We could track
down where we're getting unicode strings from and make sure that it's
converted to (or remains a) byte string instead.
We could move the str_eq code back to the two places that it exists and put
it there. (Function calls are expensive in python... 24 million function
calls can add up).
We could decide that it's okay to issue a UnicodeWarning here (not
a UnicodeError -- we'd need to keep a patch for RHEL5/python-2.4.x systems
to pass errors) and just use str1 == str2. The two effects of this would
be:
1) UnicodeWarning would be issued whenever data that is unicode gets
compared to the data that is byte str.
Of course the giant opt. threads happen when I'm on holiday! Anyway, I
don't mind any of this (I'm unlikely to NAK anything reasonable),
however I do have a couple of minor concerns:
1. If a bug ever opened saying "yum install '*'" in Fedora was slow,
I'd
be more likely to CLOSE => DONTCARE than anything else.
2. Following on from #1, I'm intrigued about how much str_eq is called
in any kind of common case.
3. Following on from #2, I'm intrigued if we can speed up "install
'*'"
with a tiny API change and something in the depsolver (assuming it was
worth it :).
4. IIRC the main reason str_eq() was introduced was because Fedora _had_
to allow utf-8 provides, and so we got bugs about:
"yum install 'font(花園明朝)'
...and we'll get them again if we start issuing warnings.
5. Following on from #4, IIRC the reason we didn't keep the utf-8
provides as bytes (and thus. not have this bug) was because then
install/provides commands on the above argument caused other bugs.
6. Given:
yum provides font\* | fgrep 'font(' | \
LANG=C egrep -v 'font\([:[:alpha:]]' | nl
...I'm half tempted to suggest someone campaign to outlaw utf-8 in
provides, on the grounds it'll speed up yum a lot :).