On Sat, Aug 28, 2010 at 02:29:12PM -0400, James Antill wrote:
On Wed, 2010-08-25 at 01:10 -0400, Toshio Kuratomi wrote:
> On Tue, Aug 24, 2010 at 08:14:11PM -0400, David Malcolm wrote:
> > On Tue, 2010-08-24 at 18:28 -0400, David Malcolm wrote:
> > > Profiling of a Fedora 14 "yum install *" showed it spending 80
seconds
> > > out of 274 within its implementation of "str_eq" (24 million
calls), so
> > > optimizing this seems worthwhile.
> >
> > (replying to self, sorry)
> >
> > I added some instrumentation to my copy of yum/i18n.py:
> >
> > str_eq_calls = {}
> > def str_eq(a, b):
> > """ convert between unicode and not and compare them, w/o
warning or
> > being annoying"""
> > typesig = (type(a), type(b))
> > if typesig in str_eq_calls:
> > str_eq_calls[typesig] += 1
> > else:
> > str_eq_calls[typesig] = 1
> > (rest of function follows)
> >
> >
> > to capture the type signature of all of these calls, adding:
> > from yum.i18n import str_eq_calls
> > from pprint import pprint
> > pprint(str_eq_calls)
> > to yummain.py, to dump the stats on output.
> >
> > The output indicates that in this case at least [1], all of the calls
> > are for the str vs str case:
> >
> > {(<type 'str'>, <type 'str'>): 24274562}
> >
> Hah! So there's a few things that we should think about here:
>
> * We're dealing almost exclusively with byte strings here.
> * This function exists because we must occassionally get unicode strings
> * I count exactly two places in the yum code where this function is called
>
> There's multiple ways that we could approach this issue. We could track
> down where we're getting unicode strings from and make sure that it's
> converted to (or remains a) byte string instead.
>
> We could move the str_eq code back to the two places that it exists and put
> it there. (Function calls are expensive in python... 24 million function
> calls can add up).
>
> We could decide that it's okay to issue a UnicodeWarning here (not
> a UnicodeError -- we'd need to keep a patch for RHEL5/python-2.4.x systems
> to pass errors) and just use str1 == str2. The two effects of this would
> be:
> 1) UnicodeWarning would be issued whenever data that is unicode gets
> compared to the data that is byte str.
Of course the giant opt. threads happen when I'm on holiday! Anyway, I
don't mind any of this (I'm unlikely to NAK anything reasonable),
however I do have a couple of minor concerns:
1. If a bug ever opened saying "yum install '*'" in Fedora was slow,
I'd
be more likely to CLOSE => DONTCARE than anything else.
2. Following on from #1, I'm intrigued about how much str_eq is called
in any kind of common case.
Dmalcolm could instrument this but my testing of
time yum install 'l*' 'python-*' --skip-broken resulted in:
2300 packages to install, 22 to update. Everything ran in the range of 50-52s
with the known best case (inlining reqn == n in the yum code) not getting
consistent times better than the worst case (current code).
3. Following on from #2, I'm intrigued if we can speed up "install
'*'"
with a tiny API change and something in the depsolver (assuming it was
worth it :).
For dmalcolm's pathological transaction he was able to come up with the
following timings:
1m44s original code
1m35s cython compiled depsolve.so and misc.so
1m20s my new str_eq
1m17s str_eq that does str1 == str2 (similar to code that would be
necessary to not throw an error on old python)
1m16s my new str_eq, cython compiled
1m15s str_eq, str1 == str2, cython compiled
1m13s inlined reqn == n
4. IIRC the main reason str_eq() was introduced was because Fedora
_had_
to allow utf-8 provides, and so we got bugs about:
"yum install 'font(花園明朝)'
...and we'll get them again if we start issuing warnings.
Yeah, I think so too. I didn't write code to just shut up this warning
while we're in str_eq() but I suspect that would be slower than my new
str_eq().
5. Following on from #4, IIRC the reason we didn't keep the
utf-8
provides as bytes (and thus. not have this bug) was because then
install/provides commands on the above argument caused other bugs.
Hmm.. something to look into if we ever do a complete review of the way we
handle unicode and byte strings.
6. Given:
yum provides font\* | fgrep 'font(' | \
LANG=C egrep -v 'font\([:[:alpha:]]' | nl
...I'm half tempted to suggest someone campaign to outlaw utf-8 in
provides, on the grounds it'll speed up yum a lot :).
Heh. You don't even need that. If you could successfully campaign to have
a single encoding in rpm you'd be able to get rid of things like this. That
encoding doesn't need to be ASCII.
-Toshio