On Mon, 2014-07-28 at 09:16 -0400, Kamil Paral wrote:
> Hah - when i was two paragraphs into your mail, I started
thinking 'hmm,
> we could amend the Result template' :P - great minds think alike, it
> seems...
>
> In a way it'd be cleaner to have the image as part of the template, but
> then you'd have to 'pad' the entry like this:
>
> {{result|pass|adamwill||||TC1}}
If you think it's better to have it as a part of the template (probably is),
let's change to order:
{{result|pass|adamwill|TC1}}
{{result|fail|adamwill|TC1|123456}}
{{result|fail|adamwill|TC1|123456|111222}}
I think that's no less readable than the original schema.
It isn't at all, but it has the rather large problem that it breaks the
API :) All existing {{result}} entries in all pages will be invalid.
That seems like a deal-breaker. We're kind of locked forever into the
first five fields being 'result', 'username', 'bug#',
'bug#', 'bug#', I
think, to preserve existing results.
>
> which would kinda trip people up (you'd have to remember the template
> has space for exactly three bug references, and pad correctly...recipe
> for disaster, I guess.) Well, we could try and get clever in the
> template and heuristically determine if any of the parameters after the
> username is a bug ID or a build and behave accordingly, but that may be
> gilding the lily. :P
I haven't studied the templating markup language too much, but I doubt
you can express such programming structures in there.
I had a quick look and I suspect you can, but it'd get really pretty
messy.
But if we place the TC identifier before the bug numbers, as seen
above, the problem should be solved.
>
> We *do* have
http://testdays.qa.fedoraproject.org/testcase_stats/ ,
> which is kind of in the 'cute nonsense hack' category but does work
> pretty well, to the extent that I've started using it as an entry point
> for testing, sometimes.
I had a quick chat with Josef about this. There is a problem with
table column in our matrices. Historically we used it mainly for
separating architectures and they used to be pretty similar between
the tables. With F21, the column headers now look like:
Server Workstation Spins ARM disks Cloud
i386 x86_64 UEFI ARM
i386 x86_64 EFI
Ext boot VFAT boot
i386 x86_64
i386 x86_64 ARM
x86[1] ARM
x86 BIOS[1] x86 UEFI ARM
LVM ext4 NTFS BTRFS
i386 x86_64 EFI ARM
That requires non-trivial semantic hardcoding inside the script (for
example by the section name), if we don't want to squash all columns
together a produce a single number. As long it only differed by
architectures, it wasn't such a problem, usually the bugs were not
specific to architecture. But now there are large differences between
the columns, e.g. Server vs Workstation vs Cloud, or LVM vs ext4 vs
BTRFS.
So, some maintenance would be needed to make the script work with F21
matrices, and either we need to hardcode a pile of hacks which would
get broken by every template update, or we would need to skip certain
section tables completely. That's why I'm looking into alternative
approaches, which would be a bit more manageable, and rawhide-style
wiki matrices seem like a viable approach to try.
Yeah, I can see the thought process there.
>
> We also do have an existing mechanism and policy for transferring
> results from one build to the next, but doing so is pretty clunky and
> it'd definitely be easier just to use the approach we've been using on
> the monthly pages.
>
> So...I guess I'm percolating this one, it seems decent but I just want
> to think about it a bit more. What do other folks think?
My only concern is that things would get out of hand and the tables
would start to grow a lot. But in that case we can either prune it by
hand when needed, or refresh the whole page (for example we have a
single page for TC1/TC2/TC3, then there's a lot of results, so we
create a fresh new page for TC4-TCx. Rinse and repeat). In this case
it would still be relatively easy to spot test cases which were not
tested in a long time, because you would have approximately just one
historical page (going not too far into history) to check and compare
to the current one, which is manageable (if the current page is for
TC4-TC5, and I can see results for TC1-TC3 summarized on a single
historic page, I can then easily compare blank cells).
I guess another problem here is it makes the process somewhat more
complex; right now we have a fairly simple SOP for doing a test release
event, which is 'create the results pages and send out the mails'. I
guess it becomes 'decide whether to create a new results page, if so
name it according to this new naming scheme'. Not that bad, I guess, but
a bit more work. How did you envisage naming the pages?
(it also means you can't reliably find the correct results page by just
plugging the appropriate fields into the known format, I guess, unless
we're careful about setting up redirects.)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net