On Fri, 2019-09-06 at 09:38 -0400, Danny Lee wrote:
Hello all,
I have a question related to regression and testing updates that I was
hoping somebody might have the time to answer. Thankfully, @kalev was
kind enough to give me an overview of whats going on this test in Bodhi
https://bodhi.fedoraproject.org/updates/FEDORA-2019-18f7fc4f08 but I
have further questions that I thought I'd bring to the mailing list.
Please treat me like a long-experienced (mostly windows) computer
user/troubleshooter, who has never studied CS and only knows fragments
of programming languages at this point in time. I used unixes back in
the 90s, but never became overly skilled/knowledgeable. I've started
watching/listening to videos on regression testing and I think I
understand the overall concept.
That all being said, I thank you for your patience, and the point of my
email is that I have questions related to the practical next steps for
this type of test -
Question 1:
Part A: I installed the .iso at
https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-31/comp...
into a virtual machine (vmware) and started running tests. I ran
into some errors, so would I now go into
https://kojipkgs.fedoraproject.org/compose/branched/ and download
the latest branch prior to the 'latest-Fedora-31'?
Part B: Could I switch my repository inside the installed system to
/etc/yum/repos.d/fedora.repo and uninstall Gnome Desktop (GD) and
then reinstall the GD from that repo and test to see if it worked
previously?
This is a bit tricky if you're thinking in terms of regression testing,
because there isn't exactly a 'known good' base for Fedora 31, or
anything. The way Fedora development works is that we have a constantly
rolling development distribution, Rawhide, which has no real formal
quality requirements so it can be (and sometimes is) completely broken.
New releases are branched from Rawhide periodically and then 'refined'
to meet the quality requirements for stable releases. So a given
release starts out in a nearly-unknown, potentially completely-broken
state and, on average, gets 'better' until it's good enough to release.
So there's no reason that any of the earlier snapshots you find there
will necessarily be better than the current one, though one may
*happen* to be.
For testing an update in updates-testing, if you're thinking in
'regression' terms, the key question is "do things work better with
this update installed than without it". If installing an update makes
things *worse* than before, then it should probably get a -1.
Testing updates for the in-development release (which we call
'Branched' - Fedora 31 is the current 'Branched' release) can
'feel' a
bit different than testing updates for a stable release. If you're
testing an update for a stable release, then it's usually the case that
without the update installed, things are generally working fine. For
Branched this is not always the case, the baseline might be...bad. :)
What I'd generally say, for doing Branched testing, is just install the
system and keep it up to date regularly, and when testing an update,
compare how the packages in that update behave with the update
installed to how they behaved before you installed it. I wouldn't
suggest 'going backwards' to try and find some sort of clean baseline
to compare the update to.
Question 2:
Part A: In filing a bug reports for issues like these, what is
standard operating procedure for where to send the bug (wiki/web
links welcome here). Do I file upstream and at redhat bugzilla
first and then upstream after some period of time? Or do I file
upstream first or both?
The simplest thing to do is always file at Red Hat Bugzilla, against
the Fedora product, against the component that is the source RPM for
the package with the bug ('rpm -qi package' will show you the source
package name). Sometimes it's a good idea to file upstream as well -
usually if the bug appears in a package that provides a new upstream
release, for instance, that's an indication the bug is likely in the
upstream release not the package. But you can't really go too far wrong
by just filing in RHBZ.
If you find a problem in an update, you should always put a comment *on
the update* to explain this, and if it's a significant problem that's
new - i.e. the previous version of the package didn't have the problem
- you should probably give the package -1 karma (click the -1 button
for 'Is the update generally functional? (karma)').
Part B: I received the command 'journalctl -a -b' as
something to
comb through, and adding screenshots, but is there anything else
that you (testers and bug-fixers) might want in your bug reports to
help make the issue clearer from the start?
This is highly contextual, to be honest - it depends a lot on the bug.
There are several 'how to debug' pages on the wiki which provide more
specific info for certain types of bug:
https://fedoraproject.org/wiki/Category:Debugging
these can help if your problem falls into one of those categories;
otherwise, journal output is indeed usually never a bad idea, and
beyond that the bug assignee should be able to tell you what else they
need. It's a good idea to specifically mention in your report that
you're happy to provide additional information, so the assignee knows
it's not just a 'drive-by' report and the effort spent explaining what
additional information is required won't be wasted.
There are some typical things, like, if the failing app has its own log
files...attach those. If the failing app is something you can run at a
console, run it at a console and see if you get any messages on the
console, if so, paste (if short) or attach (if long) those. If the app
has a man page or other docs, check it to see if there's any debugging
tips there, and look for arguments you can use to get more info like '-
-debug' or '--verbose' or '-v' or something.
Once again, thank you for your patience and insights, and thanks to
@kalev for your help. It is all much appreciated in trying to get over
this learning curve :)
No problem, thanks for writing and testing!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net