On Tue, 2017-11-14 at 05:04 +0000, Nick Coghlan wrote:
> Dear Sumantro,
>
> Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from
DNF
> QE team. There is a tool to test DNF and I know Karel already has a good set of
tests
> which covers Modular side of DNF [1], just missing some user stories.
We're also trying to keep track of the places where automated testing
happens as part of the Fedora module developer's guide at
https://doc
s.pagure.org/modularity/development/building-modules/testing.html
I think the key item we have there that hasn't been mentioned in the
thread yet is your own automated tests at
https://github.com/fedora-m
odularity/compose-tests
Cheers,
Nick.
P.S. It's entirely plausible that we're missing some aspects of the
automated testing setup for the modular server builds, so issue
reports and PRs to correct omissions at
https://pagure.io/modularity
are most welcome.
Note that there is a specific purpose to these test cases: they are for
release validation testing. Release validation testing is about
ensuring critical functionality works *as experienced by end users*.
That is, for release validation testing, we want to test that the most
common and critical interactions with modules that will be performed by
admins of actual Fedora Server installations work, exactly as those
admins would perform them. These tests need to run from clean installs
of 'real' composes, and involve using the tools as real-world users
will use them. So any tests that involve more artificial environments
or interactions are not as useful for release validation purposes.
Tests that test "too much" are also not as useful, because we don't
immediately know whether a failure is something critical to the release
process.
So, we can certainly look at existing tests / tools and see if this
testing can share resources with them, but it needs to take the above
considerations into account.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net