On Mon, 2009-06-29 at 15:27 -0400, Matthias Clasen wrote:
If you have ideas for
other areas that could benefit from this kind of attention, please let
us know.
I can think of a number of different cross-component tests...
* Audio Applications - Does the sound hardware work out of the box?
Do all applications that produce audio output work out of the box?
(For example, I had trouble in Fedora 9 with Rosegarden, which needs
the Jack audio server set up in addition to PulseAudio, so it
doesn't work out of the box.) Does it work if you try to use the
application while another application is already producing audio
output? Does volume control work? See:
https://fedoraproject.org/wiki/Bugs/Common#PulseAudio-based_volume_contro...
* Network Audio - Is it easy to play music on one computer and listen
to it from the speakers on a different computer on the same subnet?
* File Formats - For each given file type in the known universe (.txt,
.doc, .jpg, .ogg, .mp3, .iso, etc.) are they handled properly by
each application you would expect to do so? Files not handled
internally should be passed off to the right application by all web
browsers, IRC/instant message clients, Nautilus, etc. All of these
applications should use the same universal list of handlers. If the
necessary handler is not installed, one of these new features should
be triggered:
http://fedoraproject.org/wiki/Releases/FeatureCodecBuddy
https://fedoraproject.org/wiki/Features/AutoFontsAndMimeInstaller
* Help - Does activating the help system in each application work?
Does it actually provide helpful information? Is documentation for
non-network-oriented applications accessible even without a network
connection?
* Application Capabilities - Are applications honest about what file
types they can open (e.g. when you select File -> Open from the
menu, is there a filter which shows you only openable files)? If a
given application can handle a given type of file, does that show up
in the list of handlers in other applications (like a web browser)?
If you open a file from within an application which it cannot
handle, does something sensible happen (as opposed to crashing,
getting gibberish, etc.)? Do all applications handle non-ASCII
filenames properly?
* Cut and Paste - Does it work smoothly everywhere, whether using
mouse or keyboard shortcuts? Even in the To: and Cc: fields when
composing an email? Even in the URL bar? In gnome-terminal? Does
complex data (e.g. formatted text, graphics, multi-line text in
single-line input boxes) get transferred properly? Most
importantly, does cut-and-paste handle non-ASCII text encodings
properly (without corrupting characters)?
* International Content - Inside documents and in their own menus and
GUI, applications should generally be able to render glyphs in any
language or Unicode symbol set. If a suitable font is not
installed, this should kick in:
https://fedoraproject.org/wiki/Features/AutoFontsAndMimeInstaller
In some cases, suitable free fonts do not yet exist, but these cases
are generally quite exotic. A complex example:
https://bugzilla.redhat.com/show_bug.cgi?id=455510
(Undisplayable glyphs on Wikipedia)
* Menus - When someone unfamiliar with the layout thinks, "I want to
do ___", is it easy to navigate to that function in the Gnome menus?
If you install Everything, are all programs filed in the right
place? Do all programs have a suitable menu icon? Is menu
placement consistent with Add/Remove groups (e.g. Sound & Video vs
Multimedia)?
* Small Screens - Is everything still usable even when the screen
resolution is 800x600 or 648x480? Most importantly, no
functionality should be lost. One good test is to open each
application and go to Edit -> Preferences; applications with too
many preferences will often produce a window that is not scrollable
even though it is too big to fit on the screen. Less important but
good to test, is whether merely ugly or annoying things happen (like
an application starting up with a window that is larger than the
screen) which could be handled more gracefully. Is there a Fedora
desktop configuration suitable for PDAs or other small mobile
devices?
* Press All the Buttons - Do all the keys you never use (Print Screen,
Pause/Break, Windows, Menu, Play/Stop/Fast Forward, web shortcuts,
etc.) actually do something sensible, both on the desktop and in
the context of the command line and any given application
(e.g. watching a video). Handy links to descriptions of traditional
behavior:
http://en.wikipedia.org/wiki/Keyboard_(computing)
http://en.wikipedia.org/wiki/Category:Computer_keys
* Persistence - Do your desktop settings stay in place after you
logout? After you reboot? What about numlock, capslock, etc?
* Desktop Stress Test - Add lots of applets to a panel. Do sensible
things happen when notification icons appear? Do all the applets
work as advertised? Are any of them redundant? Open multiple
documents in multiple applications (OpenOffice, Firefox, Emacs,
Gimp, etc.). Does the default configuration do sensible things
(it's easy to find the document you are looking for using the window
list)? What about alternate window-navigation tools? What about if
you add multiple workspaces to the mix?
* Working Remotely - Do applications function properly if you are
running them remotely via ssh or other tool? Some known problems
are related to:
https://bugzilla.redhat.com/show_bug.cgi?id=484171
(ssh doesn't forward dbus)
Applications should function acceptably even given this limitation.
* Crashes - All crashes should bring up a user-friendly assistant to
report the problem. Bug reports should be successfully filed in a
suitable bug tracker without any special knowledge on the part of
the reporter. Crashes can be induced from the command line with
"kill -SEGV". More info:
http://fedoraproject.org/wiki/Features/ABRT
(Some applications use different crash-handling software.)
It would also be nice if applications recovered from crashes without
significant data loss. (Can be tested by opening a document and
making several changes without saving, waiting a little while to let
any periodic auto-save kick in, then inducing a crash.) Obviously
any real crashes that are encountered during testing should be
reported, since they shouldn't happen in the first place.
As testing days are finished, it might be interesting to output a
not-very-detailed but growing list similar to
http://fedoraproject.org/wiki/QA/ReleaseCriteria but for "things the
user interface should do seamlessly". As in, "if it's not doing this
for you, you should file a bug" and also as guide for future re-testing.
Or maybe it's more efficient to leave tracking up to Bugzilla and just
trust people to know something is wrong (whether it's a bug or a design
flaw) when they see it.
-B.