This email may be a little late, but at tonight's meeting I would like to talk about possible ways to preform QA for docs, I have two possible suggestions, and any others would be welcome.
The first possibility is to require all changes to be entered as patches in bugzilla, each patch would then have to be verified by a QA person prior to being included in the guide. I don't particularly like this idea as a content writer, I don't really want to have to submit tickets to bugzilla every time I make changes to a guide that I am the POC on.
The second idea is to write to the proposed docs-qa list to inform the QA team that a doc has been changed and need to be looked over. We may be able to semi-automate this process by emailing the list when commits are pushed a repo.
It may also be possible make use of both of these ideas depending on the circumstance (patches from outside sources are checked via bugzilla when a bug is submitted and internal work is done on the docs-qa list).
Zach Oglesby
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 02:52 AM, Zach Oglesby wrote:
This email may be a little late, but at tonight's meeting I would like to talk about possible ways to preform QA for docs, I have two possible suggestions, and any others would be welcome.
The first possibility is to require all changes to be entered as patches in bugzilla, each patch would then have to be verified by a QA person prior to being included in the guide. I don't particularly like this idea as a content writer, I don't really want to have to submit tickets to bugzilla every time I make changes to a guide that I am the POC on.
While this may be a cumbersome solution, it may also be the easiest to manage. If I make a change in the third paragraph on the second page of X guide then I have to convey that in some way to the QA person and we need to track it to make sure that it gets done.
In my opinion, using BZ to track the issue from start to patch to QA to inclusion is rather simple compared to doing this on the wiki or email or...
The second idea is to write to the proposed docs-qa list to inform the QA team that a doc has been changed and need to be looked over. We may be able to semi-automate this process by emailing the list when commits are pushed a repo.
You'd still need to know what changed and be able to approve or disapprove that change. Once it is in the repo it is often too late to pull back out and make more changes or know if something has already been through QA. I'd prefer to not have anything go into the repo until it has been approved.
It may also be possible make use of both of these ideas depending on the circumstance (patches from outside sources are checked via bugzilla when a bug is submitted and internal work is done on the docs-qa list).
It would also be good if everyone on the QA team was fluent in DocBook XML so tags could be checked or could be added where needed.
Zach Oglesby
- --Eric
I'm a little bit torn on this, so I'll think aloud on list since I won't be able to be at tonight's meeting...
For externally-reported bugs, BZ is the way to go. The patch, no matter who provides it, should be QA'ed before going into the repo.
For new (or newish content), e.g. a new chapter in a guide, I'm less thrilled about tossing a huge patch or series of patches into a BZ ticket. There needs to be some kind of process, but I don't know what that should look like.
At any rate, whatever process we develop for QA is better than what we've got now.
On Wed, 2010-11-24 at 08:23 -0500, Eric "Sparks" Christensen wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 02:52 AM, Zach Oglesby wrote:
The first possibility is to require all changes to be entered as patches in bugzilla, each patch would then have to be verified by a QA person prior to being included in the guide. I don't particularly like this idea as a content writer, I don't really want to have to submit tickets to bugzilla every time I make changes to a guide that I am the POC on.
While this may be a cumbersome solution, it may also be the easiest to manage. If I make a change in the third paragraph on the second page of X guide then I have to convey that in some way to the QA person and we need to track it to make sure that it gets done.
No kidding cumbersome. Probably triples the work, and why, just to get it into BZ? There is a commit message automatically generated with every change, we can easily roll back the change in git, and it doesn't affect anything but the repo until we publish, so adding a huge amount of manual work just to make the commit message show up in BZ sounds incredibly dumb beyond all belief.
That being said, I do like the idea of tracking everything in BZ, but this is just horrible.
The second idea is to write to the proposed docs-qa list to inform the QA team that a doc has been changed and need to be looked over. We may be able to semi-automate this process by emailing the list when commits are pushed a repo.
It happens now. If you subscribe to docs-commits you get it. Nothing to be done.
You'd still need to know what changed and be able to approve or disapprove that change. Once it is in the repo it is often too late to pull back out and make more changes or know if something has already been through QA. I'd prefer to not have anything go into the repo until it has been approved.
The whole point of an RCS of any flavor is that you CAN roll back a change. If we want to know something has been QAed we could use git tags. Further, if we don't push a change until something has been QAed in BZ, then we frustrate the advantages of a collaborative RCS. People would need to wait for QA before doing any change, and EVERYONE would need to know what was waiting in the wings to be pushed. So, we would need another list or database or something with the QA status of everything, preferably automatically updated with "wait a minute" whenever someone did a clone with the intention of editing.
Another thought....
1) Encourage people to commit and push frequently. I think sometimes people are reluctant to push and that leads to potential conflicts. Push ALWAYS, you can always back it out.
2) When you get to a reasonable stopping point, tag your commit with a "needs QA" tag, we would need to work out what that looks like because there could be several.
3) When something is QAed, it gets tagged as QAed.
4) Publish from a QAed tag.
Very little extra work, far less potential for problems. Only downside is that it doesn't show up in BZ.
Certainly an email filter on QA tags could be written to identify docs-commits messages about tags. Some bright bash guy might even figure out how to create a BZ entry on a "Needs QA" tag.
It would also be good if everyone on the QA team was fluent in DocBook XML so tags could be checked or could be added where needed.
Well, we need that in the POs a lot more than in the XML. And Publican finds that for you. The new Publican is fast enough that it isn't very painful to use it for a syntax check, especially now that we don't need to do the separate merge. Admittedly, finding the line in the PO that caused the error is sometime painful, but I would hope that anybody making edits at least does a build in the original language to find any tagging errors in the XML.
--McD
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 09:35 AM, John J. McDonough wrote:
On Wed, 2010-11-24 at 08:23 -0500, Eric "Sparks" Christensen wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 02:52 AM, Zach Oglesby wrote:
The first possibility is to require all changes to be entered as patches in bugzilla, each patch would then have to be verified by a QA person prior to being included in the guide. I don't particularly like this idea as a content writer, I don't really want to have to submit tickets to bugzilla every time I make changes to a guide that I am the POC on.
While this may be a cumbersome solution, it may also be the easiest to manage. If I make a change in the third paragraph on the second page of X guide then I have to convey that in some way to the QA person and we need to track it to make sure that it gets done.
No kidding cumbersome. Probably triples the work, and why, just to get it into BZ? There is a commit message automatically generated with every change, we can easily roll back the change in git, and it doesn't affect anything but the repo until we publish, so adding a huge amount of manual work just to make the commit message show up in BZ sounds incredibly dumb beyond all belief.
It isn't that cumbersome. It would take me less than a minute to post a patch on BZ.
That being said, I do like the idea of tracking everything in BZ, but this is just horrible.
Horrible?
The second idea is to write to the proposed docs-qa list to inform the QA team that a doc has been changed and need to be looked over. We may be able to semi-automate this process by emailing the list when commits are pushed a repo.
It happens now. If you subscribe to docs-commits you get it. Nothing to be done.
But once the change is in the repo it is too late. You shouldn't commit a change before it is QA'd. If the change is in the repo then there is a high chance that the change will make it to the final guide without being checked.
That said we could go the other way and just commit all the changes and have a QA person go over the entire document before release.
You'd still need to know what changed and be able to approve or disapprove that change. Once it is in the repo it is often too late to pull back out and make more changes or know if something has already been through QA. I'd prefer to not have anything go into the repo until it has been approved.
The whole point of an RCS of any flavor is that you CAN roll back a change. If we want to know something has been QAed we could use git tags. Further, if we don't push a change until something has been QAed in BZ, then we frustrate the advantages of a collaborative RCS. People would need to wait for QA before doing any change, and EVERYONE would need to know what was waiting in the wings to be pushed. So, we would need another list or database or something with the QA status of everything, preferably automatically updated with "wait a minute" whenever someone did a clone with the intention of editing.
The pending list already exists... it's called BZ.
Another thought....
- Encourage people to commit and push frequently. I think sometimes
people are reluctant to push and that leads to potential conflicts. Push ALWAYS, you can always back it out.
When was the last time you tried to roll back an edit that was made several pushes ago?
- When you get to a reasonable stopping point, tag your commit with a
"needs QA" tag, we would need to work out what that looks like because there could be several.
When something is QAed, it gets tagged as QAed.
Publish from a QAed tag.
I'm good with 2-4. I hadn't really thought about doing something like this. I don't know that this is possible in SVN, though.
Very little extra work, far less potential for problems. Only downside is that it doesn't show up in BZ.
Certainly an email filter on QA tags could be written to identify docs-commits messages about tags. Some bright bash guy might even figure out how to create a BZ entry on a "Needs QA" tag.
nb?
It would also be good if everyone on the QA team was fluent in DocBook XML so tags could be checked or could be added where needed.
Well, we need that in the POs a lot more than in the XML. And Publican finds that for you. The new Publican is fast enough that it isn't very painful to use it for a syntax check, especially now that we don't need to do the separate merge. Admittedly, finding the line in the PO that caused the error is sometime painful, but I would hope that anybody making edits at least does a build in the original language to find any tagging errors in the XML.
Well, I was talking more about a tag that isn't there but should/could be and not a broken tag.
--McD
- -- Eric
On Wed, Nov 24, 2010 at 9:35 AM, John J. McDonough wb8rcr@arrl.net wrote:
On Wed, 2010-11-24 at 08:23 -0500, Eric "Sparks" Christensen wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 02:52 AM, Zach Oglesby wrote:
The first possibility is to require all changes to be entered as patches in bugzilla, each patch would then have to be verified by a QA person prior to being included in the guide. I don't particularly like this idea as a content writer, I don't really want to have to submit tickets to bugzilla every time I make changes to a guide that I am the POC on.
While this may be a cumbersome solution, it may also be the easiest to manage. If I make a change in the third paragraph on the second page of X guide then I have to convey that in some way to the QA person and we need to track it to make sure that it gets done.
No kidding cumbersome. Probably triples the work, and why, just to get it into BZ? There is a commit message automatically generated with every change, we can easily roll back the change in git, and it doesn't affect anything but the repo until we publish, so adding a huge amount of manual work just to make the commit message show up in BZ sounds incredibly dumb beyond all belief.
That being said, I do like the idea of tracking everything in BZ, but this is just horrible.
I snipped a lot.
After seeing the problems with lack of QA/testing in the current update process for packages, I have to agree. I think we are likely struggling to keep up. (I know my own participation has been limited to a few patches to IG/IQSG this cycle.) That means that adding additional QA layer onto things will mean 1) things either don't get pushed because they haven't been QA'ed or worse, we'll go to all the trouble, they won't get QA'ed because people don't have the time, and we'll end up pushing the changes anyway.
I don't disagree with the need for additional QA, but I'd offer several suggestions. 1. apply to a small group first - a single guide - see if it scales. 2. simple is better.......sadly I don't have any suggestion for simple here - and of necessity, writing/editing/publishing the guides is already relatively complicated (though far simpler than in years past)
On Wed, 2010-11-24 at 10:22 -0500, Eric "Sparks" Christensen wrote:
It isn't that cumbersome. It would take me less than a minute to post a patch on BZ.
I challenge you to do ANYTHING in BZ in a minute. But posting the patch seems silly. Post the commmit. You need the commit anyway to identify the particular commit that was QAed.
But once the change is in the repo it is too late. You shouldn't commit a change before it is QA'd. If the change is in the repo then there is a high chance that the change will make it to the final guide without being checked.
No, no, no. The whole point of a repo like git is to make working collaboratively easy. For that to work you need to commit frequently. The idea that somehow what is in the repo is sacred is a view that makes working together hard. Sure, it may have held some sway back in the RCS days before collaborative RCSs were available, or maybe even with CVS where doing anything is so bloody hard and fraught with risk, but wit git there is no argument not to commit, certainly not if you have a way to identify QAed commits.
For a lot of the guides you can get away with it because there is only one person working on the guide. If more than one are working at the same time you absolutely must commit frequently or you stand the risk of tripping all over each other.
The pending list already exists... it's called BZ.
It only exists if you go through all the extra manual steps. And finding what you want to publish could be a real challenge. Sure, for QA its easy, just search on open bugs with a QA needed tag. But when you want to publish something, you will be looking in closed bugs which eventually will get to be quite a list. And how do you know what release? Do you go searching through all the commits until you find the matching patch?
When was the last time you tried to roll back an edit that was made several pushes ago?
I fail to see what's so tough about git checkout QA-DONE-17 or whatever. The only messy bit I see is that you would probably want to keep the history of QAed commits, so each completed QA would want to have a new name, probably a serial number. I could see multiple QA requests as well, although I see less need for keeping the history.
I'm good with 2-4. I hadn't really thought about doing something like this. I don't know that this is possible in SVN, though.
It is. Of course, like everything in SVN it's slower and more complicated, but the tagging idea is pretty similar. It's just that SVN has no concept of a global commit, so the author has to be careful to tag a complete, coherent set of commits.
nb?
That was the first thought that came to my mind.
Well, I was talking more about a tag that isn't there but should/could be and not a broken tag.
I hadn't thought about that. That could be a big win.
--McD
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2010 11:03 AM, John J. McDonough wrote:
On Wed, 2010-11-24 at 10:22 -0500, Eric "Sparks" Christensen wrote:
But once the change is in the repo it is too late. You shouldn't commit a change before it is QA'd. If the change is in the repo then there is a high chance that the change will make it to the final guide without being checked.
No, no, no. The whole point of a repo like git is to make working collaboratively easy. For that to work you need to commit frequently. The idea that somehow what is in the repo is sacred is a view that makes working together hard. Sure, it may have held some sway back in the RCS days before collaborative RCSs were available, or maybe even with CVS where doing anything is so bloody hard and fraught with risk, but wit git there is no argument not to commit, certainly not if you have a way to identify QAed commits.
Can you mark pages as draft or just entire documents?
--McD
- --Eric
-
Ben Cotton -
David Nalley -
Eric Christensen -
John J. McDonough -
Zach Oglesby