#420: Some builds are tested many times over ---------------------+------------------------ Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: tests | Keywords: Blocked By: | Blocking: ---------------------+------------------------ mkasik asked me why he has received 19 test reports from rpmlint and rpmguard about poppler-0.18.4-3.fc18 in the last few days. It seems we really tested the same build 19 times:
http://autoqa.fedoraproject.org/resultsdb/frontend/search?type=Envr&term...
I looked into autoqa.log on production server, the koji-bodhi watcher really detected it as new so many times. And it seems a lot of other builds as well. What might be the cause?
I have looked into /var/cache/autoqa/watch-koji- builds/koji_history_prevtimes.cache and there is no timestamp for f18 repository, which I find weird. But maybe I just don't understand the file properly.
Anyone feels like taking this?
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: assigned Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+------------------------- Changes (by jskladan):
* owner: => jskladan * status: new => assigned
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: assigned Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+-------------------------
Comment (by jskladan):
For some (yet) unexplained reason, sometimes the 'prevtime' timestamp for the respective repos (updates_candidate) gets set to 1336130927.25 This is weird for two reasons 1) the prevtime is set to the completion_ts of the last build returned from koji 2) there is no build with this exact completion_ts (at least I was not able to find it)
So I added logging of the whole set of builds returned from the koji query, and we'll wait for it to happen again.
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: closed Priority: major | Milestone: Hot issues Component: tests | Resolution: fixed Keywords: | Blocked By: Blocking: | --------------------+------------------------- Changes (by jskladan):
* status: assigned => closed * resolution: => fixed
Comment:
Since we added the testing outputs, the error did not happen. This might have been caused by autotest being few days 'late' with test executions (but seems unlikely). As no error of this kind happened for last 5 days, and we have no 'error run' logged, I'm closing this bug.
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: reopened Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+------------------------- Changes (by kparal):
* status: closed => reopened * resolution: fixed =>
Comment:
Problem is here again:
http://autoqa.fedoraproject.org/resultsdb/frontend/search?type=Envr&term...
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: reopened Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+-------------------------
Comment (by kparal):
We have disabled all opt-in emails until we find the problem.
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: reopened Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+-------------------------
Comment (by jskladan):
re-enabled the logging in watchers. Hope for the best!
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: reopened Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+-------------------------
Comment (by jskladan):
OK, I've identified the issue - the problem is, that when we check update- candidates builds, there might be some "yet untagged" in Koji. And since koji/bodhi watcher needs to output the kojitag as one of the parameters, we are storing these 'untagged' builds for next runs. This is not really a problem but:
1) to make sure we don't query/test too much builds, we store the last "completion_ts" of all the builds to be tested
2) in next run, we take the stored completion_ts, and ask koji for all builds completed since.
3) in case, when no new builds are returned by koji AND there were some stored 'untagged builds', the last 'completion_ts' is taken from the 'untagged builds' set, therefore setting the last "completion_ts" into past.
Solution should be quite simple, I'm working on it already.
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: reopened Priority: major | Milestone: Hot issues Component: tests | Resolution: Keywords: | Blocked By: Blocking: | --------------------+-------------------------
Comment (by jskladan):
fixed in master, commit be7e11956cb744c0aa64861fe1248c53c742305b
#420: Some builds are tested many times over --------------------+------------------------- Reporter: kparal | Owner: jskladan Type: defect | Status: closed Priority: major | Milestone: Hot issues Component: tests | Resolution: fixed Keywords: | Blocked By: Blocking: | --------------------+------------------------- Changes (by jskladan):
* status: reopened => closed * resolution: => fixed
Comment:
Hot-fixed on both staging and production.
autoqa-devel@lists.fedorahosted.org