#400: Watchers get stuck sometimes and hang for days -----------------------+------------------------ Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: watchers | Keywords: Blocked By: | Blocking: -----------------------+------------------------ I have seen several times that koji-bodhi watcher got stuck on the production server and hanged for several days before I killed him. I was able to find out that it waited for a socket pointing at koji.fedoraproject.org:80.
There's clearly some problem with TCP timeouts. We have to make sure reasonable time-outs are set for our requests and if the server doesn't respond soon, we should re-try or terminate the script.
#400: Watchers get stuck sometimes and hang for days ----------------------+------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: watchers | Resolution: Keywords: | Blocked By: Blocking: | ----------------------+-------------------------
Comment (by mkrizek):
So far, I have found out that if I set network delay to 2 minutes ([http://stackoverflow.com/questions/614795/simulate-delayed-and-dropped- packets-on-linux how]?) while koji-bodhi watcher is running, the communication between the watcher and koji stops (I see no packets via wireshark). I would expect the communication to be delayed. If I set the delay back to the previous state (no delay), the communication does not continue and the watcher still hangs - this is maybe what happened on the production server, koji got busy and the watcher hung.
Note: if I do the same with the delay of 30 seconds, the communication continues after unsetting the delay but resulting in xml.parsers.expat.ExpatError: unclosed token, which seems like delayed tokens were lost.
I am not sure if I am on the right path but I will investigate further if this is a problem in koji, xmlrpc or something else.
#400: Watchers get stuck sometimes and hang for days ----------------------+------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: watchers | Resolution: Keywords: | Blocked By: Blocking: | ----------------------+-------------------------
Comment (by kparal):
Replying to [comment:1 mkrizek]:
So far, I have found out that if I set network delay to 2 minutes
([http://stackoverflow.com/questions/614795/simulate-delayed-and-dropped- packets-on-linux how]?) while koji-bodhi watcher is running, the communication between the watcher and koji stops (I see no packets via wireshark). I would expect the communication to be delayed. If I set the delay back to the previous state (no delay), the communication does not continue and the watcher still hangs - this is maybe what happened on the production server, koji got busy and the watcher hung.
Great, that seems to be the same problem. The hung connection was "estabilished", but obviously no data flowed.
It's possible that the Koji library doesn't expect the remote side to completely stop responding over the XMLRPC connection and just waits for eternity. We have to find out how to set some timeouts.
#400: Watchers get stuck sometimes and hang for days ----------------------+------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: watchers | Resolution: Keywords: | Blocked By: Blocking: | ----------------------+-------------------------
Comment (by mkrizek):
Patch: https://fedorahosted.org/reviewboard/r/249/
The patch can be also found in the t400 branch.
#400: Watchers get stuck sometimes and hang for days ----------------------+------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: watchers | Resolution: Keywords: | Blocked By: Blocking: | ----------------------+-------------------------
Comment (by kparal):
Reviewed.
#400: Watchers get stuck sometimes and hang for days ----------------------+--------------------- Reporter: kparal | Owner: Type: defect | Status: closed Priority: major | Milestone: 0.8.0 Component: watchers | Resolution: fixed Keywords: | Blocked By: Blocking: | ----------------------+--------------------- Changes (by mkrizek):
* status: new => closed * resolution: => fixed * milestone: Hot issues => 0.8.0
Comment:
Pushed as 618c650fec6fd8224af92736d8fdc6ebe1cbbb04
autoqa-devel@lists.fedorahosted.org