On 06/24/11 - 12:53:30PM, Steven Dake wrote:
On 06/24/2011 05:53 AM, Chris Lalancette wrote:
> On 06/24/11 - 08:15:41AM, Chris Lalancette wrote:
>>> Waiting for F15-x86_64-jeos to finish installing, 49630/50000
>>> Waiting for F15-x86_64-jeos to finish installing, 49620/50000
>>> ^CLibvirt Domain Info Failed:
>>> code is 38
>>> domain is 13
>>> message is poll on socket failed: Interrupted system call
>>> level is 2
>>> str1 is %s
>>> str2 is poll on socket failed: Interrupted system call
>
> So this error message in libvirt comes from this bit of code:
>
> src/remote/remote_driver.c:remoteIOEventLoop()
>
> repoll:
> ret = poll(fds, ARRAY_CARDINALITY(fds), timeout);
> if (ret < 0 && errno == EAGAIN)
> goto repoll;
>
> ...
>
> if (ret < 0) {
> if (errno == EWOULDBLOCK)
> continue;
> virReportSystemError(errno,
> "%s", _("poll on socket
failed"));
> goto error;
> }
>
> What is going on here is that the libvirt client library is polling on the
> socket(s) that it is using to connect to the libvirtd server. For some reason
> that poll call sometimes returns EINTR, leading to the rest of the badness.
>
>>From a quick glance at the kernel, it seems like the only way poll can return
> EINTR is when there is another signal already pending either when we enter
> poll or after the poll has begun.
>
> I stopped there. It is pretty clear for now that this is a bug in libvirt,
> not handling signals properly. I would suggest opening a bug against libvirt
> for now, and we can continue there.
>
It was interrupted because I pressed CTRL-c which sends a signal to the
local process and interrupts the poll call. I pressed ctrl-c after
20-30 minutes.
Ah, I see. So really your problem is that it was hung up, not that it got
the EINTR.
Don't have any idea of which type of bug to open since it seems to me
that the VM shouldn't have exited normally after install and at the same
time this libvirt call be blocking in poll. Is it typical that the
virtual machine would be destroyed whlie this call trace would be
executing?
Actually, yes. This stuff happens asynchronous to libvirt, so the qemu process
goes away of it's own volition, and then libvirt finds out about it later
(either by getting an EOF from the monitor socket, or by looking at the process
table, I don't exactly remember which). So the fact that the VM went away
is entirely normal, and I'd say the bug is that libvirt hung up.
Probably a bug in libvirt, but also a bug in oz (I'd except
oz would not continue to poll on virtual machine info for virtual
machines that are gone.
Well, except that oz actually uses the fact that a libvirt call to the domain
object fails as indication that the VM has gone away. That is, once a second
oz calls:
dom.blockStats()
which succeeds while the domain is running. Once the domain is no longer
running, that call throws an exception, and oz uses that as indication the
domain is no longer there. However, oz is strict in exactly which exception
it uses as "the VM has gone away"; it only accepts VIR_ERR_NO_DOMAIN,
VIR_ERR_SYSTEM_ERROR, and VIR_ERR_OPERATION_FAILED as valid codes, and for
anything else (like EINTR) it throws a real exception.
I guess what we could do is try to do a lookup on the domain on every
iteration, and fail if we no longer see the domain in the running table. I'm
not sure if that will be any more robust against this problem, though; I'm
still thinking this is a bug in the poll stuff in libvirt.
--
Chris Lalancette