Hi,
I was asked this afternoon to look into the bug where some instances
(particularly on vSphere, and possibly on other providers) reported a
MAC address instead of an IP. There are a few things going on and I
wanted to update everybody on the status. We're going to need to wrap
this up tomorrow.
The problem is that before the instance has obtained an IP, all that's
available is its MAC address. This MAC gets reported over deltacloud as
the IP address, so EventLog has this as the "IP" and dbomatic picks it
up and dutifully stores it in the database. But there is no Condor event
triggered on the change of an IP; it's not considered a state change.
Ergo, we never notice that the MAC has been replaced by an IP.
Worse, after spending a bit trying to just read the data out of Condor,
we discovered that Condor is never updated with the IP at all in these
cases. Even long after deltacloud starts reporting an IP and Condor
polls for status, the MAC stores as an IP in Condor is never updated.
It looks like deltacloud will be updated tomorrow to split out the
'address' tag to have type="mac" and "type="ipv4", but
this won't fully
solve our problem -- we'll be able to tell the difference between MACs
and IPs now, but we'll still have the problem of not getting an IP in
Conductor.
Short-term, the fix we identified is to go off to deltacloud any time we
display IP and obtain the information there. It should work to define
public_addresses and private_addresses in instance.rb, and insert calls
to Deltacloud there. (As a slight optimization, we could write the
values to the database here, and use those if the values were present,
so we don't have to go out to the network each time.) This isn't ideal
in the long-term, since it's fairly slow, but it's the best we're able
to do on short notice.
Unfortunately I've got to head out for the night now, so I can't take
care of this. I think the task will be slightly complicated by trying to
find the right instance on the API-side, since vSphere instances (that
will be reported over the API) have crazy UUID-derived names. I have a
sneaking suspicion you'll also have to go out to Image Warehouse to get
the image or build UUID, and then use that to find the matching instance
in deltacloud. From there, though, it should be a simple matter of
grabbing the IP, possibly updating it in the database, and then
returning it. It'll just take queries to a couple of external services.
Is there any way someone not in UTC-5 could pick this up in the morning,
to try to get a fix out? I'm told it's the last bug blocking us from
releasing. And I'm afraid I've got to head out for the night.
-- Matt