Re: [vdsm] flowID schema
by dyasny@redhat.com
> From: "Saggi Mizrahi" <smizrahi(a)redhat.com>
> To: "Keith Robertson" <kroberts(a)redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org>
> Sent: Thursday, February 9, 2012 2:24:44 PM
> Subject: Re: [vdsm] flowID schema
>
> -1
>
> I agree that for messaging environment having a Message ID is a must
> because you sometimes don't have a particular target so when you get
> a response you need to know what this node is actually responding
> to.
>
> The message ID could be composed with <FLOWID><MSGID> so you can
> reuse the field.
>
> But that is all besides the point.
>
> I understand that someone might find it fun to go on following the
> entire flow in the Engine and in VDSM. But I would like to hear an
> actual use case where someone would have actually benefited from
> this.
> As I see it having VSDM return the task ID with every response (and
> not just for async tasks) is a lot more useful and correct.
Actually, the only way to understand what happened in a certain flow is to follow it through. From the engine log where an action was initiated, down to the hosts that did the execution. Everything RHEV does is a flow, and with no correlation between hosts executing parts of the same flow, troubleshooting turns into guesswork, because the only contact point left is time, which is useless when you're talking about vdsm - there are sometimes hundreds of log records in a single second, and not every host is in absolute sync with every other.
>
> A generic debugging scenario as I see it.
>
> 1. Something went wrong
> 2. You go looking in the ENGINE log trying to figure out what
> happend.
> 3. You see that ENGINE got SomeError.
ok, the rest are all downhill.
4. You follow the failure back to the start of the flow, then go with the flow to the point where the engine exited to vdsm
5. switch over to vdsm logs, make sure you have the timing right (with no flow ID that's the olny orientation after all)
6. find the start of the vdsm-side flow, follow it to the failure, pray the error makes sense.
In many cases the answer is not in the vdsm failure traceback but somewhere in the middle of the flow, with no errors reported, this is why we need a way to easily follow things through. Moreover, the logs should be readable enough to make sense to a typical sysadmin, and not a RHEV expert.
> 4. Check to see if this error makes sense imagining that VDSM is
> always right and is a black box.
> 5. You did your digging and now you think that VDSM is as fault.
> 6. Go look for the call that failed. (If we returned the taskID it's
> pretty simple to find that call).
> 7. Look around the call to check VDSM state.
> 8. Profit.
>
> There is never a point where you want to follow a whole flow call by
> call going back and forth, and even if you did having the VDSM
> taskID is a better anchor then flowID.
not everything is a task, flow IDs would unify entire flows, and make following them easy.
>
> VDSM is built in a way that every call takes in to account the
> current state only. Debugging it with an engine flow mindset is just
> wrong and distracting. I see it doing more harm the good by
> reinforcing bad debugging practices.
Maybe you're right, though I can't see how from my experience so far, but following the flows is the only thing that got cases resolved. Not event IDs making every possible error, and not task IDs (though these do have their uses) - slow and meticulous mapping of flows to log records.
>
> ----- Original Message -----
> > From: "Keith Robertson" <kroberts(a)redhat.com>
> > To: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org>
> > Sent: Thursday, February 9, 2012 1:34:43 PM
> > Subject: Re: [vdsm] flowID schema
> >
> > On 02/09/2012 12:18 PM, Andrew Cathrow wrote:
> > >
> > > ----- Original Message -----
> > >> From: "Ayal Baron"<abaron(a)redhat.com>
> > >> To: "Dan Kenigsberg"<danken(a)redhat.com>
> > >> Cc: "VDSM Project
> > >> Development"<vdsm-devel(a)lists.fedorahosted.org>
> > >> Sent: Monday, February 6, 2012 10:35:54 AM
> > >> Subject: Re: [vdsm] flowID schema
> > >>
> > >>
> > >>
> > >> ----- Original Message -----
> > >>> On Thu, Feb 02, 2012 at 10:32:49AM -0500, Saggi Mizrahi wrote:
> > >>>> flowID makes no sense after the initial API call as stuff like
> > >>>> cacheing\threadpools\samplingtasks\resources\asyncTasks so
> > >>>> flowing
> > >>>> a flow like that will not give you the entire picture while
> > >>>> debugging.
> > >>>>
> > >>>> Also adding it now will make everything even more ugly.
> > >>>> You know what, just imagine I wrote one of my long rambles
> > >>>> about
> > >>>> why I don't agree with doing this.
> > >>> I cannot imagine you write anything like that. Really. I do not
> > >>> understand why you object logging flowID on API entry point.
> > >> The question is, what problem is this really trying to solve and
> > >> is
> > >> there a simpler and less obtrusive solution to that problem?
> > > correlating logs between ovirt engine and potentially multiple
> > > vdsm
> > > nodes is a nightmare. It requires a lot skill to follow a
> > > transaction through from the front end all the way to the node,
> > > and even multiple nodes (eg actions on spm, then actions on other
> > > node to run a vm).
> > > Having a way to correlate the logs and follow a single event/flow
> > > is vital.
> > >
> > +1
> >
> > Knowing what command caused a sequence of events in VDSM would be
> > really
> > helpful particularly in a threaded environment. Further, wouldn't
> > such
> > an ID be helpful in an asynchronous request/response model? I'm
> > not
> > sure what the plans are for AMQP or even if there are plans, but
> > I'd
> > think that something like this would be crucial for an async
> > response.
> > So, if you implemented it you might be killing 2 birds with 1
> > stone.
> >
> > FYI: If you want to see examples of other systems that use similar
> > concepts, take a look at the correlation ID in JMS.
> >
> > Cheers,
> > Keith
> >
> >
> > >>> _______________________________________________
> > >>> vdsm-devel mailing list
> > >>> vdsm-devel(a)lists.fedorahosted.org
> > >>> https://fedorahosted.org/mailman/listinfo/vdsm-devel
> > >>>
> > >> _______________________________________________
> > >> vdsm-devel mailing list
> > >> vdsm-devel(a)lists.fedorahosted.org
> > >> https://fedorahosted.org/mailman/listinfo/vdsm-devel
> > >>
> > > _______________________________________________
> > > vdsm-devel mailing list
> > > vdsm-devel(a)lists.fedorahosted.org
> > > https://fedorahosted.org/mailman/listinfo/vdsm-devel
> >
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel(a)lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/vdsm-devel
> >
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/vdsm-devel
>
--
Regards,
Dan Yasny
Red Hat Israel
+972 9769 2280
12 years, 4 months
VDSM host network configuration
by lilei@linux.vnet.ibm.com
Hi,
We are working on VDSM network REST APIs, to support the functions we need to get
the list of configured networks. I found that VDSM network has a function
'listNetworks' in configNetwork.py. It can get and display the current configured
network like this:
# python configNetwork.py list
Networks: ['bridge_one', 'bridge_three', 'bridge_two']
Vlans: []
Nics: ['eth0']
Bondings: []
But there are some problems with it. It can not display the defined networks after
host restart, but the created config file are still there(/etc/sysconfig/network-scripts/..).
Did I miss anything? Or Is there some way to avoid this?
Your suggestion and thoughts would be appreciated.
Lei
12 years, 4 months
vdsm hangs in SamplingMethod after reinstall
by agl@us.ibm.com
Hi. I am running into a very annoying problem when working on vdsm lately. My
development process involves stopping vdsm, replacing files, and restarting it.
I do this pretty frequently. Sometimes, after restarting vdsm the XMLRPC call
getStorageDomainsList() hangs. The following line is the last to print in the
log:
Thread-18::DEBUG::2012-02-09 17:11:46,793::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
The only solution I've been able to come up with is restarting my machine. When
stopping vdsm I search for any stale threads but I am unable to find them. Do
you know what else might be causing DynamicBarrier.enter() to hang for a long
period of time? Do the threading primitives use some sort of temporary disk
storage that needs to be cleaned up? Thanks for the help!
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
12 years, 4 months
flowID schema
by Douglas Schilling Landgraf
Hello,
flowID is schema that we will be including in vdsm API to oVirt
Engine people share the ID of engine transaction to vdsm.
With this in hands, we will add the ID of transactions to our log.
I would like to know your opinion how we could do that without break our
API, like include new parameters to our calls.
Should we add at BindingXMLRPC.py -> wrapper() a code to search for a
'flowID' key into functions which use dict as parameter (like create)?
[1] Maybe change at other level inside BindingXMLRPC ?
Ideas/Thoughts?
[1] http://gerrit.ovirt.org/#patch,sidebyside,1221,3,vdsm/BindingXMLRPC.py
Thanks!
--
Cheers
Douglas
12 years, 4 months
About the wiki page installing_VDSM_from_rpm
by shuming@linux.vnet.ibm.com
Hi,
I was installing the VDSM packaging following this link:
http://www.ovirt.org/wiki/Installing_VDSM_from_rpm
It seems that the network configuration section is not correct. In
this wiki page, the static IP is assigned to the host network interface
like eth0, em1 and the outing interface of the bridge is attached to
the same host interface. However, it didn't work in my server. After
moving the static IP into the bridge script *"ifcfg-ovirtmgmt*", the
network configuration is OK now. I was wondering if it was someone's
typo to write this wiki. See the change in below:
Add the following content into a new file named:
*/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt*:
DEVICE=ovirtmgmt
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=dhcp
*---> Should be changed to:*
DEVICE=ovirtmgmt
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=static
IPADDR=XXX.XXX.XXX.XXX
NETMASK=255.255.255.0
GATEWAY=XXX.XXX.XXX.XXX
Add the following line into the configuration file of your out going
interface (usually em1/eth0) the file is located at:
*/etc/sysconfig/network-scripts/ifcfg-em1* (assuming the device is em1)
BRIDGE=ovirtmgmt
Full Example
DEVICE=em1
BOOTPROTO=static
IPADDR=192.168.1.212
NETMASK=255.255.255.0
ONBOOT=yes
BRIDGE=ovirtmgmt
---> should be changed to
DEVICE=em1
BOOTPROTO=static
ONBOOT=yes
BRIDGE=ovirtmgmt
--
Shu Ming<shuming(a)linux.vnet.ibm.com>
IBM China Systems and Technology Laboratory
12 years, 4 months
vdsm test suites
by lvroyce@linux.vnet.ibm.com
Hi folks,
I'm writing test suites about VDSM REST API, and glad to help with
tests of VDSM. I'm wandering do you have schedule to add tests to it
?And how
would you like the test code to be hosted?
Thanks a lot!
12 years, 4 months
metadata
by wangxiaofan
What is the metadata of non-master data domain used for? And master data domain?
12 years, 4 months