there is already a wiki page about hsm service standalone. http://wiki.ovirt.org/HSM_service_stand_alone
And now I'm trying to split hsm as a standalone rpm according to the wiki proposal.
as the wiki describe: V)The configuration parameters in vdsm.conf should be pull out to another hsm.conf file. And all VDSM process should not read hsm.conf directly. If it is necessary to get the HSM configurations, an API will be added into VDSM process for hsm configuration querying
I have split the the vdsm/config.py and submit a patch http://gerrit.ovirt.org/#/c/9345/ I moved 'irs'section from vdsm/config.py to a new vdsm/storage/config.py for hsm But some of vdsm python module need to get configure items form 'irs'section, such as vdsm/clientIF.py, vdsm/libvirtvm.py. I moved vdsm/storage/config.py to site-packages/hsm when install hsm. And I import all the items form 'irs'section, by "from hsm import config as hsmconfig" Is it a good way to split the config.py?
Also I have split the the constants.py and submit a patch http://gerrit.ovirt.org/#/c/9348/ And should I create a new "hsm" user for the DISKIMAGE_USER instead of "vdsm" user?
as the wiki describe: VII) All of the python modules shared by both VDSM service and HSM service should go into python site-packages directory which is like /usr/lib/python2.7/site-packages/vdsm Now, task.py doesn't sit in this directory, we should fix it in this proposal.
about the site-packages directory, there are three proposals. 1. VDSM service and HSM service share the same site-packages directory, still used site-packages/vdsm and that means there will be still a vdsm-python rpm, and both vdsm and hsm depends on them. 2. VDSM service and HSM service share the same site-packages directory, but it will renamed as used site-packages/hsm Now most codes of hsm and vdsm import modules from site-packages vdsm, so all of they should be fixed. 3. VDSM service depend on vdsm-python rpm, and HSM service depend on hsm-python rpm. I need to re-sort the public modules to hsm-python or vdsm-python rpm. such as fileUtils.py, misc.py, BetterPopen and qemuImg.py which is better of the above three?
Added Saggi, Ayal, Federico into this loop. And got two comments below.
Sheldon:
there is already a wiki page about hsm service standalone. http://wiki.ovirt.org/HSM_service_stand_alone
And now I'm trying to split hsm as a standalone rpm according to the wiki proposal.
as the wiki describe: V)The configuration parameters in vdsm.conf should be pull out to another hsm.conf file. And all VDSM process should not read hsm.conf directly. If it is necessary to get the HSM configurations, an API will be added into VDSM process for hsm configuration querying
I have split the the vdsm/config.py and submit a patch http://gerrit.ovirt.org/#/c/9345/ I moved 'irs'section from vdsm/config.py to a new vdsm/storage/config.py for hsm But some of vdsm python module need to get configure items form 'irs'section, such as vdsm/clientIF.py, vdsm/libvirtvm.py. I moved vdsm/storage/config.py to site-packages/hsm when install hsm. And I import all the items form 'irs'section, by "from hsm import config as hsmconfig" Is it a good way to split the config.py?
Also I have split the the constants.py and submit a patch http://gerrit.ovirt.org/#/c/9348/ And should I create a new "hsm" user for the DISKIMAGE_USER instead of "vdsm" user?
as the wiki describe: VII) All of the python modules shared by both VDSM service and HSM service should go into python site-packages directory which is like /usr/lib/python2.7/site-packages/vdsm Now, task.py doesn't sit in this directory, we should fix it in this proposal.
Task model is only for storage service now. We have plan to make task models to all VDSM service. So it is reasonable to move task.py into site-packages directory.
about the site-packages directory, there are three proposals.
- VDSM service and HSM service share the same site-packages
directory, still used site-packages/vdsm and that means there will be still a vdsm-python rpm, and both vdsm and hsm depends on them. 2. VDSM service and HSM service share the same site-packages directory, but it will renamed as used site-packages/hsm Now most codes of hsm and vdsm import modules from site-packages vdsm, so all of they should be fixed. 3. VDSM service depend on vdsm-python rpm, and HSM service depend on hsm-python rpm. I need to re-sort the public modules to hsm-python or vdsm-python rpm. such as fileUtils.py, misc.py, BetterPopen and qemuImg.py which is better of the above three?
Packaging HSM site package files into a another new package other than vdsm-python package doesn't prevent those files contained into a same targeting package. So at least we can do, two different package say, vdsm-core-python, vdsm-storage-python and the files in them are targeting the same directory /usr/lib/python2.7/site-packages/vdsm.
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
----- Original Message -----
From: "Sheldon" shaohef@linux.vnet.ibm.com To: vdsm-devel@lists.fedorahosted.org Cc: agl@linux.vnet.ibm.com, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Sunday, December 2, 2012 8:00:50 AM Subject: [vdsm] [VDSM][RFC] hsm service standalone
there is already a wiki page about hsm service standalone. http://wiki.ovirt.org/HSM_service_stand_alone
And now I'm trying to split hsm as a standalone rpm according to the wiki proposal.
as the wiki describe: V)The configuration parameters in vdsm.conf should be pull out to another hsm.conf file. And all VDSM process should not read hsm.conf directly. If it is necessary to get the HSM configurations, an API will be added into VDSM process for hsm configuration querying
I have split the the vdsm/config.py and submit a patch http://gerrit.ovirt.org/#/c/9345/ I moved 'irs'section from vdsm/config.py to a new vdsm/storage/config.py for hsm But some of vdsm python module need to get configure items form 'irs'section, such as vdsm/clientIF.py, vdsm/libvirtvm.py. I moved vdsm/storage/config.py to site-packages/hsm when install hsm. And I import all the items form 'irs'section, by "from hsm import config as hsmconfig" Is it a good way to split the config.py?
Also I have split the the constants.py and submit a patch http://gerrit.ovirt.org/#/c/9348/ And should I create a new "hsm" user for the DISKIMAGE_USER instead of "vdsm" user?
as the wiki describe: VII) All of the python modules shared by both VDSM service and HSM service should go into python site-packages directory which is like /usr/lib/python2.7/site-packages/vdsm Now, task.py doesn't sit in this directory, we should fix it in this proposal.
about the site-packages directory, there are three proposals.
- VDSM service and HSM service share the same site-packages
directory, still used site-packages/vdsm and that means there will be still a vdsm-python rpm, and both vdsm and hsm depends on them. 2. VDSM service and HSM service share the same site-packages directory, but it will renamed as used site-packages/hsm Now most codes of hsm and vdsm import modules from site-packages vdsm, so all of they should be fixed. 3. VDSM service depend on vdsm-python rpm, and HSM service depend on hsm-python rpm. I need to re-sort the public modules to hsm-python or vdsm-python rpm. such as fileUtils.py, misc.py, BetterPopen and qemuImg.py which is better of the above three?
-- Sheldon Feng(冯少合)shaohef@linux.vnet.ibm.com IBM Linux Technology Center
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with.
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "Sheldon" shaohef@linux.vnet.ibm.com, agl@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
On 12/03/2012 07:22 PM, Saggi Mizrahi wrote:
HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with.
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "Sheldon" shaohef@linux.vnet.ibm.com, agl@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
Contrary, It may help to split internal api and external (means rpc between vdsm to engine and rpc between vdsm and hsm), and it might be easier to understand the process flow and control the request from engine. For example, for migration you will receive vdsm request for migration, and that will initiate all the internal processes that we do during migration by sending requests to hsm service. The engine doesn't supposed to care about the internal rpc and those abilities does not supposed to be exposed via vdsm api if we don't allow those specific hsm operation from engine, but via HSM-python service that won't be available for the engine. Might be a good division. But lots of work indeed..
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
2012-12-5 23:55, ybronhei:
On 12/03/2012 07:22 PM, Saggi Mizrahi wrote:
HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with.
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "Sheldon" shaohef@linux.vnet.ibm.com, agl@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
Contrary, It may help to split internal api and external (means rpc between vdsm to engine and rpc between vdsm and hsm), and it might be easier to understand the process flow and control the request from engine. For example, for migration you will receive vdsm request for migration, and that will initiate all the internal processes that we do during migration by sending requests to hsm service. The engine doesn't supposed to care about the internal rpc and those abilities does not supposed to be exposed via vdsm api if we don't allow those specific hsm operation from engine, but via HSM-python service that won't be available for the engine. Might be a good division. But lots of work indeed..
Fortunately, the hsm directory in vdsm is almost self-contained. hsm files are depending on some files like config.py, constants.py &etc which are already in a vdsm site packages. After HSM-python package is created, both vdsm-4.1x and HSM-python-4.1.x package will depend on vdsm-python-4.1x package. vdsm-python-4.1x package might be a confusing name that time. It might be better named "vdsm-common-python-4.1x".
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Saggi,
It might be that the wiki page was a bit confusing. Let me articulate it again and help others to understand the intention of this project.
1) Now, HSM files here refers to the files under "vdsm/storage" which are storage service related code. And other VDSM files call into those functions in HSM files in local way(different from RPC). So in the production environment, HSM files are linked in the VDSM process and there is no additional HSM process standing in the ovirt node.
2) This project goal is to move all the HSM files into self-contained workspace and a new stand alone HSM process will be started from these HSM files. Also a new HSM package will be built from this new self-contained workspace. Also all of the HSM APIs are public to the original VDSM service by RPC calls. It is possible that the HSM standalone workspace and the original VDSM workspace may share some functions in misc.py,task.py&etc. We can move those common files to vdsm site packages as the other common files. After this goal is achieved, we have one more HSM process in production environment and one more release package for HSM files. For backward compatibility, vdsm will run in two possible modes. VDSM service will try to discover if HSM RPC APIs are available from the new HSM process. If the discovery is successful, VDSM service will make RPC calls to the HSM process. If the discovery fails, it will fall back the legacy way calling the HSM functions by local way.
3) In order to achieve this goal, we should remove the preconditions gradually and safely. The first step here is the patch
http://gerrit.ovirt.org/#/c/9345/
That try to decouple the HSM configuration parameters from the vdsm.conf and store them into a new different conf file.
Saggi Mizrahi:
HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with.
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "Sheldon" shaohef@linux.vnet.ibm.com, agl@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
split config file for HSM service standalone http://gerrit.ovirt.org/#/c/10363/2
I have move the 'irs' section from vdsm configure to hsm configure. There are some problems. IMHO, only hsm code should directly call 'irs'. such as vdsm/utils.py http://gerrit.ovirt.org/#/c/10363/2/vdsm/utils.py, should not call hsm configure.
also vdsm/libvirtvm.py http://gerrit.ovirt.org/#/c/10363/2/vdsm/libvirtvm.py, should not call hsm configure.
should hsm service standalone provide a new interface to access the 'irs' section?
On 12/04/2012 01:22 AM, Saggi Mizrahi wrote:
HSM is not a package it's an application. Currently it and the rest of VDSM share the same process but they use RPC to communicate. This is done so that one day we can actually have them run as different processes. HSM is not something you import, it's a daemon you communicate with.
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "Sheldon" shaohef@linux.vnet.ibm.com, agl@linux.vnet.ibm.com, vdsm-devel@lists.fedorahosted.org, "Zheng Sheng ZS Zhou" zhshzhou@cn.ibm.com Sent: Monday, December 3, 2012 12:01:28 PM Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone
On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:
There are a bunch of precondition to having HSM pulled out. On simple things is that someone needs to go through storage/misc.py and utils.py and move all the code out to logical packages. There also needs to be a bit of a rearrangement of the code files so they can import the reusable code properly.
I am also still very much against putting core VDSM in to site-packages.
Would you elaborate on your position? Do you mind the wrong implications this may give about API stability?
vdsm-devel@lists.fedorahosted.org