Hey Gris,
Thanks for the reply. I'll insert my replies inline w/ a [JH] tag.
Tldr; on this is that I want to find a way to get some semblance of all the functionality
I've put together into the hpsa plugin. I'm very open to suggestions once I get
the code reviews up, none of it is perfect but it has at least been proven useful within
one solution already, and I have line-of-sight to how I'd integrate it into Ceph.
Joe
-----Original Message-----
From: Gris Ge [mailto:fge@redhat.com]
Sent: Thursday, October 22, 2015 8:48 PM
To: Handzik, Joe
Cc: libstoragemgmt-devel(a)lists.fedorahosted.org
Subject: Re: Supporting JBOD/HBA disks in libstoragemgmt.
On Thu, Oct 22, 2015 at 03:23:05PM +0000, Handzik, Joe wrote:
Resending a second time, the mailing list didn't like that I
hadn't
signed up for it yet, and the links are wrong on the lsm wiki...
Option B would be really really rough. I have patches here:
https://github.com/HP-Scale-out-Storage/libstoragemgmt/tree/wip-all
They implement a bunch of really useful (to me and solutions I work
with, at least) features for physical disks, such as:
-support for exposure of scsi device node (some code already existed
for this, but I now expose the node via a method call)
As libstoragemgmt also
support SAN arrays, I would like suggest user to use VPD83 NAA ID like this in Linux:
RAID mode:
/dev/disk/by-id/wwn-0x<lsm.Volume.vpd83>
HBA mode:
/dev/disk/by-id/wwn-0x<lsm.Disk.id>
# Not implemented yet, but could change HBA/JBOD disk
# to use VPD83 NAA ID as lsm.Disk.id.
# MegaRAID provides so. The hpssacli has no such info not
# yet, but it contains /dev/sdX where I could get VPD83
# NAA ID from.
And meanwhile, a dual domains SAS disk could be two sdX in OS, user should not use
/dev/sdX returned by this method when multipath is enabled but search mpath using VPD83
NAA ID.
We just provide a ID where user could match it with OS stuff, that would be enough for
libstoragemgmt.
[JH] - I'm not opposed to pulling in the VPD83 data, I actually think that's a
good idea. I also understand your concern about dual domain disks...unfortunately, lsm is
equally susceptible to that issue today, correct? For our purposes though, we're just
reporting device topology, I feel it should be left up to the user to understand that
they're in that sort of multipathing config. I think lsm being able to expose that
data would actually be very valuable, my goal for lsm would be to avoid using other tools
(take a look at my patches...I'm using both lsscsi and sg_ses within the hpsa plugin
in my branch, the goal being to accomplish the sort of matching that I think you're
describing).
-support for exposure of scsi generic node
When user needs
'SCSI pass-through' they are could use /dev/bsg/<sdX_scsi_id>
The sysfs is /sys/block/sdb/device/bsg, so there should be a udev way to do so.
[JH] - I don't fully understand the harm in what I'm exposing though. As long as
the output is well understood, it's not less troublesome than most linux tools to
date. With my implementation though, the user can avoid invoking those other tools
separately. That's the win for my purposes, along with C and python bindings to query
the data.
-support for discovery of a SEP associated with a disk, and caching
of
the SEP's scsi generic node (leveraged for features below)
I believe a SES
plugin could do so.
[JH] - Yes, it certainly can...again, not sure why we couldn't put it in the hpsa
plugin too.
-support for disk's sas address
The lsm.Disk is for a
physical disk, so we should allows each disk has two SAS addresses, or be more precisely,
two port IDs, since disk might be SATA/FC/NVMe.
[JH] - With Smart Array today, both ports will not be exposed. I'm willing to go back
to the drawing board on this one a bit if necessary.
-support for disk's port/bay/box information (some code already
existed for this, but I'm querying via a mechanism that doesn't
involve hpssacli)
I check what I can do via SES.
[JH] - I know sg_ses can pull some of this, but we have ready-made ways to get this data
out of Smart Array tools and drivers today. I'd like to use those if possible, if for
no other reason than to provide a quick turnaround.
-support to enable/disable physical disk IDENT LEDs via sg_ses
Can you renamed to follow <noun>+<verb> term:
disk_ident_led_set()
disk_ident_led_clear()
[JH] - Definitely can! I'll try to remember to do that before submitting a patch.
-support to enable/disable physical disk FAULT LEDs via sg_ses
Any user case we need fault LED?
[JH] - Sure, lots. For an integrated solution, if you want to communicate to the user that
a drive should be replaced (from your software stack's perspective), you could enable
the FAULT LED through python and c APIs instead of cobbling together a bunch of bash
scripting.
There are a few RAID mode features in there too:
-support for exposure of volume's scsi device node -support for
exposure of volume's scsi generic node
Check above. It's not
libstoragemgmt's job to handle OS stuff.
[JH] - I still don't understand this. This data is valuable for multiple solutions
that I work with. I'm just implementing a passthrough mechanism and exposing what
other tools already collect...just removing the collection responsibility from other
portions of the application. I'm open to reworking this stuff, but I don't want to
throw this functionality away, I'll 100% use it in Ceph, and it's being used in
early Lustre software today.
-support to enable/disable volume's IDENT LEDs via hpssacli
Just wanted to get that out there to make sure we don't duplicate
work. If you haven't started much implementation work yet, I'd
I am working
the open source SES plugin, should be ready in 2 weeks.
https://github.com/cathay4t/libstoragemgmt/tree/ses
So no overlap yet.
[JH] - I saw that, looks good! I'll be interested to take it for a spin when it's
done. Given that it's a native C implementation, would there be python bindings for
it? If not, that'd be a dealbreaker for use in a couple of solutions that I can think
of.
prefer that you take a look at my patches as they're submitted
and we
can rework the implementation if necessary. The code I have here is
already in use by a teammate of mine for a Lustre solution, and I will
be using this code within Ceph in the near future (that work hasn't
started yet, but is my next task after upstreaming everything you see
in wip-all).
Thanks. I will try to merge them and open PR for reviews.
[JH] - I have lots of cleanup to do, the code sprawled quite a big on me and it's over
2k lines...gotta chop it down and clean it up. I'm optimistic that we can work in what
I need for some higher-level solution work!
Best regards.
--
Gris Ge