Hi Everyone, I'm looking for a solution to run some Xen VM's with fairly significant(up to ~30TB) local storage volumes. We plan to use two Sun "Thumper" x4540's with 48*1TB drives (8 controllers).
I have a few questions, and would appreciate feedback before I try to achieve something dumb or impossible :)
- Has Xen been tested / does Xen support very large volumes? We will probably end up using ext3 (as it seems its the only supported option), so no single volume in a VM will exceed ~16TB, but we will be at this end of the spectrum on several volumes. - Can anyone suggest a better option than creating 7 RAID5 arrays of 6 disks each (1 per controller); using the remaining 6 drives for RAID-10 for DomU and hot spares and then adding the 7 RAID5 volumes to a LVM group, creating a LV per Xen VM block device? Has anybody used mdadm and/or LVM to handle this number of drives? - We are required to be able to bring up VMs on one machine on another within a reasonable period of time (this is defined as days, not hours) and to keep the backup copy reasonably in sync (again, days not hours). Can anyone comment on the suitability of rsync to copy such massive block devices? We would use version 3, which I know has improved performance for this sort of move, but is it going to literally take forever to checksum 48TB? If anyone is able to give example of rsync performance (same switch or crossover - whichever we conclude is fastest) over gigabit Ethernet, that would be a bonus. We expect there to be significant spare resources available to domU at night for checksumming if required. - I know XFS is not supported by RHEL, but does it work - and/or well? I've seen several reports that suggest it performs far better (best summary http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.sto...) but what is it like for reliability? - Does anyone has any suggestions / improvements to this plan? Has anybody done this before - with similar hardware? - [An alternative scenario, if we can get it, would be a third Thumper - in which case we would run half the VMs on each of two Thumpers running RAID-10 (8 RAID-1 volumes, each a PV) for ~24TB per machine, and use the third Thumper, partitioned as above with RAID-5 for more like RAID-5 levels of storage efficiency, for backups of VMs and disaster recovery. Comments on this would be fantastic too.]
The VMs are expected to be quite IO-heavy, hence the requirement for local storage (not iSCSI) and para virtualization.
Any comments much appreciated.
Many thanks,
Alex
On Sep 5, 2008, at 13:37, Alex Davies wrote:
using the remaining 6 drives for RAID-10 for DomU
Unless you meant "for Dom0", then beware that software RAID-10, LVM and Xen doesn't work:
https://bugzilla.redhat.com/show_bug.cgi?id=223947 https://bugzilla.redhat.com/show_bug.cgi?id=224077
Thank-you everyone for your suggestions; some extremely useful points have been raised for which I am very grateful.
To answer Erin's questions: we are running RHEL/Xen because we have a business requirement for a decent number of RHEL virtual machines (~30). We could run ESX (or KVM, or...) but we (I) am also keen to try to use Xen for real - i've used ESX a lot. There are also licensing benefits for my company that make Xen on RHEL > ESX / KVM.
I would love to use ZFS, but unfortunately my Solaris foo is ~0, and consequently we shall have to stick to Linux/ext3. RedHat assure me that if I keep each filesystem to <10T this is supported and encouraged.
Based on feedback from this post (and elsewhere), we have decided to go for 2 ISCSI boxes and 2 Thumpers. Two of our virtual machines require >30T of storage, and no redundancy, so we will run one VM on each Thumper (with local storage). The other ~30 VMs will run using the ISCSI volumes to allow live migration.
I'm sure i'll be posting here when it comes to getting this lot working, but if anyone has any advice for this setup please let me know!
Many thanks,
Alex
On Thu, Sep 11, 2008 at 6:55 AM, Ask Bjørn Hansen ask@develooper.com wrote:
On Sep 5, 2008, at 13:37, Alex Davies wrote:
using the remaining 6 drives for RAID-10 for DomU
Unless you meant "for Dom0", then beware that software RAID-10, LVM and Xen doesn't work:
https://bugzilla.redhat.com/show_bug.cgi?id=223947 https://bugzilla.redhat.com/show_bug.cgi?id=224077
On Sep 14, 2008, at 9:16, Alex Davies wrote:
Two of our virtual machines require >30T of storage, and no redundancy, so we will run one VM on each Thumper (with local storage).
Are you replacing the 500GB disks with 1TB disks in the Thumbers?
Unless "no redundancy" really truly means "it's fine if it breaks a lot", then please keep in mind the numbers. 40+ disks are that many times more likely to fail than one disk. Don't just stripe all the disks into one big volume. Either keep them as separate devices so they fail individually or use RAID5 or (better) RAID6 and keep plenty spares ready. Your plan of doing RAID in sets of 8 or so disks is probably about right.
- ask