- Tue Jan 17 2012 Dave Jones davej@redhat.com
- Rawhide builds now use MAXSMP on x86.
- For release builds, set x86-64 to support 64 CPUs. If larger systems become widespread, we can increase in an update.
_today_
amd: 4sockets * 16cores = 64 intel: 4sockets * 10cores * 2threads = 80
On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
- Tue Jan 17 2012 Dave Jones davej@redhat.com
- Rawhide builds now use MAXSMP on x86.
- For release builds, set x86-64 to support 64 CPUs.
If larger systems become widespread, we can increase in an update.
_today_
amd: 4sockets * 16cores = 64
Awesome. Got that covered still.
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
I'm not opposed to bumping it up to 128 or something, but I'm curious how many people are actually going to see benefits.
josh
On Sun, May 20, 2012 at 4:07 AM, Josh Boyer jwboyer@redhat.com wrote:
On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
- Tue Jan 17 2012 Dave Jones davej@redhat.com
- Rawhide builds now use MAXSMP on x86.
- For release builds, set x86-64 to support 64 CPUs.
If larger systems become widespread, we can increase in an update.
_today_
amd: 4sockets * 16cores = 64
Awesome. Got that covered still.
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
I'm not opposed to bumping it up to 128 or something, but I'm curious how many people are actually going to see benefits.
josh
At least in my case I did run Fedora 12-16 on 4S and 8S machines to test software scalability on (extreme) high-end hardware. Though, IMHO anyone that's crazy enough to run Fedora on a high-end 4S/8S machine is more than capable of rebuilding the kernel with CONFIG_NR_CPUS 256...
However, given the fact that x86_64 machines tend to be far less memory constrained than i686 machines, I doubt that raising the limit to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?)
- Gilboa
On Sun, May 20, 2012 at 2:32 PM, Gilboa Davara gilboad@gmail.com wrote:
On Sun, May 20, 2012 at 4:07 AM, Josh Boyer jwboyer@redhat.com wrote:
On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
- Tue Jan 17 2012 Dave Jones davej@redhat.com
- Rawhide builds now use MAXSMP on x86.
- For release builds, set x86-64 to support 64 CPUs.
If larger systems become widespread, we can increase in an update.
_today_
amd: 4sockets * 16cores = 64
Awesome. Got that covered still.
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
I'm not opposed to bumping it up to 128 or something, but I'm curious how many people are actually going to see benefits.
josh
At least in my case I did run Fedora 12-16 on 4S and 8S machines to test software scalability on (extreme) high-end hardware. Though, IMHO anyone that's crazy enough to run Fedora on a high-end 4S/8S machine is more than capable of rebuilding the kernel with CONFIG_NR_CPUS 256...
However, given the fact that x86_64 machines tend to be far less memory constrained than i686 machines, I doubt that raising the limit to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?)
It's even higher these days:
grep NR_CPUS /boot/config-2.6.32-220.13.1.el6.x86_64 CONFIG_NR_CPUS=4096
I'm curious, has anyone measured what the memory overhead is of keeping NR_CPUS at 512? arch/x86/Kconfig says "This is purely to save memory - each supported CPU adds approximately eight kilobytes to the kernel image." If that's true, 512 cpus use 4MB, something I'm willing to live with on my 64bit servers.
Thanks,
Ruben
On 05/20/2012 11:23 AM, Ruben Kerkhof wrote:
On Sun, May 20, 2012 at 2:32 PM, Gilboa Davara gilboad@gmail.com wrote:
On Sun, May 20, 2012 at 4:07 AM, Josh Boyer jwboyer@redhat.com wrote:
On Sat, May 19, 2012 at 02:54:33PM +0200, Xose Vazquez Perez wrote:
- Tue Jan 17 2012 Dave Jones davej@redhat.com
- Rawhide builds now use MAXSMP on x86.
- For release builds, set x86-64 to support 64 CPUs.
If larger systems become widespread, we can increase in an update.
_today_
amd: 4sockets * 16cores = 64
Awesome. Got that covered still.
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
I'm not opposed to bumping it up to 128 or something, but I'm curious how many people are actually going to see benefits.
josh
At least in my case I did run Fedora 12-16 on 4S and 8S machines to test software scalability on (extreme) high-end hardware. Though, IMHO anyone that's crazy enough to run Fedora on a high-end 4S/8S machine is more than capable of rebuilding the kernel with CONFIG_NR_CPUS 256...
However, given the fact that x86_64 machines tend to be far less memory constrained than i686 machines, I doubt that raising the limit to 128 will cause too many issues. (Isn't NR_CPUS == 512 in el6?)
It's even higher these days:
grep NR_CPUS /boot/config-2.6.32-220.13.1.el6.x86_64 CONFIG_NR_CPUS=4096
This is from RHEL which has a markedly different user base than Fedora does. I remember setting this in RHEL based on a partner request who ships LARGE systems (4096 CPUs, many many many TB of memory).
This is not the number you want for Fedora.
I'm curious, has anyone measured what the memory overhead is of keeping NR_CPUS at 512?
I think we did that in RHEL. It is insignificant IIRC. I'll try and dig out some numbers...
P.
On 05/20/2012 03:07 AM, Josh Boyer wrote:
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
- Xeon E7: http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_Desk...
http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422... http://www.dell.com/us/business/p/poweredge-r910/pd
- Opteron 6200: http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_620...
http://www-03.ibm.com/systems/x/hardware/rack/x3755m3/index.html http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-332... http://www.dell.com/us/business/p/poweredge-r815/pd http://www.supermicro.nl/Aplus/motherboard/Opteron6000/SR56x0/H8QG6-F.cfm
[.....]
secondary archs: z196: 24sockets * 4cores = 96, but only 80IFL. power7: 32sockets * 8cores * 4 threads = 1024, 795 and 775 servers. sparc: we will see, when oracle release Oracle Enterprise Linux for SPARC.
On 05/20/2012 06:20 PM, Xose Vazquez Perez wrote:
On 05/20/2012 03:07 AM, Josh Boyer wrote:
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
- Xeon E7:
http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_Desk...
http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422...
http://www.dell.com/us/business/p/poweredge-r910/pd
- Opteron 6200:
http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_620...
http://www-03.ibm.com/systems/x/hardware/rack/x3755m3/index.html http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-332...
http://www.dell.com/us/business/p/poweredge-r815/pd http://www.supermicro.nl/Aplus/motherboard/Opteron6000/SR56x0/H8QG6-F.cfm
[.....]
Xose,
We should start supporting these in Fedora. These are not unusual to find in systems.
Josh, the reality is that 64 isn't "enough" anymore. Even deskside systems are starting to ship with 32 cores, so increasing the number of CPUs to 128 is a good idea. I've had a lot of experience with larger systems. The increase in memory footprint is minimal (a few Kb, not Mb).
I do know that some vendors who do upstream development, do install Fedora on systems with greater than 64 logical cpus. It would be good if we could support them fully.
Jumping to 128 is a very good idea. We know it is stable given RHEL's support of 4096 (again, I don't think Fedora should entertain jumping to 4096 for a variety of reasons).
In short, ACK ;)
P.
On Mon, May 21, 2012 at 08:42:12AM -0400, Prarit Bhargava wrote:
Josh, the reality is that 64 isn't "enough" anymore. Even deskside systems are starting to ship with 32 cores, so increasing the number of CPUs to 128 is a good idea. I've had a lot of experience with larger systems. The increase in memory footprint is minimal (a few Kb, not Mb).
I'll get this into rawhide shortly unless someone else brings up counter-points (doubtful).
Jumping to 128 is a very good idea. We know it is stable given RHEL's support of 4096 (again, I don't think Fedora should entertain jumping to 4096 for a variety of reasons).
I'm not concerned with the stability aspect of it. Before the recent-ish change, we had it set to 256 on released kernels and 512 for rawhide/debug kernels. We know it works. For rawhide/debug kernels we already set it to MAXSMP.
For the record, I'm not going to change 32-bit kernels. I don't see much of a need to increase that beyond what it is today and the same arguments about more cores being added don't really apply to 32-bit chips.
(Before someone points out that you can run 32-bit kernels on 64-bit hardware, I just wanted to say I don't care. If you have a CPU that has 128 cores and want to use them all, use a 64-bit kernel ;) .)
josh
For the record, I'm not going to change 32-bit kernels. I don't see much of a need to increase that beyond what it is today and the same arguments about more cores being added don't really apply to 32-bit chips.
I'd like to see 32-bit go away :) Please? :) :) :)
Seriously though, I think giving people yet another excuse to migrate away from 32-bit is a good idea and I cannot see any merit in increasing the 32-bit NR_CPUS.
P.
On 05/21/2012 12:20 AM, Xose Vazquez Perez wrote:
On 05/20/2012 03:07 AM, Josh Boyer wrote:
intel: 4sockets * 10cores * 2threads = 80
Which particular CPU/Motherboard combo is that, and how often do we see it in Fedora?
- Xeon E7:
http://en.wikipedia.org/wiki/Nehalem_%28microarchitecture%29#Server_.2F_Desk...
http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422... http://www.dell.com/us/business/p/poweredge-r910/pd
bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160
Sun Fire X4800 M2 Server: http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4800...
Super Micro X8OBN-F: http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm
2x HP ProLiant DL980 G7 Server (HP PREMA): http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422...
2x IBM System x3950 X5 (2-node): http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
SGI servers need MAXSMP.
On 05/24/2012 01:21 AM, Xose Vazquez Perez wrote:
bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160
Sun Fire X4800 M2 Server: http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4800...
Super Micro X8OBN-F: http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm
2x HP ProLiant DL980 G7 Server (HP PREMA): http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422...
2x IBM System x3950 X5 (2-node): http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
SGI servers need MAXSMP.
Also http://www.fujitsu.com/fts/products/computing/servers/primergy/rack/rx900/
... and more are coming.
Good bye Itanium, SPARC and POWER.
On Thu, May 24, 2012 at 01:35:59AM +0200, Xose Vazquez Perez wrote:
On 05/24/2012 01:21 AM, Xose Vazquez Perez wrote:
bigger *Xeon* server, 8_sockets * 10_cores * 2_threads = 160
Sun Fire X4800 M2 Server: http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4800...
Super Micro X8OBN-F: http://www.supermicro.com/products/motherboard/Xeon7000/7500/x8obn-f.cfm
2x HP ProLiant DL980 G7 Server (HP PREMA): http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-3328412-241644-422...
2x IBM System x3950 X5 (2-node): http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/index.html
SGI servers need MAXSMP.
Also http://www.fujitsu.com/fts/products/computing/servers/primergy/rack/rx900/
... and more are coming.
We made the change 2 days ago. You're belaboring the point just a bit now.
Also, the cheapest of these is $13k. That's roughly 10x the cost of a mid-high end home desktop and well outside Fedora's normal target user. If someone wants to spend that kind of money on an enterprise server and run Fedora on it, I'll gladly thank them for being a user but most of those kinds of purchases are done by businesses that want something a bit more stable and supported.
Good bye Itanium, SPARC and POWER.
Hello ARM?
josh
kernel@lists.fedoraproject.org