Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
[1]: https://en.opensuse.org/Feature_Planning_15.4 [2]: https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-li... [3]: https://ml.mageia.org/l/arc/dev/2021-02/msg00583.html [4]: https://www.phoronix.com/scan.php?page=news_item&px=Arch-Linux-x86-64-v3...
On Tue, 15 Jun 2021 at 17:35, Neal Gompa ngompa13@gmail.com wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Wasn't the last time this was looked at was https://fedoraproject.org/wiki/Changes/x86-64_micro-architecture_update? That spawned the multi hundred thread "Fedora 32 System-Wide Change proposal: x86-64 micro-architecture update" but I think this was to push towards level 3. However my review of that thread and some others seemed to show there was no stomach to move Fedora up without many people dropping packages etc.
I used this https://unix.stackexchange.com/questions/631217/how-do-i-check-if-my-cpu-sup... to see what cpu instructions are at each level
```
#!/usr/bin/awk -f
BEGIN { while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1 if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1 if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2 if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3 if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4 if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 } exit 1 }
```
level 2 is avx and avx2 plus some others.
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Stephen John Smoogen wrote:
#!/usr/bin/awk -f
BEGIN { while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1 if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1 if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2 if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3 if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4 if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 } exit 1 }
That script says "x86-64-v1" on my previous workstation, which is now a server. But I'm already planning to reinstall that one with Debian, to get less upgrade churn while keeping Ada and ZFS, so it won't hurt me if Fedora stops working there.
Björn Persson
Am 16.06.2021 um 13:47 schrieb Björn Persson Bjorn@xn--rombobjrn-67a.se: .... But I'm already planning to reinstall that one with Debian, ... so it won't hurt me if Fedora stops working there.
Do we really want to recommend this to our users?
The ability to continue using ‚mature', functional hardware was and is one of the significant arguments in favor of Linux.
* Stephen John Smoogen:
I used this https://unix.stackexchange.com/questions/631217/how-do-i-check-if-my-cpu-sup... to see what cpu instructions are at each level
#!/usr/bin/awk -f BEGIN { while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1 if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1 if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2 if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3 if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4 if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 } exit 1 }
Hmm. I believe the script is almost correct, not sure about “xsave” part. The “fma” match is problematic because it also applies to “fma4”, which is definitely not correct.
On Fedora 34 or later, you can use “/lib64/ld-linux-x86-64.so.2 --help”. If x86-64-v2 shows up as “supported”, there is compatibile:
| Subdirectories of glibc-hwcaps directories, in priority order: | x86-64-v4 | x86-64-v3 (supported, searched) | x86-64-v2 (supported, searched)
On older Fedora, you can run:
podman run fedora:latest /lib64/ld-linux-x86-64.so.2 --help
Thanks, Florian
On Wed, 16 Jun 2021 at 08:19, Florian Weimer fweimer@redhat.com wrote:
- Stephen John Smoogen:
I used this
https://unix.stackexchange.com/questions/631217/how-do-i-check-if-my-cpu-sup...
to see what cpu instructions are at each level
#!/usr/bin/awk -f BEGIN { while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1 if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/)level = 1
if (level == 1 &&/cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2
if (level == 2 &&/avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3
if (level == 3 &&/avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4
if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1}
exit 1}
Hmm. I believe the script is almost correct, not sure about “xsave” part. The “fma” match is problematic because it also applies to “fma4”, which is definitely not correct.
I think changing that to /fma[[:space:]]/ would fix that..
On Fedora 34 or later, you can use “/lib64/ld-linux-x86-64.so.2 --help”. If x86-64-v2 shows up as “supported”, there is compatibile:
| Subdirectories of glibc-hwcaps directories, in priority order: | x86-64-v4 | x86-64-v3 (supported, searched) | x86-64-v2 (supported, searched)
On older Fedora, you can run:
podman run fedora:latest /lib64/ld-linux-x86-64.so.2 --help
oh cool. this even works on CentOS and RHEL systems: ``` smooge@xanadu ~]$ podman run fedora:latest /lib64/ld-linux-x86-64.so.2 --help ... Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 x86-64-v3 (supported, searched) x86-64-v2 (supported, searched)
Legacy HWCAP subdirectories under library search path directories: haswell (AT_PLATFORM; supported, searched) tls (supported, searched) avx512_1 x86_64 (supported, searched) [smooge@xanadu ~]$ uname -a Linux xanadu.int.smoogespace.com 4.18.0-305.0.1.el8.x86_64 #1 SMP Fri May 28 11:04:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ``` from my oldest system.
Thanks, Florian
On 6/16/21 8:45 AM, Stephen John Smoogen wrote:
oh cool. this even works on CentOS and RHEL systems:
smooge@xanadu ~]$ podman run fedora:latest /lib64/ld-linux-x86-64.so.2 --help ... Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 x86-64-v3 (supported, searched) x86-64-v2 (supported, searched) Legacy HWCAP subdirectories under library search path directories: haswell (AT_PLATFORM; supported, searched) tls (supported, searched) avx512_1 x86_64 (supported, searched) [smooge@xanadu ~]$ uname -a Linux xanadu.int.smoogespace.com <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fxanadu.int.smoogespace.com%2F&data=04%7C01%7Cprzemek.klosowski%40nist.gov%7Cf32c4bad2b2e4a58374b08d930c4afc2%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C637594443609004509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FN9pE6UInNOsHnlh0Tz5KVzvBjFsWkl3xKMwWdh8DWk%3D&reserved=0> 4.18.0-305.0.1.el8.x86_64 #1 SMP Fri May 28 11:04:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linuxfrom my oldest system.
I'm missing something---I get identicaloutput on my v3 Core i7-4810MQ
Is this supposed to run HWCAP and show the result, or just show the HWCAP configuration and possible choices?
* przemek klosowski via devel:
I'm missing something---I get identicaloutput on my v3 Core i7-4810MQ
Why do you expect different output?
Is this supposed to run HWCAP and show the result, or just show the HWCAP configuration and possible choices?
It shows capabilities (as defined by the hardware, the kernel and glibc) and indicates which are supported in the current execution environment.
Thanks, Florian
On 6/16/21 12:09 PM, Florian Weimer wrote
I'm missing something---I get identicaloutput on my v3 Core i7-4810MQ
Why do you expect different output?
Stephen was showing off his 'oldest' system and I assumed that it was some Penryn-era relic, so I expected a <= v1 result. One cohort of people in this discussion argue that even the oldest systems are quite capable, and another cohort argues the opposite by showing off their 'black beauty' (CarTalk(TM)) antiques that they are loathe to retire. I mistook Stephen for the latter :)
In my limited understanding of HWCAPs, it run-time switches between different code paths for differently capable exec environments---and misreading Stephen's post made me wonder if it somehow emulates the newer ones, which didn't make sense.
On Wed, 16 Jun 2021 at 12:45, przemek klosowski via devel devel@lists.fedoraproject.org wrote:
On 6/16/21 12:09 PM, Florian Weimer wrote
I'm missing something---I get identicaloutput on my v3 Core i7-4810MQ
Why do you expect different output?
Stephen was showing off his 'oldest' system and I assumed that it was some Penryn-era relic, so I expected a <= v1 result. One cohort of people in this discussion argue that even the oldest systems are quite capable, and another cohort argues the opposite by showing off their 'black beauty' (CarTalk(TM)) antiques that they are loathe to retire. I mistook Stephen for the latter :)
Sorry I did not give enough information. I was mainly using it to show that the command Florian gave works on CentOS-8 systems. That said, I thought my oldest system was a 2009 system but it turns out it is a 2012 system. [I forgot the 2009 system lost some capacitors a couple of years ago.]
In my limited understanding of HWCAPs, it run-time switches between different code paths for differently capable exec environments---and misreading Stephen's post made me wonder if it somehow emulates the newer ones, which didn't make sense.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Jun 16, 2021 at 2:20 PM Florian Weimer fweimer@redhat.com wrote:
On Fedora 34 or later, you can use “/lib64/ld-linux-x86-64.so.2 --help”. If x86-64-v2 shows up as “supported”, there is compatibile:
| Subdirectories of glibc-hwcaps directories, in priority order: | x86-64-v4 | x86-64-v3 (supported, searched) | x86-64-v2 (supported, searched)
Hmm, I am wondering if we can't use output from that and gather this information as a part of DNF Count Me?
We'll at least gather information about capabilities of Fedora users hardware.
On Wed, Jun 16, 2021 at 8:57 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 16.06.2021 14:45, Frantisek Zatloukal wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
So how do you propose we figure out what kind of hardware we need to work with, further develop, or such? We used such telemetry in the past to do targeted improvements to hardware support (cf Smolt and graphics support).
* Neal Gompa:
On Wed, Jun 16, 2021 at 8:57 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 16.06.2021 14:45, Frantisek Zatloukal wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
So how do you propose we figure out what kind of hardware we need to work with, further develop, or such? We used such telemetry in the past to do targeted improvements to hardware support (cf Smolt and graphics support).
We can change glibc to enforce hardware requirements before the entire distribution and see how many people complain. This way, going back to the earlier state doesn't require a fresh mass rebuild.
Thanks, Florian
On Wed, Jun 16, 2021 at 02:57:17PM +0200, Vitaly Zaitsev via devel wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
On 16.06.2021 22:22, Matthew Miller wrote:
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
Built-in system level keylogger in one well-known operating system also helps its developers?
On Thu, 17 Jun 2021 at 04:45, Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 16.06.2021 22:22, Matthew Miller wrote:
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
Built-in system level keylogger in one well-known operating system also helps its developers?
1. This conversation is not about other operating systems. 2. No one is talking about installing keyloggers or similar tools.
I am saying the following from learning this the hard way. Bringing up extremes only causes people who are 'neutral' minded that you want to bring to your side to think the other side is probably more valid. If you want to talk about the evils of telemetry, stick to the case in point as limited as it is.
The case being: adding such information to dnf reporting so that countme or similar centralized logging can survey the Fedora user audience better.
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations. HTTPD logs already have the ip address, timestamp, requested information, and user-agent of the system. However it is very hard to say which of 10,000 systems behind a VPN was a particular request because the information just says x86_64 versus.. x86_64-v1 (v2 or v3). While not really pinpointing an individual it helps narrow down a search extremely.
There are useful points in getting telemetry data of this sort. We could know better that we have a 60% population of v2 systems or that 90% of our arm systems are not system ready but raspberry pi's. That would allow for engineering to focus its resources on things which the consumers of the OS need versus what a 'vocal population' want.
It is also a slippery slope that quickly adds more and more to the point where 'keyboard stroke telemetry' doesn't look like a big leap because you already have so many other ways to keep track of people, but you aren't sure if your new autospell or extra code key features are working well.
-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 17.06.2021 13:43, Stephen John Smoogen wrote:
- This conversation is not about other operating systems.
It was just an example of telemetry usage.
- No one is talking about installing keyloggers or similar tools.
Appetite comes with eating. More and more telemetry will be introduced over time. We will open Pandora's box.
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations.
GDPR is the best thing that has happened to the industry. I love it.
There are useful points in getting telemetry data of this sort. We could know better that we have a 60% population of v2 systems or that 90% of our arm systems are not system ready but raspberry pi's. That would allow for engineering to focus its resources on things which the consumers of the OS need versus what a 'vocal population' want.
And will also spy on users, violating their freedom.
Hi,
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations.
Well, depends on how you store the data. You can store one record per machine (with all properties in there), or you can store one record per property per machine.
With the latter you basically kill query on subgroups (like "how many x86_64-v3 machines use UEFI?") because that grouping information is gone if you store each end every little piece of information in its own record. But it'll also much harder to do fingerprinting on such a data base ...
Standard disclaimer: IANAL.
take care, Gerd
On Fri, 18 Jun 2021 at 01:51, Gerd Hoffmann kraxel@redhat.com wrote:
Hi,
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations.
Well, depends on how you store the data. You can store one record per machine (with all properties in there), or you can store one record per property per machine.
With the latter you basically kill query on subgroups (like "how many x86_64-v3 machines use UEFI?") because that grouping information is gone if you store each end every little piece of information in its own record. But it'll also much harder to do fingerprinting on such a data base ...
Standard disclaimer: IANAL.
The problem with IANAL, is that we all come up with great solutions which seem to match the single document we read. However the law is an interpreted language where every court is a slightly different architecture and has different libraries which have to be slowly interpreted and patched at a top level. This means that you end up with finding out that the document and 2500 years of law rulings have to be interpreted together.
The way things are interpreted currently, it doesn't matter that you stored it differently.. it matters that you collected it... mainly because there is a long history of people finding ways to de-anonymize data, people lying about anonymizing it, and people somehow collecting the data in the middle. Because of that you end up having to delete all the data when someone asks to be deleted because you can't prove this record/count was their system or not.
In general we computer people like to dive in and just collect data and go about doing analysis. The various privacy laws are written to make us do a LOT of hard work before we start doing that. You end up spending a lot of time with lawyers versed in European, Brazilian, and various other countries laws/regulations/past history to figure out what you can collect, how you can collect it, how you are going to delete it, how you are going to inform people that things are happening, and having clear processes that are followed. Then you can start writing the code.. while doing that you have to review the code to make sure it is still meeting current rulings. [Doing it another way ends up with you writing code and either finding you have to delete it all or waiting months for an approval before rolling it out.]
take care, Gerd _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
We had several discussions recently that could use some real-world data on e.g.:
- x86_64-v2 prevalence
- GUILE usage in make/gdb
- count of systems with UEFI/GPT vs BIOS/MBR
- debugd server usage
- etc
The common thread is that some sort of measurement would help figuring out the best technical solution for the Fedora community, but such measurement would require transmitting and collecting data in the Fedora infrastructure. Such telemetry is usually criticized from two related angles: a general objection to online telemetry, and a practical argument about the complicated legal ramifications of online data collection in multiple jurisdictions. Stephen recently responded with an eloquent argument why it's very hard to come up with an acceptable collection scheme (see below).
This problem is of course not unique to Fedora---everyone is in the same boat of finding a middle ground between anonymity and indiscriminate data collection. I think most people would agree that data-driven decisions are better than gut feeling-based ones, so it is to our benefit to control and possibly allow _some_ data to be used for that purpose.
Few days ago I attended a talk by a practicing data scientist working with social data where the privacy issues are even more important: some of the data can literally have life-altering consequences to people covered by the collection. I asked him about the best practices and guidelines for responsible data collection, and he directed me to this presentation:
https://the-engine-room.github.io/responsible-data-handbook/pages/slides.htm...
My personal take-aways from reading this material are:
- we're not alone in this---other people have thought through these issues and came up with workable ideas
- it helps to keep things in perspective----the data about people's computers is less consequential than e.g data about their ethnicity or politics
- there are legal requirements for Consent (notice/disclosure): they are workable, though
- it matters how the data is used: publishing full logs vs. using the aggregated data for internal improvement
Anyway, this is my personal, uneducated take on it. Hope it is helpful.
On 6/18/21 9:39 AM, Stephen John Smoogen wrote:
On Fri, 18 Jun 2021 at 01:51, Gerd Hoffmann kraxel@redhat.com wrote:
Hi,
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations.
Well, depends on how you store the data. You can store one record per machine (with all properties in there), or you can store one record per property per machine.
With the latter you basically kill query on subgroups (like "how many x86_64-v3 machines use UEFI?") because that grouping information is gone if you store each end every little piece of information in its own record. But it'll also much harder to do fingerprinting on such a data base ...
Standard disclaimer: IANAL.
The problem with IANAL, is that we all come up with great solutions which seem to match the single document we read. However the law is an interpreted language where every court is a slightly different architecture and has different libraries which have to be slowly interpreted and patched at a top level. This means that you end up with finding out that the document and 2500 years of law rulings have to be interpreted together.
The way things are interpreted currently, it doesn't matter that you stored it differently.. it matters that you collected it... mainly because there is a long history of people finding ways to de-anonymize data, people lying about anonymizing it, and people somehow collecting the data in the middle. Because of that you end up having to delete all the data when someone asks to be deleted because you can't prove this record/count was their system or not.
In general we computer people like to dive in and just collect data and go about doing analysis. The various privacy laws are written to make us do a LOT of hard work before we start doing that. You end up spending a lot of time with lawyers versed in European, Brazilian, and various other countries laws/regulations/past history to figure out what you can collect, how you can collect it, how you are going to delete it, how you are going to inform people that things are happening, and having clear processes that are followed. Then you can start writing the code.. while doing that you have to review the code to make sure it is still meeting current rulings. [Doing it another way ends up with you writing code and either finding you have to delete it all or waiting months for an approval before rolling it out.]
On Wed, 7 Jul 2021 at 16:26, przemek klosowski via devel devel@lists.fedoraproject.org wrote:
We had several discussions recently that could use some real-world data on e.g.:
x86_64-v2 prevalence
GUILE usage in make/gdb
count of systems with UEFI/GPT vs BIOS/MBR
debugd server usage
etc
The common thread is that some sort of measurement would help figuring out the best technical solution for the Fedora community, but such measurement would require transmitting and collecting data in the Fedora infrastructure. Such telemetry is usually criticized from two related angles: a general objection to online telemetry, and a practical argument about the complicated legal ramifications of online data collection in multiple jurisdictions. Stephen recently responded with an eloquent argument why it's very hard to come up with an acceptable collection scheme (see below).
This problem is of course not unique to Fedora---everyone is in the same boat of finding a middle ground between anonymity and indiscriminate data collection. I think most people would agree that data-driven decisions are better than gut feeling-based ones, so it is to our benefit to control and possibly allow _some_ data to be used for that purpose.
Few days ago I attended a talk by a practicing data scientist working with social data where the privacy issues are even more important: some of the data can literally have life-altering consequences to people covered by the collection. I asked him about the best practices and guidelines for responsible data collection, and he directed me to this presentation:
https://the-engine-room.github.io/responsible-data-handbook/pages/slides.htm...
My personal take-aways from reading this material are:
Thank you for doing this research. One of the items towards doing any sort of 'data analysis' is working through what is currently done in the field.
- we're not alone in this---other people have thought through these
issues and came up with workable ideas
- it helps to keep things in perspective----the data about people's
computers is less consequential than e.g data about their ethnicity or politics
People choose things based on the social groups they are 'bound' with so you will find that people with XYZ brand computers trend more towards being some ethnicity or politics. It isn't an absolute rule, but it tends towards a large enough that 'possessions' are considered PII in some places. [There was a strong link between people having systems with multiple nvidia cards and libertarian politics because a lot of people in both groups mined bitcoin. There is a correlation between languages installed on a system and the ethnicity/demographics of the person. ]
It is also important to plan the research with the following in mind: 1. No matter how hard you try it is hard to anonymize data. 2. No matter how hard you try, you will be collecting items which can 'tell' about people. 3. People have the right to have their data deleted anytime/anywhere. [There are exceptions to this but when you are getting volunteer data and not part of maintaining legal records for business transactions, it will need to be deleted.] 4. You need to design the collection to 'mine' gross items from the data and then destroy that data as quickly as possible. 5. You need to design the collection with the expectation that it will be something people want to steal and so needs to be kept secure. 6. When you are collecting data, there is a segment of the population who will lie/cheat/steal as much as they can for various personal and ideological reasons. You need to make the experiment expect that and try to determine when it is happening. In either case, the data you have will be noisy.]
- there are legal requirements for Consent (notice/disclosure): they are
workable, though
- it matters how the data is used: publishing full logs vs. using the
aggregated data for internal improvement
Anyway, this is my personal, uneducated take on it. Hope it is helpful.
On 6/18/21 9:39 AM, Stephen John Smoogen wrote:
On Fri, 18 Jun 2021 at 01:51, Gerd Hoffmann kraxel@redhat.com wrote:
Hi,
The problems with this is that we are taking a fairly fuzzy data set and making it much easier to track individual users in ways seen as problematic by various laws and regulations.
Well, depends on how you store the data. You can store one record per machine (with all properties in there), or you can store one record per property per machine.
With the latter you basically kill query on subgroups (like "how many x86_64-v3 machines use UEFI?") because that grouping information is gone if you store each end every little piece of information in its own record. But it'll also much harder to do fingerprinting on such a data base ...
Standard disclaimer: IANAL.
The problem with IANAL, is that we all come up with great solutions which seem to match the single document we read. However the law is an interpreted language where every court is a slightly different architecture and has different libraries which have to be slowly interpreted and patched at a top level. This means that you end up with finding out that the document and 2500 years of law rulings have to be interpreted together.
The way things are interpreted currently, it doesn't matter that you stored it differently.. it matters that you collected it... mainly because there is a long history of people finding ways to de-anonymize data, people lying about anonymizing it, and people somehow collecting the data in the middle. Because of that you end up having to delete all the data when someone asks to be deleted because you can't prove this record/count was their system or not.
In general we computer people like to dive in and just collect data and go about doing analysis. The various privacy laws are written to make us do a LOT of hard work before we start doing that. You end up spending a lot of time with lawyers versed in European, Brazilian, and various other countries laws/regulations/past history to figure out what you can collect, how you can collect it, how you are going to delete it, how you are going to inform people that things are happening, and having clear processes that are followed. Then you can start writing the code.. while doing that you have to review the code to make sure it is still meeting current rulings. [Doing it another way ends up with you writing code and either finding you have to delete it all or waiting months for an approval before rolling it out.]
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Dne 07. 07. 21 v 22:25 przemek klosowski via devel napsal(a):
- count of systems with UEFI/GPT vs BIOS/MBR
BTW:
https://linux-hardware.org/?view=os_boot_mode
Hmm, I wanted to run
|systemctl enable --now hw-probe.timer|
|But |
| https://github.com/linuxhw/hw-probe/tree/master/periodic%7C
|is not packaged as part of hw-probe. Anyone knows why?|
|Miroslav |
On 6/17/21 4:44 AM, Vitaly Zaitsev via devel wrote:
On 16.06.2021 22:22, Matthew Miller wrote:
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
Built-in system level keylogger in one well-known operating system also helps its developers?
You are arguing a 'slippery slope' of increasingly intrusive telemetry, but we should judge each thing on its own merits. Yes, a keylogger is a bad idea--but it does not follow that keeping track of the hardware configurations, or logging software failures is also super-bad. If the benefits outweigh costs, the tradeoff is worth considering---this is true of software just as well as of, say, vaccines.
The problem with 'slippery slope' arguments is that they themselves can be abused to say that no change is possible because it could be taken too far and lead to abuse.
On Wed, Jun 16, 2021 at 3:23 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Jun 16, 2021 at 02:57:17PM +0200, Vitaly Zaitsev via devel wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
I think there can be a lot of benefit in anonymized hardware data (not mandatory). It does help answer questions like this, but more importantly, it would make a lot of the kernel work a bit easier, or at least more focused. It answers questions like, "should we enable these drivers as they are likely to be used?" or "can we disable these drivers because no one is using them?". It is also very helpful in working out bug priority in drivers. A lot of people never bother filing bugs, and are happy to keep booting a known good kernel since we allow parallel installs. If we get a few users chiming in, and realize the hardware in question is used by a significant chunk of users, it would tell me that perhaps that should take priority over a bug which impacts hardware with considerably fewer users. Yes, you have to be extremely careful about what data you collect, and how that data is handled, but if done correctly, there are a lot of benefits.
Justin
On Thu, Jun 17, 2021 at 10:26 AM Justin Forbes jmforbes@linuxtx.org wrote:
On Wed, Jun 16, 2021 at 3:23 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Jun 16, 2021 at 02:57:17PM +0200, Vitaly Zaitsev via devel wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
I think there can be a lot of benefit in anonymized hardware data (not mandatory). It does help answer questions like this, but more importantly, it would make a lot of the kernel work a bit easier, or at least more focused. It answers questions like, "should we enable these drivers as they are likely to be used?" or "can we disable these drivers because no one is using them?". It is also very helpful in working out bug priority in drivers. A lot of people never bother filing bugs, and are happy to keep booting a known good kernel since we allow parallel installs. If we get a few users chiming in, and realize the hardware in question is used by a significant chunk of users, it would tell me that perhaps that should take priority over a bug which impacts hardware with considerably fewer users. Yes, you have to be extremely careful about what data you collect, and how that data is handled, but if done correctly, there are a lot of benefits.
I agree. I'm curious how many folks have TPMs at all, if they're 1.2 enabled or 2.0; whether the system is Secure Boot capable, enabled or disabled; whether the firmware is BIOS, UEFI, coreboot.
For Fedora, linux-hardware.org says 78% use EFI and 16% have Secure Boot enabled. Not a very good data set, though Fedora telemetry wouldn’t be either.
On 6/18/21 1:42 AM, Mark Otaris wrote:
For Fedora, linux-hardware.org says 78% use EFI and 16% have Secure Boot enabled. Not a very good data set, though Fedora telemetry wouldn’t be either.
Of ~ 1K linux boxes in&around my env -- bare metal & VM, servers & desktops -- ~ 550 are now running fully updated F34.
Of those, ~ 95% are AMD; remainder are Intel. No ARM.
Of the total, per a quick survey,
~ 15% have _no_ avx/avx2 support ~ 40% have avx-only support rest have avx/avx2 support
NONE of the no-avx or avx-only machines are "suffering" from any performance problems for their current business/technical usage.
Here, the 1st response to "my OS suddenly no longer supports this hardware" will seldom-if-ever be to replace the hardware.
At best, I'll freeze the machine's s/w base @ last-supported, and look to replace the OS.
There are real/significant costs to H/W upgrade/replacement -- both capital & expense; Planning & budgeting for them is often on a multi-year timeframe. Particularly at scale. And my env is relatively quite small on the global stage.
This seemingly endless stream of drop-the-old-hardware discussions, even if reasonable for some, is a cause for tangible concern that I've gotten myself into a tactical, eggs-in-one-basket mess. Again.
With RockyLinux today at GA, time to explore/compare a 'plan B' & whether it provides some differently sensible support planning. Or whether YA-x-distro firedrill is inevitable.
( "fun fact" for me:
my own laptop that sits ~ 3 ft from me as I type this, serves as my diagnostic/troubleshooting box on my LAN, and when traveling, obviously falls into the currently discussed 'decrepit' pile, lacking modern flags
egrep "model name|flags" /proc/cpuinfo | head -n2 model name : Intel(R) Core(TM) i3 CPU M 370 @ 2.40GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm arat flush_l1d
but _very_ capably runs
lsb_release -rd Description: Fedora release 34 (Thirty Four) Release: 34
uname -rm 5.12.11-300.fc34.x86_64 x86_64
with
KDE 5.82.0 / Plasma 5.21.5
and, although I don't use, or plan to use, it as a networked build farm anytime soon ... bind9 certainly builds from source well enough on it; though, admittedly, takes awhile.
)
On Thu, 17 Jun 2021 at 12:27, Justin Forbes jmforbes@linuxtx.org wrote:
On Wed, Jun 16, 2021 at 3:23 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Jun 16, 2021 at 02:57:17PM +0200, Vitaly Zaitsev via devel wrote:
We'll at least gather information about capabilities of Fedora users hardware.
Telemetry is evil. It must not be allowed.
Well, that's certainly A Position. I don't think it's anything nearly so absolute, though, and depends on what, who, how, why, and a host of other things. And "it can help us answer questions like this for our community" is a pretty non-evil "why".
I think there can be a lot of benefit in anonymized hardware data (not mandatory). It does help answer questions like this, but more importantly, it would make a lot of the kernel work a bit easier, or at least more focused. It answers questions like, "should we enable these drivers as they are likely to be used?" or "can we disable these drivers because no one is using them?". It is also very helpful in working out bug priority in drivers. A lot of people never bother filing bugs, and are happy to keep booting a known good kernel since we allow parallel installs. If we get a few users chiming in, and realize the hardware in question is used by a significant chunk of users, it would tell me that perhaps that should take priority over a bug which impacts hardware with considerably fewer users. Yes, you have to be extremely careful about what data you collect, and how that data is handled, but if done correctly, there are a lot of benefits.
The major problem is that 'anonymized' data does not exist. Pretty much every method which says it 'anonymizes' stuff does not and can lead to a strong 'fingerprint' back to an individual or group. The only methods which truly do seem to stop this basically add so much random noise to the data that it is 'useless' for whatever analysis. [AKA you might as well just tell /dev/urandom to give you a couple of gigabytes of answers.]
This means that any program you use to collect information needs to assume that it will have to be regularly purged/cleaned/etc. You will have to only take snapshots of very high level data to compare to other timeframes. It also has to assume that there are enough people who do not want to be watched (even if you ask for them to volunteer info) that they will feed you bad data. This is why we had more PDP-11's in smolt than we had some valid architectures we shipped.
Once you start finding these limits, you realize you don't want to mix it with data you need for continual operation of service. If you do, you will lose that data regularly also, may be told to turn off, or find yourself spending too many resources to keep it. This is the reason I object to adding too much to mirrorlist data. That is a service which we need to keep up and we need to keep some history for general operations. [We need to know how many resources we are servicing and how long it takes to respond. Having multiple years helped show when the mirror program began to fall over from too many customers and the fact that the change in certain yum cron jobs were increasingly causing issues.]
On Wed, 16 Jun 2021 at 08:46, Frantisek Zatloukal fzatlouk@redhat.com wrote:
On Wed, Jun 16, 2021 at 2:20 PM Florian Weimer fweimer@redhat.com wrote:
On Fedora 34 or later, you can use “/lib64/ld-linux-x86-64.so.2 --help”. If x86-64-v2 shows up as “supported”, there is compatibile:
| Subdirectories of glibc-hwcaps directories, in priority order: | x86-64-v4 | x86-64-v3 (supported, searched) | x86-64-v2 (supported, searched)
Hmm, I am wondering if we can't use output from that and gather this information as a part of DNF Count Me?
We'll at least gather information about capabilities of Fedora users hardware.
Not really.. adding in exact architectures and stuff starts getting into areas where GDPR and similar regulations consider profiling even if it isn't easy for us or what we want to do. [And no it doesn't matter if people who have read the various regulations say that isn't really what GDPR says.. the courts and lawyers have said that once you add in all the back laws, rulings and other things.. it is what the GDPR (and other laws) become. ]
At best we can try some sort of 'volunteer' popcorn/smolt system, but it is rife with needing to design in 2 things common with them: 1. The information being given is going to be abused by malefactors so you need to make sure it is filtered. 2. The information even if volunteered can and must be expunged when the volunteer wants it to be.
Those two items are a lot of work which usually is more than anyone wants to put into a survey system.
On Tue, Jun 15, 2021 at 11:35 PM Neal Gompa ngompa13@gmail.com wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
Uhmmmm ... you mean something like this? https://fedoraproject.org/wiki/Changes/x86-64_micro-architecture_update Rejected by FESCo two years ago: https://pagure.io/fesco/issue/2198
If I remember correctly, it was shot down pretty quickly on the devil list, because Intel still produced CPUs that do not support AVX2. Bumping the baseline to x86_64v2 would be different, since AVX is not part of that (but only x86_64v3).
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by now?
Fabio
On Tue, Jun 15, 2021 at 5:50 PM Fabio Valentini decathorpe@gmail.com wrote:
On Tue, Jun 15, 2021 at 11:35 PM Neal Gompa ngompa13@gmail.com wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
Uhmmmm ... you mean something like this? https://fedoraproject.org/wiki/Changes/x86-64_micro-architecture_update Rejected by FESCo two years ago: https://pagure.io/fesco/issue/2198
If I remember correctly, it was shot down pretty quickly on the devil list, because Intel still produced CPUs that do not support AVX2. Bumping the baseline to x86_64v2 would be different, since AVX is not part of that (but only x86_64v3).
Yeah, I think that proposal was not workable because of AVX2. The x86_64-v2 subarch adds SSSE3, SSE4.2, POPCNT, and CMPXCHG16B to the current x86_64 baseline. All of these instructions were present in the first Intel Macs launched in 2007, as I recall.
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by now?
No idea, good question, though!
-- 真実はいつも一つ！/ Always, there's only one truth!
On 15/06/2021 22:54, Neal Gompa wrote:
On Tue, Jun 15, 2021 at 5:50 PM Fabio Valentini decathorpe@gmail.com wrote:
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by now?
No idea, good question, though!
If you mean feature level based loading of shared libraries via hwcaps then it is supposed to be supported in 2.33 which is in F34.
Tom
On Tue, Jun 15, 2021 at 9:55 PM Neal Gompa ngompa13@gmail.com wrote:
Yeah, I think that proposal was not workable because of AVX2. The x86_64-v2 subarch adds SSSE3, SSE4.2, POPCNT, and CMPXCHG16B to the current x86_64 baseline. All of these instructions were present in the first Intel Macs launched in 2007, as I recall.
I believe the earliest 64-bit capable Intel Macs were using core 2 processors, and they would not include a couple of those instructions (which happened with nehalem).
Neal Gompa wrote:
Yeah, I think that proposal was not workable because of AVX2. The x86_64-v2 subarch adds SSSE3, SSE4.2, POPCNT, and CMPXCHG16B to the current x86_64 baseline. All of these instructions were present in the first Intel Macs launched in 2007, as I recall.
Still means my Core 2 Duo notebook would no longer be able to run Fedora.
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by now?
No idea, good question, though!
Indeed, runtime detection is clearly the way to go.
Why does this proposal to desupport hardware for no good reason (because the performance gain can be obtained in a compatible way, where it is even noticeable at all) keep coming up again and again?
Kevin Kofler
On Wed, Jun 16, 2021 at 6:08 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Neal Gompa wrote:
Yeah, I think that proposal was not workable because of AVX2. The x86_64-v2 subarch adds SSSE3, SSE4.2, POPCNT, and CMPXCHG16B to the current x86_64 baseline. All of these instructions were present in the first Intel Macs launched in 2007, as I recall.
Still means my Core 2 Duo notebook would no longer be able to run Fedora.
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by now?
No idea, good question, though!
Indeed, runtime detection is clearly the way to go.
Why does this proposal to desupport hardware for no good reason (because the performance gain can be obtained in a compatible way, where it is even noticeable at all) keep coming up again and again?
It keeps coming up because we don't have support for subarches in RPM for anything but ARM architectures. That deficiency forces us to keep considering raising the baseline instead of being able to have select packages with subarch content. This is important because not everything *can* use glibc-hwcap hardware detection at runtime, and sometimes you need optimized binaries.
-- 真実はいつも一つ！/ Always, there's only one truth!
On June 17, 2021 12:08:44 AM UTC, Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jun 16, 2021 at 6:08 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Neal Gompa wrote:
Yeah, I think that proposal was not workable because of AVX2. The x86_64-v2 subarch adds SSSE3, SSE4.2, POPCNT, and CMPXCHG16B to the current x86_64 baseline. All of these instructions were present in
the
first Intel Macs launched in 2007, as I recall.
Still means my Core 2 Duo notebook would no longer be able to run
Fedora.
Different question: How is the runtime CPU feature detection / dispatch support in glibc coming along? Shouldn't this "work" by
now?
No idea, good question, though!
Indeed, runtime detection is clearly the way to go.
Why does this proposal to desupport hardware for no good reason
(because the
performance gain can be obtained in a compatible way, where it is
even
noticeable at all) keep coming up again and again?
It keeps coming up because we don't have support for subarches in RPM for anything but ARM architectures. That deficiency forces us to keep considering raising the baseline instead of being able to have select packages with subarch content. This is important because not everything *can* use glibc-hwcap hardware detection at runtime, and sometimes you need optimized binaries.
Honestly, this sounds to me like we rather should find a solution how to "fix" this in RPM (I know of the previous unsuccessful attempts). Otherwise I don't really see compelling reasons why we should raise the baseline to x86_64v2 when it provides very small gain. Because in contrast to runtime detection or cutting edge software, this has the disadvantage that once you raise it, you throw people out. So it better be worth it and ATM I don't see why it is.
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
* Neal Gompa:
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Right, I have yet to encounter anyone who can't run the new el9 binaries on their hardware. We had a few issues, but they were all misconfiguration of hypervisors or software emulators.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
An add-on architecture seems unlikely.
I do not want to propose the change for Fedora 35 myself. Maybe for Fedora 36. We could do the change in Fedora 35 if someone else champions it and gets it through the process.
Thanks, Florian
Florian Weimer wrote:
Right, I have yet to encounter anyone who can't run the new el9 binaries on their hardware. We had a few issues, but they were all misconfiguration of hypervisors or software emulators.
Let me introduce you to my notebook:
[kevin@laptop64 ~]$ cat /etc/fedora-release Fedora release 33 (Thirty Three) [kevin@laptop64 ~]$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz stepping : 10 microcode : 0x95 cpu MHz : 1197.087 cache size : 4096 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm vmx flags : vnmi flexpriority tsc_offset vtpr vapic bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 4788.34 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz stepping : 10 microcode : 0x95 cpu MHz : 1197.075 cache size : 4096 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm vmx flags : vnmi flexpriority tsc_offset vtpr vapic bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 4788.34 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
As you can see from fedora-release, it runs Fedora 33 just fine. I have yet to try Fedora 34 on it, but I expect that to work fine, too. But anything requiring flags like sse4_1 sse4_2 popcnt etc. will not.
Kevin Kofler
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
There is of course always a balance between bumping min hardware specs and the impact on maintainers & users, but I'm not convinced that we have the balance right in targeting our x86_64 baseline at the very first generation of 64-bit CPUs from 15 years ago. I can't imagine such old CPUs makes up a significant portion of our users.
Regards, Daniel
Hi,
On 6/16/21 10:28 AM, Daniel P. Berrangé wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
There is of course always a balance between bumping min hardware specs and the impact on maintainers & users, but I'm not convinced that we have the balance right in targeting our x86_64 baseline at the very first generation of 64-bit CPUs from 15 years ago. I can't imagine such old CPUs makes up a significant portion of our users.
I don't know about that, all I can offer is my own anecdotal to the contrary. Of the 7 PCs/laptops which are in more or less daily use in our houshold 3 of them are still core2 duo systems.
Once the core2 duo / amd64 machines came out we really started hitting the point of diminishing returns wrt PC performance for day 2 day use. For a lot of simple day2 day use there really is no reason to replace and x86_64-v1 capable machines unless they are actually broken.
Perhaps more importantly though, is that there we are also very much at the point where bumping the processor architecture requirements also leads to strongly diminishing returns.
Also see Mateusz Jończyk excellent reply in this thread, how rebuilding packages for x86_64-v2 vs x86_64 results in a barely measurable performance improvement.
Of course there are some specific algorithms which greatly benefit from sse4.2, but those typically benefit even more from avx/avx2 which are not included in x86_64-v2; and often libraries already contain avx optimized code-paths for this which they automatically use where possible.
You talk about we "hold back our software by trying to keep supporting long obsolete hardware". Let me flip the question can you provide hard proof, as in concrete numbers showing significant improvements, that switching to x86_64-v2 actually buys us anything meaningful ?
Because this whole switch to x86_64-v2 sounds to me like it is mostly being pursued because it is a higher number so it must be better...
Dropping 32 bit x86 support was a good thing to do because the virtual memory space offered by 32 bits was simply becoming too small for many desktop applications like web-browsers; and things were starting to become noticeably broken there because upstream where no longer testing on it.
Forcing all Fedora PC users to move to x86_64 users as such was a good thing to do and also had a significant benefits in the form of lessening maintainer loads. Switching to x86_64-v2 OTOH simply does not seem to give any significant benefits.
Regards,
Hans
On Wed, Jun 16, 2021 at 12:01:29PM +0200, Hans de Goede wrote:
Hi,
On 6/16/21 10:28 AM, Daniel P. Berrangé wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
There is of course always a balance between bumping min hardware specs and the impact on maintainers & users, but I'm not convinced that we have the balance right in targeting our x86_64 baseline at the very first generation of 64-bit CPUs from 15 years ago. I can't imagine such old CPUs makes up a significant portion of our users.
I don't know about that, all I can offer is my own anecdotal to the contrary. Of the 7 PCs/laptops which are in more or less daily use in our houshold 3 of them are still core2 duo systems.
Once the core2 duo / amd64 machines came out we really started hitting the point of diminishing returns wrt PC performance for day 2 day use. For a lot of simple day2 day use there really is no reason to replace and x86_64-v1 capable machines unless they are actually broken.
Perhaps more importantly though, is that there we are also very much at the point where bumping the processor architecture requirements also leads to strongly diminishing returns.
Also see Mateusz Jończyk excellent reply in this thread, how rebuilding packages for x86_64-v2 vs x86_64 results in a barely measurable performance improvement.
Of course there are some specific algorithms which greatly benefit from sse4.2, but those typically benefit even more from avx/avx2 which are not included in x86_64-v2; and often libraries already contain avx optimized code-paths for this which they automatically use where possible.
You talk about we "hold back our software by trying to keep supporting long obsolete hardware". Let me flip the question can you provide hard proof, as in concrete numbers showing significant improvements, that switching to x86_64-v2 actually buys us anything meaningful ?
I wasn't so much thinking about the performance benefit, rather the CMPXCHG16B support which IIUC is required for atomics on 128 bit quantities and isn't present in the x86_64 baseline. QEMU already unconditionally adds -mcx16 to its CFLAGS to enable usage of this instruction.
I did think there might be some performance benefits too, so it was interesting to see the disappointing results posted elsewhere in this thread.
Regards, Daniel
Hi,
On 6/16/21 12:19 PM, Daniel P. Berrangé wrote:
On Wed, Jun 16, 2021 at 12:01:29PM +0200, Hans de Goede wrote:
Hi,
On 6/16/21 10:28 AM, Daniel P. Berrangé wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
There is of course always a balance between bumping min hardware specs and the impact on maintainers & users, but I'm not convinced that we have the balance right in targeting our x86_64 baseline at the very first generation of 64-bit CPUs from 15 years ago. I can't imagine such old CPUs makes up a significant portion of our users.
I don't know about that, all I can offer is my own anecdotal to the contrary. Of the 7 PCs/laptops which are in more or less daily use in our houshold 3 of them are still core2 duo systems.
Once the core2 duo / amd64 machines came out we really started hitting the point of diminishing returns wrt PC performance for day 2 day use. For a lot of simple day2 day use there really is no reason to replace and x86_64-v1 capable machines unless they are actually broken.
Perhaps more importantly though, is that there we are also very much at the point where bumping the processor architecture requirements also leads to strongly diminishing returns.
Also see Mateusz Jończyk excellent reply in this thread, how rebuilding packages for x86_64-v2 vs x86_64 results in a barely measurable performance improvement.
Of course there are some specific algorithms which greatly benefit from sse4.2, but those typically benefit even more from avx/avx2 which are not included in x86_64-v2; and often libraries already contain avx optimized code-paths for this which they automatically use where possible.
You talk about we "hold back our software by trying to keep supporting long obsolete hardware". Let me flip the question can you provide hard proof, as in concrete numbers showing significant improvements, that switching to x86_64-v2 actually buys us anything meaningful ?
I wasn't so much thinking about the performance benefit, rather the CMPXCHG16B support which IIUC is required for atomics on 128 bit quantities and isn't present in the x86_64 baseline. QEMU already unconditionally adds -mcx16 to its CFLAGS to enable usage of this instruction.
CMPXCHG16B is indeed supported on pretty much any x86_64 machine, including on Intel Conroe and Penryn AFAICT.
Only very old AMD64 CPUs, which are still using DDR1 don't support this (AFAICT).
So yes requiring that is probably fine.
I did think there might be some performance benefits too, so it was interesting to see the disappointing results posted elsewhere in this thread.
Ack, I suspect that the cases where there are really significant gains in using SSE4 are already covered by (optional) SSE4 optimized code paths.
I think it would be better if we want to look into using newer instructions into looking into things like this.
E.g. if there is some library which does significantly benefit from gcc's auto-vectorisation with sse4 and/or avx then we could build it multiple times with different settings and use the hwcaps based library loading mechanism to make an optimized version get loaded on hw which supports it. This way we can even use avx / x86_64-v3 / -v4 in cases where this actually is worth the extra effort + diskspace.
Regards,
Hans
On Wed, Jun 16, 2021 at 09:28:47AM +0100, Daniel P. Berrangé wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
What many seem to confuse here is when there were the first CPUs that support a particular set of instruction versus the last one that does not. Intel almost prides itself on its product segmentation, and there is a ton of recently (in 2020!) released Pentium/Celeron/Atom CPUs that still do not support even AVX[1][2].
[1] https://ark.intel.com/content/www/us/en/ark/products/203896/intel-pentium-go... released in Q2 2020, only SSE4.1/4.2 is listed, cf. https://ark.intel.com/content/www/us/en/ark/products/199283/intel-core-i3-10... [2] https://www.tomshardware.com/news/intels-latest-celeron-and-pentium-cpus-fin...
On Wed, Jun 16, 2021 at 12:37:57PM +0200, Eugene Syromiatnikov wrote:
On Wed, Jun 16, 2021 at 09:28:47AM +0100, Daniel P. Berrangé wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
What many seem to confuse here is when there were the first CPUs that support a particular set of instruction versus the last one that does not. Intel almost prides itself on its product segmentation, and there is a ton of recently (in 2020!) released Pentium/Celeron/Atom CPUs that still do not support even AVX[1][2].
[1] https://ark.intel.com/content/www/us/en/ark/products/203896/intel-pentium-go... released in Q2 2020, only SSE4.1/4.2 is listed, cf. https://ark.intel.com/content/www/us/en/ark/products/199283/intel-core-i3-10... [2] https://www.tomshardware.com/news/intels-latest-celeron-and-pentium-cpus-fin...
Oops, I confused "Gen 3" above with x86_64-v3; however, with regards to v2 support, Westmere-based Pentiums/Celerons were released in 2011 and do not support SSE4.1/4.2.
On Wed, Jun 16, 2021 at 12:52:22PM +0200, Eugene Syromiatnikov wrote:
Oops, I confused "Gen 3" above with x86_64-v3; however, with regards to v2 support, Westmere-based Pentiums/Celerons were released in 2011 and do not support SSE4.1/4.2.
And until the Silvermont core (in 2013) the Atom family didn't support SSE4.1/4.2 either.
AMD supported SSE4.1/4.2 starting with Bulldozer in 2011, but their embedded series didn't get it until 2013 with Jaguar.
Oh, one other data point -- VIA didn't support SSE4.2 until their Nano C in 2015, though they had support for SSE4.1 starting with their Nano 3000 series in 2009.
Anyway. Personally, I only have one Fedora system still in use that isn't at least x86_64v2 -- a dual-socket pre-Bulldozer Opteron server that is probably the single most important system I have.
- Solomon
On Wed, 16 Jun 2021 at 04:29, Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
I think it comes down to what various people who maintain or help maintain large amounts of hardware have available to use or feel comfortable using. People in academia usually have tight capex budgets and when they get a system they like, they are probably going to be using it for decades. [Queue several universities in the last 10 years I have seen where people have computers older than they are on their desks as their work computer.] Other people have other concerns and find newer systems not able to meet them (screen may be wrong, keyboard feels wrong, their lab is still running N year old hardware etc.) [Looking at the many internal mailing lists where people would prefer not to have their hardware updated every 3 years but still pine for that 10 year old computer they started with and none of the others have been as good as.]
Finally, most of the changes in the architecture code don't really make things 'faster' in ways that 'matter'. Using AVX2 versus SSE2 versus SSE does not make compilations faster
There is of course always a balance between bumping min hardware specs and the impact on maintainers & users, but I'm not convinced that we have the balance right in targeting our x86_64 baseline at the very first generation of 64-bit CPUs from 15 years ago. I can't imagine such old CPUs makes up a significant portion of our users.
In this case it doesn't 'matter' it is a small segment of users. It is a segment of our maintainers who are. We either have to listen to them, 'fire them', or buy them replacement hardware. Since we are already overloaded, firing them has not been on the table. Buying replacement hardware is expensive in multiple ways (time, capex, and various legal aspects). That leaves listening to the maintainers.
On Wed, 16 Jun 2021 at 08:16, Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 16 Jun 2021 at 04:29, Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Jun 15, 2021 at 05:34:02PM -0400, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Yes, you loose primarily Intel Conroe and Penryn generations and AMD Opteron Gen 1 -> Gen 3. I doubt this is a significant portion of Fedora installs.
Slight tangent but I find Fedora's approach to hardware somewhat at odds with our approach to software.
On the one hand we portray our project as a place for cutting edge Linux software & innovation.
On the other hand we hold back our software by trying to keep supporting long obsolete hardware.
I think it comes down to what various people who maintain or help maintain large amounts of hardware have available to use or feel comfortable using. People in academia usually have tight capex budgets and when they get a system they like, they are probably going to be using it for decades. [Queue several universities in the last 10 years I have seen where people have computers older than they are on their desks as their work computer.] Other people have other concerns and find newer systems not able to meet them (screen may be wrong, keyboard feels wrong, their lab is still running N year old hardware etc.) [Looking at the many internal mailing lists where people would prefer not to have their hardware updated every 3 years but still pine for that 10 year old computer they started with and none of the others have been as good as.]
Finally, most of the changes in the architecture code don't really make things 'faster' in ways that 'matter'. Using AVX2 versus SSE2 versus SSE does not make compilations faster
sorry my editing skills are poor this morning. I meant that things like compilations, tools and graphics don't really increase for the things that maintainers care about.
Am 16.06.2021 um 14:16 schrieb Stephen John Smoogen smooge@gmail.com:
… feel comfortable using. People in academia usually have tight capex budgets
++1
As an example, we have still to use as a server for production
[root@hydra ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz stepping : 6 ... cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm
The machine is an IBM blade center from shortly after 2010. It's not necessarily a joy, but it's not a particular limitation for mail and web.
In this case it doesn't 'matter' it is a small segment of users. It is a segment of our maintainers who are. We either have to listen to them, 'fire them', or buy them replacement hardware. Since we are already overloaded, firing them has not been on the table. Buying replacement hardware is expensive in multiple ways (time, capex, and various legal aspects). That leaves listening to the maintainers.
At the risk of overextending an already well-elaborated thread, I would like to point out that my main workstation, for Fedora packaging and other purposes, has an Intel Q6600 (Core 2 Quad) that does NOT meet the requirements for x86_64-v2. I built it in 2007, and it has exceeded all expectations for how long it would remain useful. The desktop I maintain for my parents uses an AMD Phenom II X4 965 processor, circa 2009-2010, and it doesn’t support x86_64-v2 either—but it just keeps on working.
Now, I can afford to replace my own workstation if I must—and I’m planning to do so in another year or two when the rolling component shortages settle out a little—but I suspect there are still many others like me, some of whom might not be in a position to just sigh and buy new hardware. Even for those who can, the pandemic and the crypto crazes have made it an exceptionally bad time to be forced into an upgrade.
On Wed, Jun 16, 2021 at 9:52 PM Benjamin Beasley code@musicinmybrain.net wrote:
At the risk of overextending an already well-elaborated thread, I would like to point out that my main workstation, for Fedora packaging and other purposes, has an Intel Q6600 (Core 2 Quad) that does NOT meet the requirements for x86_64-v2. I built it in 2007, and it has exceeded all expectations for how long it would remain useful. The desktop I maintain for my parents uses an AMD Phenom II X4 965 processor, circa 2009-2010, and it doesn’t support x86_64-v2 either—but it just keeps on working.
Now, I can afford to replace my own workstation if I must—and I’m planning to do so in another year or two when the rolling component shortages settle out a little—but I suspect there are still many others like me, some of whom might not be in a position to just sigh and buy new hardware. Even for those who can, the pandemic and the crypto crazes have made it an exceptionally bad time to be forced into an upgrade.
So ... maybe the following approach would be a way forward that would benefit everybody (TM):
1. stay with x86-64-baseline for Fedora for now (performance critical software often has runtime CPU feature detection and dynamic dispatch anyway) 2. identify "performance sensitive" libraries in Fedora that do not have runtime CPU feature detection, and which would benefit from having the instructions that are added with x86-64-v2 available (looks like the the performance benefit of enabling this overall is small (?), but maybe there are exceptions, where bumping from x86-64 to x86-64-v2 would make a bigger difference for some library) 3. make it easy to build the libraries identified under 2. twice (or three times, with x86-64-v3?) and install them in the locations where the loader can find them (leveraging the new HWCAPS functionality)
This approach would allow older machines to continue to run the latest Fedora just fine, while libraries that *would* benefit from more available CPU instructions would run faster for the people who have newer hardware.
Fabio
On Wed, Jun 16, 2021 at 11:02:23PM +0200, Fabio Valentini wrote:
On Wed, Jun 16, 2021 at 9:52 PM Benjamin Beasley code@musicinmybrain.net wrote:
At the risk of overextending an already well-elaborated thread, I would like to point out that my main workstation, for Fedora packaging and other purposes, has an Intel Q6600 (Core 2 Quad) that does NOT meet the requirements for x86_64-v2. I built it in 2007, and it has exceeded all expectations for how long it would remain useful. The desktop I maintain for my parents uses an AMD Phenom II X4 965 processor, circa 2009-2010, and it doesn’t support x86_64-v2 either—but it just keeps on working.
Now, I can afford to replace my own workstation if I must—and I’m planning to do so in another year or two when the rolling component shortages settle out a little—but I suspect there are still many others like me, some of whom might not be in a position to just sigh and buy new hardware. Even for those who can, the pandemic and the crypto crazes have made it an exceptionally bad time to be forced into an upgrade.
So ... maybe the following approach would be a way forward that would benefit everybody (TM):
- stay with x86-64-baseline for Fedora for now (performance critical
software often has runtime CPU feature detection and dynamic dispatch anyway) 2. identify "performance sensitive" libraries in Fedora that do not have runtime CPU feature detection, and which would benefit from having the instructions that are added with x86-64-v2 available (looks like the the performance benefit of enabling this overall is small (?), but maybe there are exceptions, where bumping from x86-64 to x86-64-v2 would make a bigger difference for some library) 3. make it easy to build the libraries identified under 2. twice (or three times, with x86-64-v3?) and install them in the locations where the loader can find them (leveraging the new HWCAPS functionality)
A good idea. But in that case it'd make sense to raise the bar quite a bit higher, and e.g. compile specifically for modern Intel CPUs and e.g. AMD Ryzen. If we don't have to make the baseline acceptable to everyone, we should make a big jump.
Zbyszek
This approach would allow older machines to continue to run the latest Fedora just fine, while libraries that *would* benefit from more available CPU instructions would run faster for the people who have newer hardware.
Hello,
I have already benchmarked this when Arch was considering a similar move:
https://openbenchmarking.org/result/2103142-HA-UARCHLEVE55
There is no or negligible performance benefit of compiling for x86_64-v2 versus amd64 baseline. For discussion see [1]. Beforehand, I have explicitly selected benchmarks that gave the largest performance difference when run with -O1 and -O3 compiler flags and were expected to show the biggest benefit from uarch optimizations.
I think that there is really no point in requiring x86_64-v2 as the compilers simply do not take much advantage from it.
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD.
AMD introduced SSE4.2 much later then Intel - starting with Jaguar in 2013-2014.
Greetings,
Mateusz
[1] https://lists.archlinux.org/pipermail/arch-general/2021-March/048739.html
Apologies in advance if this is laughable naïveté, but would a possible solution be to have a different repo for packages compiled against the latest-n-greatest architectures, and packagers could choose to include their packages in there, similar to EPEL? Packages in this hypothetical repo could be marked to replace existing ones via %obsoletes or some other mechanism; this would give the user the control to decide if he or she wants to switch to a “higher-performant” version and not have to rely on HW checks or telemetry.
Just a thought.
On 15 Jun 2021, at 16:34, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, 17 Jun 2021 at 11:38, Ron Olson tachoknight@gmail.com wrote:
Apologies in advance if this is laughable naïveté, but would a possible solution be to have a different repo for packages compiled against the latest-n-greatest architectures, and packagers could choose to include their packages in there, similar to EPEL? Packages in this hypothetical repo could be marked to replace existing ones via %obsoletes or some other mechanism; this would give the user the control to decide if he or she wants to switch to a “higher-performant” version and not have to rely on HW checks or telemetry.
The problem mainly comes from a) making the build system do this work b) spreading a limited number of build systems to do more work c) additional time to do composes of these extra repositories and deliverables d) dealing with the usual packager backlash of 'I don't want to deal with more bugs so I am exclusive arching my stuff out of that repo'
Just a thought.
On 15 Jun 2021, at 16:34, Neal Gompa wrote:
Hey all,
Earlier this week, I was helping with processing features for openSUSE Leap 15.4[1] and I discovered that they're planning on introducing x86_64-v2 to openSUSE soon. The reference for this change was that RHEL 9 is going to use x86_64-v2[2]. Additionally, other distributions have been considering bumping up to v2 or v3[3][4].
Some cursory examination of the new x86_64 sublevels seem to indicate that x86_64-v2 goes back to roughly 2007~2008, merely cutting off the first couple of generations of x86_64 CPUs from Intel and AMD. I personally don't have any computers that don't have support for x86_64-v2 anymore.
Does anyone know if anyone is planning to propose this for Fedora anytime soon, either as an addon architecture (like what Arch is doing) or an upgrade of our x86_64 baseline like RHEL is doing?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jun 17, 2021 at 10:36:47AM -0500, Ron Olson wrote:
Apologies in advance if this is laughable naïveté, but would a possible solution be to have a different repo for packages compiled against the latest-n-greatest architectures, and packagers could choose to include their packages in there, similar to EPEL? Packages in this hypothetical repo could be marked to replace existing ones via %obsoletes or some other mechanism; this would give the user the control to decide if he or she wants to switch to a “higher-performant” version and not have to rely on HW checks or telemetry.
Just a thought.
Sure, thats possible, however it would use a lot of resources. ;(
* builder cpu cycles and disk * mirror disk space * buildsystem disk space * mirroring BW etc.
and... the most important resource: everyone's time. If you have multiple versions of your package you have to ask everyone in bug reports which one they are using, you might not be able to duplicate due to lack of hardware, you have 2x as many things to worry about, etc.
So, while thats very possible, I fear we don't have the resources to do it. (Nor do I think the benifits are worth it currently).
kevin
Okay, So here comes another naive suggestion.
The metrics that are desired are really innocuous content which should not risk anonymity, correct? Such things like BIOS/UEFI/?, CPU, Memory, installed edition or spin, when it was installed, optional packages, storage setup, etc... I presume. Why not create install time json file with desired info, which is available info at the time, show the user the content and ask for permission to transmit the file encrypted to <fedora server of your choice>. Or even make it a non-fungable file maybe, which gets generated at update time too with additional info like package diff etc... Instead of a tool say that is continuously on.
Stephen Snow
On Thu, 2021-06-17 at 13:10 -0700, Kevin Fenzi wrote:
On Thu, Jun 17, 2021 at 10:36:47AM -0500, Ron Olson wrote:
Apologies in advance if this is laughable naïveté, but would a possible solution be to have a different repo for packages compiled against the latest-n-greatest architectures, and packagers could choose to include their packages in there, similar to EPEL? Packages in this hypothetical repo could be marked to replace existing ones via %obsoletes or some other mechanism; this would give the user the control to decide if he or she wants to switch to a “higher-performant” version and not have to rely on HW checks or telemetry.
Just a thought.
Sure, thats possible, however it would use a lot of resources. ;(
- builder cpu cycles and disk
- mirror disk space
- buildsystem disk space
- mirroring BW
etc.
and... the most important resource: everyone's time. If you have multiple versions of your package you have to ask everyone in bug reports which one they are using, you might not be able to duplicate due to lack of hardware, you have 2x as many things to worry about, etc.
So, while thats very possible, I fear we don't have the resources to do it. (Nor do I think the benifits are worth it currently).
kevin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Support glibc-hwcaps in rpm https://github.com/rpm-software-management/rpm/issues/1812
There seems to be support now in rpm: https://github.com/rpm-software-management/rpm/discussions/2022
Dne 01. 02. 23 v 23:25 Reon Beon via devel napsal(a):
There seems to be support now in rpm:https://github.com/rpm-software-management/rpm/discussions/2022
Where? When following the link, it leads to the hole of anothers issues, and refences. But I see no merged PR.
Miroslav
Dne 02. 02. 23 v 17:20 Miroslav Suchý napsal(a):
Dne 01. 02. 23 v 23:25 Reon Beon via devel napsal(a):
There seems to be support now in rpm:https://github.com/rpm-software-management/rpm/discussions/2022
Where? When following the link, it leads to the hole of anothers issues, and refences. But I see no merged PR.
Probably this:
https://github.com/rpm-software-management/rpm/pull/2315#issuecomment-137530...
Vít
-
Benjamin Beasley -
Björn Persson -
Chris Murphy -
Dan Čermák -
Daniel P. Berrangé -
Eugene Syromiatnikov -
Fabio Valentini -
Florian Weimer -
Frantisek Zatloukal -
Gary Buhrmaster -
Gerd Hoffmann -
Hans de Goede -
Justin Forbes -
Kevin Fenzi -
Kevin Kofler -
Mark Otaris -
Mateusz Jończyk -
Matthew Miller -
Miroslav Suchý -
Neal Gompa -
Peter Boy -
PGNet Dev -
przemek klosowski -
Reon Beon -
Ron Olson -
Solomon Peachy -
Stephen John Smoogen -
Stephen Snow -
Tom Hughes -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek