Request for testers: glibc update to work around Intel TSX errata microcode_ctl problems.
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
https://admin.fedoraproject.org/updates/glibc-2.20-4.fc21
For Fedora 21 and Fedora 20 we will be disabling Intel TSX support in POSIX threads effective immediately. This disabling of TSX support in POSIX threads is in order to prevent users from crashing their systems when they upgrade microcode_ctl.
The upgrade to the new microcode_ctl applies the microcode update for the Intel errata HSW136 and makes all TSX instructions invalid opcodes. This means that any process on your system already using POSIX threads, and already using TSX, will fault with invalid opcode errors when it tries to execute TSX instructions.
The best solution is for the microcode update to be done before userspace starts as part of a single-threaded init process or something similar.
Until we support updating microcode in early bootup we won't enable TSX in POSIX threads. The glibc maintainers have decided not add code to detect which devices the errata applies to, nor will code be added to re-read the cpuid values at every TSX usage. We are going with the simples approach which is to disable TSX for POSIX threads.
Cheers, Carlos.
Hi All,
On 09/26/2014 08:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
https://admin.fedoraproject.org/updates/glibc-2.20-4.fc21
For Fedora 21 and Fedora 20 we will be disabling Intel TSX support in POSIX threads effective immediately. This disabling of TSX support in POSIX threads is in order to prevent users from crashing their systems when they upgrade microcode_ctl.
The upgrade to the new microcode_ctl applies the microcode update for the Intel errata HSW136 and makes all TSX instructions invalid opcodes. This means that any process on your system already using POSIX threads, and already using TSX, will fault with invalid opcode errors when it tries to execute TSX instructions.
The best solution is for the microcode update to be done before userspace starts as part of a single-threaded init process or something similar.
Until we support updating microcode in early bootup we won't enable TSX in POSIX threads. The glibc maintainers have decided not add code to detect which devices the errata applies to, nor will code be added to re-read the cpuid values at every TSX usage. We are going with the simples approach which is to disable TSX for POSIX threads.
I've just tested the F-21 update for this, and I'm afraid that the problem is still present there. I've also regenerated my initrd to make sure that that included the new glibc too, but that did not help.
Regards,
Hans
Cheers, Carlos.
On Fri, Sep 26, 2014 at 2:53 PM, Hans de Goede hdegoede@redhat.com wrote:
Hi All,
On 09/26/2014 08:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
https://admin.fedoraproject.org/updates/glibc-2.20-4.fc21
For Fedora 21 and Fedora 20 we will be disabling Intel TSX support in POSIX threads effective immediately. This disabling of TSX support in POSIX threads is in order to prevent users from crashing their systems when they upgrade microcode_ctl.
The upgrade to the new microcode_ctl applies the microcode update for the Intel errata HSW136 and makes all TSX instructions invalid opcodes. This means that any process on your system already using POSIX threads, and already using TSX, will fault with invalid opcode errors when it tries to execute TSX instructions.
The best solution is for the microcode update to be done before userspace starts as part of a single-threaded init process or something similar.
Until we support updating microcode in early bootup we won't enable TSX in POSIX threads. The glibc maintainers have decided not add code to detect which devices the errata applies to, nor will code be added to re-read the cpuid values at every TSX usage. We are going with the simples approach which is to disable TSX for POSIX threads.
I've just tested the F-21 update for this, and I'm afraid that the problem is still present there. I've also regenerated my initrd to make sure that that included the new glibc too, but that did not help.
Test this instead:
https://admin.fedoraproject.org/updates/dracut-038-29.git20140903.fc21,kerne...
josh
On Fri, Sep 26, 2014 at 08:53:59PM +0200, Hans de Goede wrote:
I've just tested the F-21 update for this, and I'm afraid that the problem is still present there. I've also regenerated my initrd to make sure that that included the new glibc too, but that did not help.
Do you still see crashes in systemd within pthread code or is it something else? The update should have disabled lock elision completely, so if it SIGILLs on haswell, it ought to SIGILL on everything, which it doesn't.
Either way, an early microcode update is indeed the best way out, so if that works, we could just drop the glibc update. But it would be great to know what's actually going wrong so that if it's something in glibc even with elision disabled, we can get that fixed.
Siddhesh
Hi,
On 09/26/2014 09:26 PM, Siddhesh Poyarekar wrote:
On Fri, Sep 26, 2014 at 08:53:59PM +0200, Hans de Goede wrote:
I've just tested the F-21 update for this, and I'm afraid that the problem is still present there. I've also regenerated my initrd to make sure that that included the new glibc too, but that did not help.
Do you still see crashes in systemd within pthread code or is it something else? The update should have disabled lock elision completely, so if it SIGILLs on haswell, it ought to SIGILL on everything, which it doesn't.
I'm still seeing SIGILL in systemd pthread usage. The problem is that the glibc update still has a code path which uses TSX, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1146967#c14
There is a scratch build here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=7707772
Which fixes the last TSX usage in glibc and I can confirm that that really fixes things.
Either way, an early microcode update is indeed the best way out, so if that works, we could just drop the glibc update. But it would be great to know what's actually going wrong so that if it's something in glibc even with elision disabled, we can get that fixed.
Agreed.
Regards,
Hans
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
I unpushed the -4 update, and pushed the -5 update, and it's ready for testing.
https://admin.fedoraproject.org/updates/FEDORA-2014-11673/glibc-2.20-5.fc21
Cheers, Carlos.
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
Cheers, Carlos.
Am 28.09.2014 um 20:13 schrieb Carlos O'Donell:
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
then he needs to build it right
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
On Sun, Sep 28, 2014 at 8:17 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 28.09.2014 um 20:13 schrieb Carlos O'Donell:
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
then he needs to build it right
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
Well the microcode update *does* disable TSX (so that only applies to new yet to be introduced cpus) ... but yeah if you build your own kernel you should try to be as close to possible to the distro config.
Am 28.09.2014 um 21:15 schrieb drago01:
On Sun, Sep 28, 2014 at 8:17 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 28.09.2014 um 20:13 schrieb Carlos O'Donell:
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote:
Developers.
Testers wanted immediately for:
https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20
and
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
then he needs to build it right
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
Well the microcode update *does* disable TSX (so that only applies to new yet to be introduced cpus) ...
that's the point of what i said:
* you buy a new CPU in 2 months * the microcode don't disable TSX there * if glibc now is built without TSX support you gain nothing from new hardware * if kernel loads microcode early and you have hardware supporting TSX and glibc also supports it -> fine you hav ethe feature
it would be a big mistake to disable completly TSX forever and if not forever how would someone decide when enable it - the only sane way is to get microcode applid as early as possible and after that use the CPU feautures which are enabled
but yeah if you build your own kernel you should try to be as close to possible to the distro config
exactly what i said
On Sep 28, 2014 12:25 PM, "Reindl Harald" h.reindl@thelounge.net wrote:
Am 28.09.2014 um 21:15 schrieb drago01:
On Sun, Sep 28, 2014 at 8:17 PM, Reindl Harald h.reindl@thelounge.net
wrote:
Am 28.09.2014 um 20:13 schrieb Carlos O'Donell:
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell carlos@redhat.com
wrote:
On 09/26/2014 02:34 PM, Carlos O'Donell wrote: > Developers. > > Testers wanted immediately for: > > https://admin.fedoraproject.org/updates/glibc-2.18-16.fc20 > > and > > https://admin.fedoraproject.org/updates/glibc-2.20-4.fc21
For FC21 was missed a case where TSX was being used and had to push one more update -5 to cover this case.
Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
then he needs to build it right
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
Well the microcode update *does* disable TSX (so that only applies to new yet to be introduced cpus) ...
that's the point of what i said:
- you buy a new CPU in 2 months
- the microcode don't disable TSX there
What hypothetical CPU is this? I don't think Broadwell has TSX, so I think we're talking about Skylake here.
- if glibc now is built without TSX support you gain nothing from new
hardware
- if kernel loads microcode early and you have hardware supporting TSX and glibc also supports it -> fine you hav ethe feature
it would be a big mistake to disable completly TSX forever and if not forever how would someone decide when enable it - the only sane way is to get microcode applid as early as possible and after that use the CPU feautures which are enabled
but yeah if you build your own kernel you should try to be as close to possible to the distro config
exactly what i said
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Am 29.09.2014 um 00:21 schrieb Andrew Lutomirski:
On Sep 28, 2014 12:25 PM, "Reindl Harald" <h.reindl@thelounge.net mailto:h.reindl@thelounge.net> wrote:
Am 28.09.2014 um 21:15 schrieb drago01:
On Sun, Sep 28, 2014 at 8:17 PM, Reindl Harald <h.reindl@thelounge.net mailto:h.reindl@thelounge.net> wrote:
Am 28.09.2014 um 20:13 schrieb Carlos O'Donell:
On 09/28/2014 01:24 PM, drago01 wrote:
On Sun, Sep 28, 2014 at 6:49 PM, Carlos O'Donell <carlos@redhat.com mailto:carlos@redhat.com> wrote: Does that update still make sense given that the kernel / dracut update enables early microcode loading?
What about the case where the user runs a custom kernel?
then he needs to build it right
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
Well the microcode update *does* disable TSX (so that only applies to new yet to be introduced cpus) ...
that's the point of what i said:
- you buy a new CPU in 2 months
- the microcode don't disable TSX there
What hypothetical CPU is this? I don't think Broadwell has TSX
TSX was introduced with Haswell otherwise the thread and the microcode to disable it now would not exist
http://www.anandtech.com/show/8376/intel-disables-tsx-instructions-erratum-f...
so I think we're talking about Skylake here
see above - who other than Intel knows if the next Haswell/Broadwell becomes a new "Step" with fixes and enables TSX again but only for the newer ones
however - how does that matter?
you can either build all without TSX forever or load microcode early enough that it is masked controlled by the microcode before it is used while apply the microcode later disables the instructions which is the current topic
- if glibc now is built without TSX support you gain nothing from new hardware
- if kernel loads microcode early and you have hardware supporting TSX and glibc also supports it -> fine you hav ethe feature
it would be a big mistake to disable completly TSX forever and if not forever how would someone decide when enable it - the only sane way is to get microcode applid as early as possible and after that use the CPU feautures which are enabled
but yeah if you build your own kernel you should try to be as close to possible to the distro config
exactly what i said
On 09/28/2014 09:25 PM, Reindl Harald wrote:
- you buy a new CPU in 2 months
- the microcode don't disable TSX there
- if glibc now is built without TSX support you gain nothing from new hardware
Or the fixed TSX implementation is slightly different, and glibc breaks again. We don't know. Maybe even Intel doesn't know.
On Mon, Sep 29, 2014 at 06:39:01AM +0200, Florian Weimer wrote:
On 09/28/2014 09:25 PM, Reindl Harald wrote:
- you buy a new CPU in 2 months
- the microcode don't disable TSX there
- if glibc now is built without TSX support you gain nothing from new hardware
Or the fixed TSX implementation is slightly different, and glibc breaks again. We don't know. Maybe even Intel doesn't know.
Furthermore, it is very unlikely to be 2 months, from the info provided/leaked so far, it seems that Broadwell-E/Broadwell-EP are the first CPUs to have fixed TSX again, and Haswell-E has only been released a month ago. Note, Haswell-E usually seems to have TSX disabled already in BIOS, so microcode_ctl doesn't do anything regarding TSX there.
Jakub
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 09/28/2014 02:17 PM, Reindl Harald wrote:
What about the case where the user runs a custom kernel?
then he needs to build it right
That's not sufficiently conservative for a core runtime.
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
I am not disabling TSX completely. I am disabling the use of TSX in POSIX threads (particularly rwlocks and default mutex locks). I would also not disable it forever, I would disable it until we have more discussion and consensus around the best solution.
At the present moment I'm reluctant to commit to hardware blacklists in glibc, but this might be one way forward.
Alternately we forgo the use of cpuid and rely on the kernel to always tell us which features are present and useful.
Cheers, Carlos.
Am 29.09.2014 um 15:14 schrieb Carlos O'Donell:
On 09/28/2014 02:17 PM, Reindl Harald wrote:
What about the case where the user runs a custom kernel?
then he needs to build it right
That's not sufficiently conservative for a core runtime.
from the viewpoint of a distribution in context of the kernel?
well, try to write a kernel bugreport if you are using a custom built one on the RH bugzilla....
don't get me wrong but you can't seriously disable TSX completly because a *possible* out-of-distribution kernel
I am not disabling TSX completely. I am disabling the use of TSX in POSIX threads (particularly rwlocks and default mutex locks)
in context of the issue of the topic you need to disable it completly without early mircocode loading because the whole CPU capability is gone after it is applied
I would also not disable it forever, I would disable it until we have more discussion and consensus around the best solution.
agreed - that's what i liked to hear confirmed to not have it disabled in 2 years because there maybe old and affected hardware "in the wild"
At the present moment I'm reluctant to commit to hardware blacklists in glibc, but this might be one way forward.
Alternately we forgo the use of cpuid and rely on the kernel to always tell us which features are present and useful
* how do you do that? * from where is cpuid coming - IMHO it's from the kernel (/proc/cpuinfo)
the issue is the time where microcode get updated due boot to not break already loaded code if a capability is gone like TSX after that
On Mon, Sep 29, 2014 at 3:27 PM, Reindl Harald h.reindl@thelounge.net wrote:
[...]
- how do you do that?
- from where is cpuid coming - IMHO it's from the kernel (/proc/cpuinfo)
cpuid is a x86 instruction that allows you to query information about the cpu (name, model, ...) and its capabilities. It has nothing to do with the kernel.
-
Andrew Lutomirski -
Carlos O'Donell -
drago01 -
Florian Weimer -
Hans de Goede -
Jakub Jelinek -
Josh Boyer -
Reindl Harald -
Siddhesh Poyarekar