Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
See the details in https://bugzilla.redhat.com/show_bug.cgi?id=2025716.
Just a question, do we have any mechanism to migrate users from sshd.socket to sshd.service? Otherwise someone may suddenly lose access to their headless systems after an upgrade.
在 2023/8/3 10:29, Dmitry Belyavskiy 写道:
Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
See the details in https://bugzilla.redhat.com/show_bug.cgi?id=2025716.
On Thu, Aug 03, 2023 at 11:29:03AM +0200, Dmitry Belyavskiy wrote:
Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
See the details in https://bugzilla.redhat.com/show_bug.cgi?id=2025716.
The DoS attack is described here:
https://bugs.archlinux.org/task/62248
... and it sounds like a bug in systemd. Surely this same attack applies to any socket-activated service so should be fixed in systemd? I don't recall inetd having the same problem.
Rich.
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
The DoS attack is described here:
https://bugs.archlinux.org/task/62248
... and it sounds like a bug in systemd. Surely this same attack applies to any socket-activated service so should be fixed in systemd? I don't recall inetd having the same problem.
(x)inetd would shut a port under heavy net-connection load for a short period, but systemd seems to shut it permanently under those conditions. For systemd to replace inetd-type socket activation, it needs to have a timeout on the disable.
This probably isn't a high priority though, because very few things support inetd-type modes anymore.
On Friday, August 4, 2023 8:42:18 AM EDT Chris Adams wrote:
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
The DoS attack is described here:
https://bugs.archlinux.org/task/62248
... and it sounds like a bug in systemd. Surely this same attack applies to any socket-activated service so should be fixed in systemd? I don't recall inetd having the same problem.
(x)inetd would shut a port under heavy net-connection load for a short period, but systemd seems to shut it permanently under those conditions. For systemd to replace inetd-type socket activation, it needs to have a timeout on the disable.
Yes, as one of the authors of xinetd, I pointed this out long ago. But they said they were not trying to replace xinetd and if people want a more full featured experience, use xinetd.
This probably isn't a high priority though, because very few things support inetd-type modes anymore.
This would be a problem for MLS systems. The way the role/level/category is negotiated between systems is with VPN keys which maps to SE Linux policy. Once the key is negotiated, it connects via the socket API where the sshd instances is started with the right SE Linux labels. This is a small but important use case.
I suppose the fallback would be to go back to using xinetd if this is not fixed in systemd.
-Steve
Once upon a time, Steve Grubb sgrubb@redhat.com said:
Yes, as one of the authors of xinetd, I pointed this out long ago. But they said they were not trying to replace xinetd and if people want a more full featured experience, use xinetd.
Except... wasn't there a big push to replace xinetd with systemd socket activation in Fedora? In fact, xinetd was orphaned and retired, so the only way to have that kind of functionality in Fedora is through systemd, so systemd needs to support the same functionality.
On Do, 03.08.23 11:29, Dmitry Belyavskiy (dbelyavs@redhat.com) wrote:
Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
Hmm, why drop it? For many setups, it makes not sense to continously run sshd, so socket activation should be fine.
I don't understand the reasoning behind this change. You claim a DoS. Which DoS is that supposed to be? That we enforce a trigger time limit on socket units by default? If you don't want this, turn it off, that's what TriggerLimitIntervalSec=/TriggerLimitBurst= is for, see docs.
The discussion makes this sound as if there was a bug in systemd or so, but there really isn't, it's literally a safety feature you ran into. It might not make sense to have the trigger rate limit in place for all usecases, ssh might be one where it is not advisable, but then the right approach is to just turn that part off, as documented, via the aforementioned options.
See for details:
https://www.freedesktop.org/software/systemd/man/systemd.socket.html#Trigger...
I don't care too much whether you make ssh socket-activated by default or not. But at least the option should exist, already for compat with existing setups.
Lennart
-- Lennart Poettering, Berlin
Dear Lennart,
I'm sorry, I don't get.
Quoting the https://www.freedesktop.org/software/systemd/man/systemd.socket.html#Trigger...
Configures a limit on how often this socket unit may be activated within a specific time interval. The TriggerLimitIntervalSec= may be used to configure the length of the time interval in the usual time units "us", "ms", "s", "min", "h", … and defaults to 2s (See systemd.time(7) for details on the various time units understood). The TriggerLimitBurst= setting takes a positive integer value and specifies the number of permitted activations per time interval, and defaults to 200 for Accept=yes sockets (thus by default permitting 200 activations per 2s), and 20 otherwise (20 activations per 2s). Set either to 0 to disable any form of trigger rate limiting. If the limit is hit, the socket unit is placed into a failure mode, and will not be connectible anymore until restarted. Note that this limit is enforced before the service activation is enqueued.
But this behavior (the last sentence) exactly matches the DoS described here: https://bugs.archlinux.org/task/62248 Too many connections to an sshd server, configured using socket activation can cause the socket to be disabled permanently ("sshd.socket: Trigger limit hit, refusing further activation.").
On Mon, Aug 7, 2023 at 11:48 AM Lennart Poettering mzerqung@0pointer.de wrote:
On Do, 03.08.23 11:29, Dmitry Belyavskiy (dbelyavs@redhat.com) wrote:
Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
Hmm, why drop it? For many setups, it makes not sense to continously run sshd, so socket activation should be fine.
I don't understand the reasoning behind this change. You claim a DoS. Which DoS is that supposed to be? That we enforce a trigger time limit on socket units by default? If you don't want this, turn it off, that's what TriggerLimitIntervalSec=/TriggerLimitBurst= is for, see docs.
The discussion makes this sound as if there was a bug in systemd or so, but there really isn't, it's literally a safety feature you ran into. It might not make sense to have the trigger rate limit in place for all usecases, ssh might be one where it is not advisable, but then the right approach is to just turn that part off, as documented, via the aforementioned options.
See for details:
https://www.freedesktop.org/software/systemd/man/systemd.socket.html#Trigger...
I don't care too much whether you make ssh socket-activated by default or not. But at least the option should exist, already for compat with existing setups.
Lennart
-- Lennart Poettering, Berlin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Di, 15.08.23 18:17, Dmitry Belyavskiy (dbelyavs@redhat.com) wrote:
Dear Lennart,
I'm sorry, I don't get.
Quoting the https://www.freedesktop.org/software/systemd/man/systemd.socket.html#Trigger...
Configures a limit on how often this socket unit may be activated within a specific time interval. The TriggerLimitIntervalSec= may be used to configure the length of the time interval in the usual time units "us", "ms", "s", "min", "h", … and defaults to 2s (See systemd.time(7) for details on the various time units understood). The TriggerLimitBurst= setting takes a positive integer value and specifies the number of permitted activations per time interval, and defaults to 200 for Accept=yes sockets (thus by default permitting 200 activations per 2s), and 20 otherwise (20 activations per 2s). Set either to 0 to disable any form of trigger rate limiting. If the limit is hit, the socket unit is placed into a failure mode, and will not be connectible anymore until restarted. Note that this limit is enforced before the service activation is enqueued.
But this behavior (the last sentence) exactly matches the DoS described here: https://bugs.archlinux.org/task/62248 Too many connections to an sshd server, configured using socket activation can cause the socket to be disabled permanently ("sshd.socket: Trigger limit hit, refusing further activation.").
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
Lennart
-- Lennart Poettering, Berlin
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
On Thu, Aug 17, 2023 at 08:25:10AM -0500, Chris Adams wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
Very recent systemd versions have exponentially-growing restart intervals. I guess you can use very large limit for number of restart, and use RestartSteps=/RestartSecMax= to make "cooldown periods".
Am 17.08.23 um 20:14 schrieb Tomasz Torcz:
On Thu, Aug 17, 2023 at 08:25:10AM -0500, Chris Adams wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
Very recent systemd versions have exponentially-growing restart intervals. I guess you can use very large limit for number of restart, and use RestartSteps=/RestartSecMax= to make "cooldown periods".
for systemd.sockets ?
On Do, 17.08.23 08:25, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
There's a TODO list item for that upstream already.
https://github.com/systemd/systemd/blob/main/TODO#L153
Definitely makes sense, and should be very easy to add, the underlying concepts are all implemented, it's just a matter of exposing this under new options.
Lennart
-- Lennart Poettering, Berlin
Sounds good!
Leslie Satenstein
On Monday, August 21, 2023 at 05:08:06 a.m. GMT-4, Lennart Poettering mzerqung@0pointer.de wrote:
On Do, 17.08.23 08:25, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
There's a TODO list item for that upstream already.
https://github.com/systemd/systemd/blob/main/TODO#L153
Definitely makes sense, and should be very easy to add, the underlying concepts are all implemented, it's just a matter of exposing this under new options.
Lennart
-- Lennart Poettering, Berlin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Mo, 21.08.23 11:07, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Do, 17.08.23 08:25, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Yes, and if this is not what you want, then disable the ratelimit. That's what I am saying?
It would be useful for systemd to have "cooldown periods" for things, similar to inetd and classic init, where misbehaving things (whether services or sockets) were paused for a time (configurable even) and then returned to service. AFAIK this is a flaw in general in systemd's limits; if something exceeds them, it takes manual intervention to reset them.
There's a TODO list item for that upstream already.
https://github.com/systemd/systemd/blob/main/TODO#L153
Definitely makes sense, and should be very easy to add, the underlying concepts are all implemented, it's just a matter of exposing this under new options.
I started working on this now:
https://github.com/systemd/systemd/pull/29159
Lennart
-- Lennart Poettering, Berlin
Wouldn't a relative simple change to fix this would be explicit TriggerLimitBurst=0 until some form of timed reactivation is implemented? Especially for sshd.socket that change would seem safer. It is not a big deal for sshd, it seems to be quite small anyway.
Could simple
[Unit] Restart=on-failure
solve this issue instead? If it fails, this should be able to keep it up. Does it manifest itself a different way?
On 03. 08. 23 11:29, Dmitry Belyavskiy wrote:
Dear colleagues,
I've pushed a fresh build of OpenSSH to rawhide. We decided to drop the sshd.socket unit from rawhide. We don't think it's worth going through the changes process, but would like to provide a heads-up.
See the details in https://bugzilla.redhat.com/show_bug.cgi?id=2025716.