Hello all,
our RAID storage got stuck this night (CET). Likely some deadlock caused by
high I/O - caused by the periodic "RAID check" process together with the Fedora
Rawhide => F38 branching task that has been done yesterday. But the actual
cause is unknown (if this looks familiar to you, let us know).
We ended up with about 400 processes hanged, waiting for disk (kill -9 has
no effect). This is the second time this happened on F37 but "echo idle >
/sys/block/md127/md/sync_action" did not help now (nor 'frozen'), only
hard reboot using the AWS console helped eventually.
We were more than 7 hours offline (the team was sleeping):
https://pagure.io/fedora-infrastructure/issue/11120
Sorry for the inconvenience, and thank you for all the reports!
Pavel