mattdm suggested the upcoming scheduler change should be discussed here. I might not have enough time to talk details at the moment, but I noticed this is coming up relatively soon. This is what I understand so far about the decision.
Thanks for all the software,
## CFQ is scheduled for removal
Jens Axboe is planning to remove the CFQ I/O scheduler in 4.21. That is, CFQ is part of the "legacy" single-queue block layer, which is going to be removed for ease of maintenance.
Axboes last comment on this timing was made *after* the fix for data corruption on MQ. I.e. the data corruption covered by the recent thread on this list.
*Obligatory disclaimer*. Read the paragraph above, and consider waiting for the next stable kernel update, before you test MQ (including BFQ) on your own machine :-).
 "It's definitely still going" - Jens Abxoe. https://bugzilla.kernel.org/show_bug.cgi?id=201685#c279
## The kernel wants us to choose our new default
For devices which have only one hardware queue, the new upstream default is mq-deadline. Going from CFQ to mq-deadline is a significant change. For example, the deadline scheduler does not support ionice.
The alternative to the MQ deadline scheduler will be BFQ. Upstream discussed this, and the powers that be (mostly Axboe :-) are explicitly expecting us (downstreams) to make this decision.
 BFQ: http://algo.ing.unimo.it/people/paolo/disk_sched/description.php
 "I'd prefer if the distros would lead the way on this, as theyare the ones that will most likely see the most bug reports" - Jens Axboe, https://www.spinics.net/lists/linux-block/msg31062.html
## Arguments for (or against) BFQ?
Paolo Valente kindly wrote an informative response, which I will copy in to a separate message. The following is my very limited first impression.
Personally I lean towards BFQ by default. It appears nominated as the successor to CFQ, I think it's worthy as such, and it makes distinct improvements of it's own. I would recommend it with more confidence if I understood how the improvements work :-).
The deadline scheduler probably isn't a *complete* disaster. Ubuntu ran with the deadline scheduler for a while. I haven't checked whether they changed the tuning knobs though!
RHEL7 defaults to CFQ for SATA drives. This is notable given that it recommends avoiding (or tuning) CFQ on basically any other server hardware (and specifically to avoid it on hardware RAID).
I've tried BFQ on my laptops hard drive (not SSD). It has some associated tests for responsiveness (the "S" tests). I don't have a real-world feel for it, but I agree the test numbers are an impressive improvement over CFQ.
I don't have results for the S tests on the deadline scheduler. I did note the eponymous "deadline" for sync reads has a default of 500 ms.
The other test I have, is that "deadline" doesn't match CFQ's level of fairness for reads v.s. writes, even with the recent addition of WBT. Neither approached what I would actually call fairness. BFQ did. This is due to BFQ's "compensations" for device writeback caching and NCQ. And allegedly for Linux writeback. These extra compensations are the part I don't understand, so far.
 RHEL7 IO schedulers:
 I tried to closely match the test from the cover letters from the WBT patch series. I don't have detailed statistics, but I believe deadline+WBT was less fair than CFQ. It was definitely not more fair.
As you probably have seen, kernel 5.0 was released last night. This
has been built in rawhide. We'll be following the usual rebase
procedure for f29 and f28: Around the time when 5.0.2 or 5.0.3
is released, that kernel will go into f29. f28 will be rebased
shortly afterwards. We don't have any control over when upstream
will release a stable kernel but best guess would be that f29
will get 5.0 by early to mid April.
f30 is building 5.0 right now and will be getting the usual
stable updates as they come out. I'll also be doing a rebase
on the stabilization branch and will be doing builds in the
kernel-stabilization copr. If you want early access to 5.0,
I recommend just using the f30 builds (if you find bugs,
please report them so we can get them fixed)
P.S. If you are using F28, please make sure to keep giving
those kernels karma in bodhi. We're seeing a fall off in