Still another option would be to build a Big Data Spin.
That corresponds to the SCL (Software Collection) suggestion, though more complete.
The Big Data Spin would allow to deploy stand-alone as well as full clusters.
Note that the Big Data Spin would be very similar to the classical Hadoop distributions
(Cloudera, Hortonworks).
Hence, by the way, I am not sure Fedora would add much value here... I mean, why a user
would use a Fedora Big Data Spin rather than, say a CDH (Cloudera Hadoop) image?
So, not so sure it is a good idea, then...
----
Properly packaging Big Data software is something upstream developers should want more, as
not doing so costs a lot in maintenance. And I am sure that upstream (most, like
Databricks, Cloudera, or even Data Artisans, are commercial companies, which care about
development costs) would rather develop new features than maintain patches on old versions
of their bundled libraries.
By the way, that practice in Apache Spark causes headaches to a lot of users (search, for
instance, for "Spark NoSuchFieldError"), even experienced ones, as runtime
errors pop up out of nowhere, and debugging them is quite difficult. The dependency graph
of a typical Spark-based application lists dozens of libraries with not always compatible
versions, most being duplicated, part of them being bundled and patched.
Fixing that kind of headache should not be the work of a user. To my belief, that is
exactly what Linux distributions are done for: ship an ecosystem of components, for which
the versions are known to work well together.