Reading the whole thread, I now get a better view of the pros and cons, and of the
suggested solutions (or, rather, work-arounds).
You surely know that Cloudera ships their own Hadoop distribution, which is based on a
RedHat/CentOS Linux distribution(!), and which is made of exactly the same components that
we intend to ship with Fedora, including Apache Spark.
So, why couldn't we just use the (surely hard) work Cloudera is doing to build their
distribution?
I have not looked at the details, but from what I understand, their distribution of Hadoop
(and Spark, and Impala) relies only on open source components. They have proprietary
software, for instance the Cloudera Manager (used to configure, deploy and administer a
cluster; but it can be replaced by Apache Ambary, which is open source).
Note that there are other Hadoop distributions, such as Hortonworks (I believe those
latter even ship only 100% open source components).
I do not mention MapR, as their stack is mainly based on proprietary software.
Joining our forces with Cloudera (and/or Hortonworks) would help all of us in front of the
upstream developers. For sure, Cloudera and Hortonworks face the same issues as us when
packaging the components and when dealing with the Hadoop upstream eco-system.
Just food for thoughts.
Kind regards
Denis