On 1/10/16 6:16 PM, Denis Arnaud wrote:
Reading the whole thread, I now get a better view of the pros and
cons, and of the suggested solutions (or, rather, work-arounds).
You surely know that Cloudera ships their own Hadoop distribution, which is based on a
RedHat/CentOS Linux distribution(!), and which is made of exactly the same components that
we intend to ship with Fedora, including Apache Spark.
So, why couldn't we just use the (surely hard) work Cloudera is doing to build their
distribution?
I have not looked at the details, but from what I understand, their distribution of
Hadoop (and Spark, and Impala) relies only on open source components. They have
proprietary software, for instance the Cloudera Manager (used to configure, deploy and
administer a cluster; but it can be replaced by Apache Ambary, which is open source).
Pretty sure they assemble their RPMs directly from the jar artifacts
directly derived from a source+patches maven build, without mapping on
to officially packaged "system" jars as we have done in the past for
Hadoop and other related Big Data projects in Fedora. Those RPMs violate
existing Fedora packaging rules. That is an essential part of this debate.
Note that there are other Hadoop distributions, such as Hortonworks (I believe those
latter even ship only 100% open source components).
I do not mention MapR, as their stack is mainly based on proprietary software.
Joining our forces with Cloudera (and/or Hortonworks) would help all of us in front of
the upstream developers. For sure, Cloudera and Hortonworks face the same issues as us
when packaging the components and when dealing with the Hadoop upstream eco-system.
Just food for thoughts.
Kind regards
Denis
_______________________________________________
bigdata mailing list
bigdata(a)lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/bigdata@lists.fedoraproject.org