adding a bit .. been using Hortonworks Hadoop for a while now .. yes they bundle everything into /usr/hadoop and the packages are not compatible with fedora packaging guideline .. but rather packaged in a way that Ambari can handle them easily ..

On Tue, Jan 12, 2016 at 7:50 AM, Gerald Henriksen <ghenriks@gmail.com> wrote:
On Mon, 11 Jan 2016 18:13:53 -0000, you wrote:

>Properly packaging Big Data software is something upstream developers should want more, as not doing so costs a lot in maintenance. And I am sure that upstream (most, like Databricks, Cloudera, or even Data Artisans, are commercial companies, which care about development costs) would rather develop new features than maintain patches on old versions of their bundled libraries.

The problem is that there is also a development cost to upgrading to
newer libraries, both in the work to update the code base and then the
resulting testing.

Despite efforts by Fedora packagers (as well as others on occasion on
the Hadoop mailing lists) the upstream developers have no desire to
spend the time and effort to update libraries until they are forced
to.

For that matter, the only fully approved version of Java for Hadoop is
the official Oracle Java 7, which is now 9 month beyond any publicly
available updates for security issues.  They do not support OpenJDK or
Java 8.

In short, as previously discussed, Hadoop is like so much of the
software developed in the last decade where the need to constantly
update with the language and/or libraries is viewed as a bug and
alternative methods have been developed to allow the project to exist
in its own little bubble - in the case of Hadoop its Maven but even
the recently open sourced Swift is developing its own package system
just like Python, Perl, Go, etc.

As long as they can stay in that bubble there is little to no
development cost to not updating the code base.

As long as that continues Hadoop is impossible to package properly in
rpm form for Fedora.

[obviously one can wrap up the jar files into an rpm, as some
commercial Hadoop companies do.  But they could never be in Fedora
proper, and if you are going to go through the hassle to wrap a jar
into an rpm you may as well move with the present and do a Docker
container instead]

>By the way, that practice in Apache Spark causes headaches to a lot of users (search, for instance, for "Spark NoSuchFieldError"), even experienced ones, as runtime errors pop up out of nowhere, and debugging them is quite difficult. The dependency graph of a typical Spark-based application lists dozens of libraries with not always compatible versions, most being duplicated, part of them being bundled and patched.

I have no experience with Spark, so I cannot comment on your example
other than to say fixing that sort of issue relies on buy in from
upstream to keep updating the code base, and upstream may not be
willing.  The secondary issue is that even if the Spark (or any other
Hadoop related project) developers want to do things correctly they
are still stuck with the decisions made by Hadoop.

>Fixing that kind of headache should not be the work of a user. To my belief, that is exactly what Linux distributions are done for: ship an ecosystem of components, for which the versions are known to work well together.

Two things:

1) the distributions simpy don't have the available manpower to do it.
At this point there are so many different projects with different
library needs it would take years by a lot of programmers/packagers
just to get everybody up do date, and with too many upstreams
unwilling to update their codebases with the changes it would remain
an ongoing problem.

2) the users don't care - as posted by one of the Red Hat(?) members
of the list the users don't trust a properly packaged Hadoop.

Part of the problem is the Java is too much of a nightmare to package,
it results in far too many rpm files that to a user it looks too
complicated compared to downloading 1 jar file.

But in the end it doesn't matter, as no user support means it is
difficult to justify the hours of work required.
_______________________________________________
bigdata mailing list
bigdata@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/bigdata@lists.fedoraproject.org



--
Mohd Izhar Firdaus Bin Ismail / KageSenshi
Fedora Malaysia Contributor & Ambassador
http://blog.kagesenshi.org
92C2 B295 B40B B3DC 6866  5011 5BD2 584A 8A5D 7331