On Fri, 31 Jul 2015 19:22:50 -0400, you wrote:
On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen
<ghenriks(a)gmail.com> wrote:
> So the discussion about Spark has revealed:
>
> 1) the Big Data community doesn't trust the Fedora packages
From my perspective, I see a lot of things happening in the "Big Data
community" which indicates not so much distrust for Fedora packages
(or RHEL, for that matter), but more a complete lack of understanding
about the benefits of packaging for these projects.
As posted in the Spark thread, the feedback to Red Hat employee(s) is
specifically that the packages in Fedora are not trusted.
In may well be a lack of understanding, or it may be that they don't
like the changes required to package for a distro that has newer
libraries than they use, but the end result is the same.
> 2) it seems a consensus that the realities of the Big Data
ecosystem
> don't work very well with making packages for Fedora
I don't think that's necessarily true. I think a lot of the community
is following the "Hadoop Way", where developers are just trying to
mimic the way Hadoop does things, but Hadoop seems to be pretty poorly
packaged and has had a lot of problems over time, from complete lack
of dependency convergence, lack of packaging standards, improper use
of the Linux filesystem hierarchy, confusing use of shell scripts and
environment variables set all over the place, etc.
No arguement, Hadoop and some of its ecosystem have what I would agree
are significant issues.
A lot of this is where downstream packagers can bring good things
back
to the upstream communities. So, I don't think it's that the ecosystem
doesn't work well with Fedora packaging... I think it's more that they
haven't had sufficient feedback from those communities to push them to
make their software more easily integrated. (Part of this is a Java
thing more than a big data thing... let's face it... Java packaging
hasn't really had a lot of good support until quite recently, with
xmvn and related tools.)
I have followed the Hadoop devel list for over 6 months now, and the
perhaps incorrect impression I have is that the Hadoop community is
largely being governed by feedback from places that are using Hadoop.
And those places seem to have made it clear while they want bug fixes
and new features, they want it done so in a way that creates the least
amount of disruption (and hence testing) required.
It will be very difficult for a downstream like Fedora to overcome
that sort of influence.
> 3) result of 2, move towards newer (Docker) style of making Big
Data
> available on Fedora, perhaps with a side benefit of it then also being
> possible to have it available on RHEL.
Docker still has some limitations, and is not necessarily suitable to
everybody's situation. I'd still prefer Fedora RPMs over Docker-based
packaging. I think the community could benefit from both.
I am only getting started with Docker, but I agree it is not an ideal
solution. I too would prefer proper rpm packages in Fedora, but it
just doesn't seem to be feasible without significant compromises
leading me to think that Docker containers offer best of a bunch of
bad options.
> Should we then also have a discussion, or just do it, of removing
the
> existing packages from Fedora 23?
I'd be against that. It's one thing if there's no maintainer, but it's
another to decide to oust a whole class of projects.
I was thinking of a case where Fedora switched to providing Hadoop
using Docker containers, in which case also having older versions of
the software installable via rpm could cause issues.
Anyway, we are already going to lose Spark because it no longer builds
and I would guess it is only a matter of time before a library update
means the rest of the Big Data stack also fails to build.