--
Javi Roman
On Sat, Aug 1, 2015 at 1:22 AM, Christopher <ctubbsii-fedora(a)apache.org> wrote:
On Fri, Jul 31, 2015 at 10:27 AM, Gerald Henriksen
<ghenriks(a)gmail.com> wrote:
> So the discussion about Spark has revealed:
>
> 1) the Big Data community doesn't trust the Fedora packages
From my perspective, I see a lot of things happening in the "Big Data
community" which indicates not so much distrust for Fedora packages
(or RHEL, for that matter), but more a complete lack of understanding
about the benefits of packaging for these projects.
> 2) it seems a consensus that the realities of the Big Data ecosystem
> don't work very well with making packages for Fedora
I don't think that's necessarily true. I think a lot of the community
is following the "Hadoop Way", where developers are just trying to
mimic the way Hadoop does things, but Hadoop seems to be pretty poorly
packaged and has had a lot of problems over time, from complete lack
of dependency convergence, lack of packaging standards, improper use
of the Linux filesystem hierarchy, confusing use of shell scripts and
environment variables set all over the place, etc.
A lot of this is where downstream packagers can bring good things back
to the upstream communities. So, I don't think it's that the ecosystem
doesn't work well with Fedora packaging... I think it's more that they
haven't had sufficient feedback from those communities to push them to
make their software more easily integrated. (Part of this is a Java
thing more than a big data thing... let's face it... Java packaging
hasn't really had a lot of good support until quite recently, with
xmvn and related tools.)
>
> 3) result of 2, move towards newer (Docker) style of making Big Data
> available on Fedora, perhaps with a side benefit of it then also being
> possible to have it available on RHEL.
Docker still has some limitations, and is not necessarily suitable to
everybody's situation. I'd still prefer Fedora RPMs over Docker-based
packaging. I think the community could benefit from both.
>
> Should we then also have a discussion, or just do it, of removing the
> existing packages from Fedora 23?
I'd be against that. It's one thing if there's no maintainer, but it's
another to decide to oust a whole class of projects.
>
> The packages that are there are now in some cases at least several
> versions out of date, which isn't doing either Fedora or the Big Data
> SIG any favours.
Big Data packages aren't the only packages to suffer from this.
There's many projects that are out of date. I'd rather Fedora have a
relatively stable version than no version at all. Maybe instead we
should try working with the upstream communities more to ease
integration, so it's easier to update?
The problem here is the big data tools, for example Apache Flume, or
Apache Spark are evolving quickly with new important features, however
some of the libraries used by those tools (for example Java artifacts in Flume)
have frozen versions, because it's works for the developer. In Fedora
some of those
libraries are increasing their versions (the most updated versions
from the upstream project), and the big data tool affected breaks in
compilation time.
So we have two options:
1. The package maintainer (of the tools) must to patch the tool over
and over, in order to adapt the code to the new interfaces, sometimes
very complex task. And very difficult to follow the Fedora release policy cycle.
2. Try to convince the upstream developer for update the version of
the library of his tool. This is very improbable to achieve, because
the developer is thinking in the new features and he is not thinking
about change the way the tools are writing in the log (log4j), for
example.
I think we can not solve easily this situation without multi-version
packages. On the other hand we can help with things such as FHS,
environment variables, weird shell scripts and so forth.
>
> Additionally, if we do create Docker images they offer the possibility
> of creating bug/error opportunities if a user somehow brings an rpm
> package into the Docker image.
>
> To use the now favourite term, do we "reboot" the Big Data SIG and
> start fresh?
>
> Thoughts?
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
_______________________________________________
bigdata mailing list
bigdata(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/bigdata