Bruno -
Feel free to join the dev-lists and IRC channels(fedora-devel, fedora-bigdata), as there
are numerous items we are still hashing out here
(
), and we would be happy to coordinate.
For example, getting parity across the tests is a stubborn foe. Once Hadoop we've
tackled the main package we can eval and coordinate other ecosystem packages.
Cheers,
Tim
----- Original Message -----
From: "Bruno Mahé" <bruno(a)bmahe.net>
To: "Matthew Farrellee" <matt(a)redhat.com>
Cc: puntogil(a)libero.it, tstclair(a)redhat.com
Sent: Wednesday, June 12, 2013 1:56:57 AM
Subject: Re: Packaging Hadoop in the Fedora community
+Tim
Hi Matthew and Tim,
I noticed Tim's email on the Apache Bigtop/Hadoop mailing list
and was wondering how we could all collaborate and work together to
improve the state of Apache Hadoop (and its ecosystem) on
Fedora/CentOS/RHEL.
Thanks,
Bruno
On 04/26/2013 10:38 AM, Bruno Mahé wrote:
> On 04/25/2013 06:49 AM, Matthew Farrellee wrote:
>> On 04/25/2013 01:55 AM, Bruno Mahé wrote:
>>> On 04/24/2013 05:40 AM, Matthew Farrellee wrote:
>>>> Hello Big Data SIG!
>>>>
>>>> A few of us in the Fedora community have been working to package Apache
>>>> Hadoop for Fedora. The feature page is,
>>>>
>>>>
https://fedoraproject.org/wiki/Features/Hadoop
>>>>
>>>> You'll find what we plan to package, our progress working through
all
>>>> the dependencies, a process for how we're attacking dependencies and
a
>>>> pre-Fedora repo of the packages in progress.
>>>>
>>>> It's a big effort, but will form a foundation for packaging more
Hadoop
>>>> ecosystem packages for Fedora.
>>>>
>>>> We're at a point where we need a better place to engage the
community
>>>> and the Big Data SIG is a natural choice. I hope you will welcome us to
>>>> the list and IRC, and maybe lend a hand!
>>>> _______________________________________________
>>>> bigdata mailing list
>>>> bigdata(a)lists.fedoraproject.org
>>>>
https://admin.fedoraproject.org/mailman/listinfo/bigdata
>>>
>>>
>>> Hi,
>>>
>>> I am not CC-ing the Bigdata SIG as to not spam the list.
>>>
>>> One of the project I work on intersects closely with this SIG. This
>>> project is Apache Bigtop (
http://bigtop.apache.org/).
>>> The goal of Apache Bigtop is three folds:
>>> 1/ Provide top notch packages for Apache Hadoop related projects
>>> 2/ Provide a point of integration and testing for all these projects
>>> 3/ Provide means to reliably deploy a complete stack.
>>>
>>> So there is quite a bit of overlapping between Apache Bigtop, this SIG
>>> and your efforts.
>>>
>>> Would you see any way we could collaborate?
>>>
>>>
>>> Please, see this previous message for an explanation about Apache Bigtop
>>> with regards to this SIG and Fedora:
>>>
http://lists.fedoraproject.org/pipermail/bigdata/2013-March/000012.html
>>>
>>>
>>> Thanks,
>>> Bruno
>>
>> Definitely. I hold an assumption that Bigtop is a good place to -
>>
>> 0) drive consistency across the Hadoop ecosystem
>> 1) provide deliverables that are integrated into various OS platforms
>>
>> For (0), our packaging effort is finding old and inconsistent (different
>> versions of the same) dependencies across projects.
>>
>> For (1), at a minimum I assume that the packaging work done for Fedora
>> can be contributed to Bigtop. This would provide integration with Fedora
>> at a level that does not currently exist.
>>
>> Beyond that. I'm interested in the testing framework that Bigtop
>> provides. Immediately, for the Fedora packaging effort, we need a way to
>> make sure the packages we provide aren't breaking core functionality. I
>> don't think TestDFSIO and some GridMix2 will be sufficient.
>>
>> Best,
>>
>>
>> matt
>>
>
> Hi,
>
> For (0), we are already working on this. We also spend a lot of efforts
> helping verifying that release candidates do not break their downstream
> projects. So the more people helping on that front, the merrier!
>
> For (1), we would definitely gladly accepts all contributions! And we
> would look at the best way to integrate your work.
> The sticky point may be related to Fedora specific macros. Our spec file
> works across a bunch of RPM distributions, so Fedora specific macros
> would have to be inside some ifdef or encapsulated or any other solution
> (even using Apache Bigtop as a common effort and forking the specs to
> add the Fedora specific macros). But in any case that would be
> implementation details and we can look at it whenever you feel to
> contribute your work.
>
> Apache Bigtop has some smoke tests, package tests (verify they can be
> installed, removed, upgraded, permissions are rights...) as well as
> integration tests. So I believe that could be leveraged by you whether
> or not you decide to put in common the efforts to work on packaging or
> not. If you have any issue using the tests, feel free to ping me or to
> send an email to the mailing list.
>
>
>
> Thanks,
> Bruno