Fwd: [fedora-java] Maven update in rawhide
by Peter MacKinnon
FYI
-------- Original Message --------
Subject: [fedora-java] Maven update in rawhide
Date: Thu, 20 Feb 2014 16:02:01 +0100
From: Mikolaj Izdebski <mizdebsk(a)redhat.com>
To: Fedora Java Development List <java-devel(a)lists.fedoraproject.org>
I've just updated Maven in rawhide to next upstream version -- 3.2.1.
Please let me know if you encounter any issues.
--
Mikolaj Izdebski
IRC: mizdebsk
--
java-devel mailing list
java-devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/java-devel
10 years, 2 months
Packaging projects with dependencies required in remote HDFS/Task classpaths
by Peter MacKinnon
Hello,
The Fedora Big Data SIG is making significant strides in rounding out
the foundational Hadoop and BDAS packages
<https://fedoraproject.org/wiki/SIGs/bigdata/packaging#Packages_available_...>
in rawhide. We are at the point of review and inclusion of what could be
considered "application" packages, that is projects which are clients
(or some other type of integration) of the MapReduce and HDFS frameworks
themselves. Some of these packages need a subset of their jars and
classes deployed at the site of task execution in order to report back
status and results to some type of client, be it a shell, web interface,
or other consumer. A current example of this is amplab-tachyon (in F20)
and Apache Hive (under review): both these packages need specific jars
to be deployed (essentially) at the DataNode host in order to be
functional. However, this can include dependency jars in common: in this
example, libthrift-java.
The overall Hadoop2 classpath is lengthy and spans client, common, hdfs,
mapreduce, and yarn sub-components. There is not an easy way for a new
application to modify and co-habit that classpath from a configuration
standpoint with other applications. This is a problem that afflicts
Hadoop deployments independent of Fedora. In the domain, there are
different views and differing approaches:
1. the *-site.xml config files: they carry classpath properties but
most users view this approach as unmanageable where multiple
applications are involved
2. the distributed cache: Hadoop supports a distributed cache for
applications to share jars and other data but it requires a
programmatic approach not necessarily supported by all ecosystem
projects
3. the libJar hammer: explode and reconstitute a single monolithic jar
with every class your application needs (including dependencies), to
be passed through the framework for delivery at the task nodes
4. site deployment of dependency jars into the existing Hadoop2 lib
directories
The last option is the most expedient (currently) and predictable for
our purposes. But it requires package ownership of potentially redundant
dependencies.
So, the SIG is proposing to adopt a convention for these types of
dependencies . Since Hadoop and its sub-packages are invariably a
runtime requirement for the packages of higher-level abstraction, those
new packages should softlink any dependencies they need into the
well-known Hadoop lib directories at a location that makes the most
sense for their application.
* /usr/share/hadoop/{client,common,hdfs,mapreduce,yarn}/lib
* The softlink should be prefixed with the package name,
e.g.,"amplab-tachyon-libthrift.jar" in order to retain clean and
obvious ownership
This deals with the general case of dependency packages that currently
do not have older compats. Situations where lower-level compat conflicts
could arise (e.g., asm?) should be raised and discussed in the SIG.
To our knowledge, this should be suitable according to current packaging
guidelines.
Let us know if you have any questions or comments.
\Pete
--
Peter MacKinnon
MRG Grid/Big Data
Red Hat Inc.
Raleigh, NC
10 years, 2 months
Fwd: Thinking ahead to Hadoop 2.4
by Tim St. Clair
Folks -
Rob and I had been conversing earlier today, and we were both in agreement that it makes sense to plan our next update around 2.4 for several reasons. The thread below also adds an extra +1.
Comments solicited...
Cheers,
Tim
> ----- Forwarded Message -----
> > From: "Arun C Murthy" <acm(a)hortonworks.com>
> > To: common-dev(a)hadoop.apache.org, hdfs-dev(a)hadoop.apache.org,
> > yarn-dev(a)hadoop.apache.org,
> > mapreduce-dev(a)hadoop.apache.org
> > Sent: Saturday, February 15, 2014 4:59:45 PM
> > Subject: Thinking ahead to 2.4
> >
> > Folks,
> >
> > With hadoop-2.3 nearly done, I think it's time to think ahead to
> > hadoop-2.4.
> > I think it was a good idea to expedite release of 2.3 while we finished up
> > pieces that didn't make it in such as HDFS Caching & Support for
> > Heterogenous Storage.
> >
> > Now, most of the key pieces incl. Resource Manager Automatic Failover
> > (YARN-149), Application History Server (YARN-321) & Application Timeline
> > Server (YARN-1530) are either complete or very close to done, and I think
> > we will benefit with an extended test-cycle for 2.4 - similar to what
> > happened with 2.2. To provide some context: 2.2 went through nearly 6
> > weeks
> > of extended testing and it really helped us push out a very stable
> > release.
> >
> > I think it will be good to create a 2.4 branch ASAP and start testing. As
> > such, I plan to cut the branch early next week. With this, we should be
> > good shape sometime to release 2.4 in mid-March.
> >
> > I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this.
> >
> > Also, we should start thinking ahead to 2.5 and what folks would like to
> > see
> > in it. If we continue our 6-week cycles, we could shoot to get that out in
> > April.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.
> >
>
> --
> Cheers,
> Tim
>
--
Cheers,
Tim
10 years, 2 months
Tachyon updated. 0.4.1-SNAPSHOT
by Tim St. Clair
Greetings Folks,
As some are aware, there were rev's on nearly all the upstream BDAS projects last week in preparation for the Amp Camp @ Strata. We've been working through a series of integration issues and bug fixes, but it appears to be sorted out now.
To get folks started, I've put up a blog post that sets up a simple example:
http://timothysc.github.io/blog/2014/02/17/bdas-tachyon/
--
Cheers,
Tim
10 years, 2 months
Re: EPEL 7 dependencies
by Matthew Farrellee
On 02/17/2014 02:08 AM, Mattias Ellert wrote:
> Hi!
>
> I am trying to build my packages for EPEL 7. I have been successful for
> many of them, but for some I am blocked by missing dependencies. Could
> you comment on the status and plans for the missing dependencies?
>
> Package: Missing dependencies:
>
> arc-gui-clients: nordugrid-arc-devel
> canl-java: bouncycastle, bouncycastle-mail
> jglobus: bouncycastle, maven-patch-plugin, axis
> lhapdf: octave-devel
> maven-patch-plugin: maven-docck-plugin
> nordugrid-arc: pylint, xmlsec1-devel, xmlsec1-openssl-devel
> php-voms-admin: php-mysql, php-gd, php-soap
> pythia8: lhapdf
> root: google-droid-sans-fonts
> wine-symbol-fonts, wine-wingdings-fonts
> gfal-devel (obsolete - port to gfal2-devel)
> hadoop-devel
> libAfterImage-devel
> pythia8-devel
> texlive-stix
> voms-api-java: bouncycastle
>
> Mattias Ellert
mattias,
i don't have any and i don't know of any plans to migrate hadoop to
epel7. if you're interested, feel free to take a shot at it.
i'm cc'ing the big data sig on the off chance others are interested in
hadoop on epel7.
best,
matt
10 years, 2 months
Fwd: Election Results: Fedora Board, FESCo, and FAmSCo
by Matthew Farrellee
congrats number80!
-------- Original Message --------
Subject: Election Results: Fedora Board, FESCo, and FAmSCo
Date: Tue, 11 Feb 2014 12:02:27 -0500 (EST)
From: Robyn Bergeron <rbergero(a)redhat.com>
Reply-To: devel(a)lists.fedoraproject.org
To: announce(a)lists.fedoraproject.org, devel-announce
<devel-announce(a)lists.fedoraproject.org>, famsco(a)fedoraproject.org
CC: Fedora community advisory board <advisory-board(a)lists.fedoraproject.org>
Greetings, all:
The elections for the Fedora Engineering Steering Committee (FESCo) and
the Fedora Ambassadors Steering Committee (FAmSCo) have concluded, and
the results are shown below.
In addition to the election results, there were also 2 candidates for 2
open seats on the Fedora Project Board. As the candidates were
unopposed, voting was not held. Therefore, Neville A. Cross (FAS: yn1v)
and Haïkel Guémar (FAS: number80) are both elected to the Fedora Board
for a full two-release term.
******
FESCo is electing 4 seats this cycle. A total of 265 ballots were cast,
meaning a candidate could accumulate up to 1590 votes (265 * 6). The
result of the FESCo election is as follows:
# votes | name
- ------+----------------------------------
952 | Dennis Gilmore (FAS: ausil)
947 | Toshio Kuratomi (FAS: abadger1999)
905 | Stephen Gallagher (FAS: sgallagh)
849 | Miloslav Trmač (FAS: mitr)
-------------------------------------------
768 | Marcela Mašláňová (FAS: mmaslano)
649 | Kyle McMartin (FAS: kyle)
Therefore, Dennis Gilmore, Toshio Kuratomi, Stephen Gallagher, and
Miloslav Trmač are each elected to FESCo for a full two-release term.
******
FAmSCo is electing 3 seats this cycle. A total of 173 ballots were cast,
meaning a candidate could accumulate up to 692 votes (173 * 4). The
result of the FAmSCo election is as follows:
# votes | name
- ------+----------------------------------
435 | Neville A. Cross (FAS: yn1v)
396 | Jon Disnard (FAS: masta)
379 | Truong Anh Tuan (FAS: tuanta)
-------------------------------------------
291 | Marcel Ribeiro Dantas (FAS: mribeirodantas)
Therefore, Neville Cross, Jon Disnard, and Truong Anh Tuan are each
elected to FAmSCo for a full two-release term.
Congratulations to all.
-Robyn
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
10 years, 2 months
Any Fedorans interested in running the Cascading Compatibility Test Suite?
by Bob Taylor
Fedorans,
We are hoping a member of the community could run our Cascading
Compatibility Test Suite with the version of Hadoop packaged for Fedora.
For those not familiar with Cascading, it is a is an Open Source
application framework for Java developers to simply develop robust Data
Analytics and Data Management applications on Apache Hadoop. In addition,
a number of DSLs have been developed for Cascading enabling Scala and
Clojure developers access to Cascading and Hadoop.
You can access the Compatibility suite here -
https://github.com/Cascading/cascading.compatibility
This project will test the binary compatibility of Cascading with differing
releases of an underlying platform.
Once the compatibility testing is completed we hope to start work on a RPM
for Cascading.
Thanks.
Bob
--
Bob Taylor
www.concurrentinc.com
bobt(a)concurrentinc.com
10 years, 2 months
Fwd: FESCo announces acceptance of Fedora.next PRDs
by Matthew Farrellee
Great news, the Cloud and Env&Stacks working groups have heard us and
added some of our use cases to their F21 PRDs!
Here are two great opportunities for folks to lead collaboration between
the Big Data SIG and working groups...
https://fedoraproject.org/wiki/Cloud_PRD includes,
Requirement #4: Needs for Big Data Image
Since large and complex data sets that are difficult to process using
RDBMS and traditional data processing tools are now common, and the big
data open source big data ecosystem is now quite mature, we ought to
provide a ready-to-go image for these usages. Hadoop and its ecosystem
is clearly a key player and the Big Data SIG is working to bring it on
Fedora. Should. Real-time analysis is also growing, and components like
Spark and Storm are also something we'd like to include.
https://fedoraproject.org/wiki/Env_and_Stacks/Product_Requirements_Document
includes,
Persona #1: Alan the Big Data analyst
Alan is a Big Data analyst and member of the Fedora Big Data SIG. He
uses a number of applications written in different languages to perform
the data analysis. He wants to focus his time and effort in the
application and the actual data analysis. He want to minimize the time
and hassle spent obtaining, compiling, packaging and maintaining the
applications that he needs. The form of packaging (rpm, deb, npm, other)
isn't as important to him.
Problem statement: Currently, there is a lot of hassle and pain in
dealing with non-primary (i.e. C/C++, Python, Perl) language stacks in
Fedora. Although Alan wants to focus his time in the application and the
actual data analysis, he too often finds himself spending time managing
the language-specific toolchain needed by the application.
Cause #1: The upstream application authors usually assume that any
developer would just be using the language-specific packaging ecosystem
rather that also taking into account the downstream distribution-based
packaging and dependency management systems. This is a problem since
there are many differences between language-specific packaging
ecosystems and the Fedora way of packaging software.
Cause #2: Many upstream applications use very brittle versioning. In
other words, each application expects the user will be able to have its
particular versions of the runtime, compiler and libraries available.
Example #1: Applications written in Scala require different versions of
Scala, some need v2.9 and some need v2.10. Although Fedora could provide
both versions in the same release (via SCLs or by manually maintaining
two Scala packages), they would both need to be resolvable via Apache
Ivy, rather than just providing both binaries, "scalac29" and "scalac210".
Example #2: The version of Jetty in Fedora does not work with Java 6 and
the Apache Hadoop upstream isn't ready to abandon Java 6 yet. So the Big
Data SIG ends up maintaining a patch to Apache Hadoop for Jetty 9 that
will live purely in Fedora for quite a while to come. Although this
could be worked-around by using compat packages for Jetty for a couple
of Fedora releases, it just hides the fact that there is a mismatch in
expectations between Fedora and upstream application authors.
Example #3: Node.js has its package/dep management tool available in
Fedora, but very little of the language ecosystem / base libraries are
packaged and available in Fedora.
Possible solution: Fedora would need a way to provide language-specific
ecosystems in a way that aligns with how these language itself are used
and applications written in them are developed and distributed.
-------- Original Message --------
Subject: FESCo announces acceptance of Fedora.next PRDs
Date: Tue, 4 Feb 2014 11:21:46 -0500
From: Bill Nottingham <notting(a)redhat.com>
Reply-To: devel(a)lists.fedoraproject.org
To: devel-announce(a)lists.fedoraproject.org
Recently, as part of the Fedora.next effort, FESCo has accepted the PRDs
from
the following Working Groups:
- Workstation
https://fedoraproject.org/wiki/Workstation/Workstation_PRD
- Server
https://fedoraproject.org/wiki/Server/Product_Requirements_Document
- Cloud
https://fedoraproject.org/wiki/Cloud_PRD
- Environments and Stacks
https://fedoraproject.org/wiki/Env_and_Stacks/Product_Requirements_Document
FESCo would like to thank all the members of these working groups, and all
those that have contributed to these documents, for all their time and
effort on creating these on short notice, with next to no requirements,
while
still doing their other Fedora work. Thanks to the all the contributors
that worked on this process.
We are now ready to move on to concrete technical plans for how the
products can be created, tested, distributed and marketed. Help Fedora
dream big and we can work with our infrastructure, QA, releng, websites,
design, translation and marketing teams to do great things for Fedora
21. We encourage those interested in these areas to check out these links
and participate in these communities and discussions, as we work together
to make Fedora the best it can be.
Bill Nottingham, on behalf of FESCo
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
10 years, 2 months