Fwd: F21 Self Contained Change: Apache Spark
by Matthew Farrellee
FYI
-------- Original Message --------
Subject: F21 Self Contained Change: Apache Spark
Date: Fri, 21 Mar 2014 13:28 +0100
From: Jaroslav Reznik <jreznik(a)redhat.com>
Reply-To: devel(a)lists.fedoraproject.org
Organization: Red Hat, Inc.
To: devel-announce(a)lists.fedoraproject.org
= Proposed Self Contained Change: Apache Spark =
https://fedoraproject.org/wiki/Changes/ApacheSpark
Change owner(s): William Benton <willb(a)redhat.com>
Apache Spark is a fast and general engine for large-scale data processing.
This change brings Spark to Fedora, allowing easy deployment and
development
of Spark applications on Fedora.
== Detailed Description ==
Apache Spark is a fast and general engine for large-scale data
processing. It
supports developing custom analytic processing applications over large data
sets or streaming data. Because it has the capability to cache intermediate
results in cluster memory and schedule DAGs of computations, Spark programs
can run up to 100x faster than equivalent Hadoop MapReduce jobs. Spark
applications are easy to develop, parallel, fast, and resilient to failure,
and they can operate on data from in-memory collections, local files, a
Hadoop-
compatible filesystem, or from a variety of streaming sources. Spark also
includes libraries for distributed machine learning and graph algorithms.
== Scope ==
* Proposal owners: Currently our Spark package has been accepted into
Fedora
[1]. It features nearly all of the functionality available from the
upstream
release. (The missing features -- specifically, Python bindings, the Spark
REPL, Kryo-based serialization, primitives for approximate cardinalities of
very large sets, and Mesos integration -- were missing from the initial
packages due to unavailable dependencies and bundling issues; we're
working to
close the gap with upstream as quickly as possible.) This work depended
upon
Fedora 21's improved support for the Scala ecosystem [2].
* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
[1] http://pkgs.fedoraproject.org/cgit/spark.git
[2] https://fedoraproject.org/wiki/Changes/ImprovedScalaEcosystem
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
10 years, 1 month
Fwd: F21 Self Contained Change: Improved Scala Ecosystem Support
by Matthew Farrellee
FYI
-------- Original Message --------
Subject: F21 Self Contained Change: Improved Scala Ecosystem Support
Date: Fri, 21 Mar 2014 13:21:34 +0100
From: Jaroslav Reznik <jreznik(a)redhat.com>
Reply-To: devel(a)lists.fedoraproject.org
Organization: Red Hat, Inc.
To: devel-announce(a)lists.fedoraproject.org
= Proposed Self Contained Change: Improved Scala Ecosystem Support =
https://fedoraproject.org/wiki/Changes/ImprovedScalaEcosystem
Change owner(s): William Benton <willb(a)redhat.com>
Fedora now supports several essential parts of the Scala language
ecosystem as
well as building packages with sbt, the de facto build tool for the Scala
community.
== Detailed Description ==
Fedora has had a Scala package for some time, but the larger Scala
ecosystem
has been absent from Fedora. In fact, until very recently, Fedora
included no
packages that depended on Scala. The main obstacle to getting Scala
ecosystem
projects packaged for Fedora was the difficulty in packaging sbt, the
Simple
Build Tool, which many Scala projects use for build, dependency, and
release
management. Fedora 21 now includes sbt as well as several interesting and
foundational Scala ecosystem projects, most notably:
* akka, a toolkit for developing actor-based systems;
* json4s, a unified interface to JSON parsers and generators;
* sbinary, a typed Scala interface for reading and writing binary formats;
* sbt, the simple build tool for Scala and Java projects;
* scala-stm, a software transactional memory implementation for Scala;
* scalacheck, a property-based testing framework for Scala; and
* scalaz, a set of extensions to the Scala standard library to facilitate
functional programming.
== Scope ==
* Proposal owners: The change is complete as described; other ecosystem
packages and additional Fedora-specific developer documentation will
continue
to become available.
* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
10 years, 1 month
easy review request
by Will Benton
I just got the last vote I needed on the bundling exception for stream-lib, so if someone is willing to do an easy review (and grant approval pending the ticket update), I'd appreciate it:
https://bugzilla.redhat.com/show_bug.cgi?id=1074625
stream-lib is necessary to enable some functionality that is currently turned off in our Spark package.
best,
wb
10 years, 1 month
Fwd: Thinking ahead to 2.4
by Tim St. Clair
----- Forwarded Message -----
> From: "Arun C Murthy" <acm(a)hortonworks.com>
> To: hdfs-dev(a)hadoop.apache.org, yarn-dev(a)hadoop.apache.org, common-dev(a)hadoop.apache.org,
> mapreduce-dev(a)hadoop.apache.org
> Sent: Friday, March 14, 2014 10:23:41 PM
> Subject: Re: Thinking ahead to 2.4
>
> Update: We are now down to just 8 blockers of which 4 are already PA.
>
> I know it's getting in good shape per our QE gang too. If things go well, I
> plan to create an RC later half of next week.
>
> thanks,
> Arun
>
> On Mar 11, 2014, at 7:11 AM, Arun C Murthy <acm(a)hortonworks.com> wrote:
>
> > Sorry, the previous link had a bug, the correct one is:
> > http://s.apache.org/hadoop-2.4.0-blockers.
> >
> > We are currently down to 12 blockers; with several PA.
> >
> > thanks,
> > Arun
> >
> > On Mar 6, 2014, at 1:40 PM, Arun C Murthy <acm(a)hortonworks.com> wrote:
> >
> >> Gang,
> >>
> >> Most of the big-ticket items are already in, awesome!
> >>
> >> I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we
> >> get through the list of blockers. Here is a handy link:
> >> http://s.apache.org/hadoop-2.4-blockers
> >>
> >> If you find more, please set Target Version to 2.4.0 and mark it a
> >> blocker. I'll try nudging people to start closing these soon, appreciate
> >> any help!
> >>
> >> thanks,
> >> Arun
> >>
> >>
> >> On Feb 20, 2014, at 3:45 PM, Arun C Murthy <acm(a)hortonworks.com> wrote:
> >>
> >>> Thanks Azuryy & Suresh. I've updated the roadmap wiki to reflect this.
> >>>
> >>> Arun
> >>>
> >>> On Feb 20, 2014, at 2:01 PM, Suresh Srinivas <suresh(a)hortonworks.com>
> >>> wrote:
> >>>
> >>>> Arun,
> >>>>
> >>>> Some of the previously 2.4 targeted features were made available in 2.3:
> >>>> - Heterogeneous storage support
> >>>> - Datanode cache
> >>>>
> >>>> The following are being targeted for 2.4:
> >>>> - Use protobuf for fsimge (already in)
> >>>> - ACLs (in trunk. In a week or so, this will be merged to branch-2.4)
> >>>> - Rolling upgrades (last bunch of jiras being worked in feature branch.
> >>>> Will be in 2.4 in around two weeks. Currently testing is in progress)
> >>>>
> >>>> So HDFS features should be ready in two weeks.
> >>>>
> >>>>
> >>>> On Sat, Feb 15, 2014 at 4:47 PM, Azuryy <azuryyyu(a)gmail.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>> I think you omit some key pieces in 2.4
> >>>>>
> >>>>> Protobuf fsimage, rolling upgrade are also targeting 2.4
> >>>>>
> >>>>>
> >>>>>
> >>>>> Sent from my iPhone5s
> >>>>>
> >>>>>> On 2014年2月16日, at 6:59, Arun C Murthy <acm(a)hortonworks.com> wrote:
> >>>>>>
> >>>>>> Folks,
> >>>>>>
> >>>>>> With hadoop-2.3 nearly done, I think it's time to think ahead to
> >>>>> hadoop-2.4. I think it was a good idea to expedite release of 2.3 while
> >>>>> we
> >>>>> finished up pieces that didn't make it in such as HDFS Caching &
> >>>>> Support
> >>>>> for Heterogenous Storage.
> >>>>>>
> >>>>>> Now, most of the key pieces incl. Resource Manager Automatic Failover
> >>>>> (YARN-149), Application History Server (YARN-321) & Application
> >>>>> Timeline
> >>>>> Server (YARN-1530) are either complete or very close to done, and I
> >>>>> think
> >>>>> we will benefit with an extended test-cycle for 2.4 - similar to what
> >>>>> happened with 2.2. To provide some context: 2.2 went through nearly 6
> >>>>> weeks
> >>>>> of extended testing and it really helped us push out a very stable
> >>>>> release.
> >>>>>>
> >>>>>> I think it will be good to create a 2.4 branch ASAP and start testing.
> >>>>> As such, I plan to cut the branch early next week. With this, we should
> >>>>> be
> >>>>> good shape sometime to release 2.4 in mid-March.
> >>>>>>
> >>>>>> I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this.
> >>>>>>
> >>>>>> Also, we should start thinking ahead to 2.5 and what folks would like
> >>>>>> to
> >>>>> see in it. If we continue our 6-week cycles, we could shoot to get that
> >>>>> out
> >>>>> in April.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> thanks,
> >>>>>> Arun
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Arun C. Murthy
> >>>>>> Hortonworks Inc.
> >>>>>> http://hortonworks.com/
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://hortonworks.com/download/
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or entity
> >>>> to
> >>>> which it is addressed and may contain information that is confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>>> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> --
> >>> Arun C. Murthy
> >>> Hortonworks Inc.
> >>> http://hortonworks.com/
> >>>
> >>>
> >>
> >> --
> >> Arun C. Murthy
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >>
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata
10 years, 1 month