On Thu, Jul 4, 2019 at 7:45 PM Ankur Sinha <sanjay.ankur@gmail.com> wrote:
On Thu, Jul 04, 2019 12:41:26 +0800, Mohd Izhar Firdaus Ismail wrote:
> Hi all,

Hello!

Long time no see!!

yeah its been a while ^^
 

> I'm wondering if there are people who are working on data engineering is
> interested in working on a SIG focusing on DE. 
>
> Currently my idea on this SIG would be:
> 1 - packaging data engineering related softwares into Fedora, and make them
> easy to install, covering from workflow tools (eg: airflow, luigi), data
> processing engines (eg: apache spark, flink), visualization tools (superset,
> redash) and make life easier for that. I'm not sure how much these tools can
> fit into fedora packaging guidelines (lots of bundled jars, and users expects
> upstream binaries, esp on engines such as spark/flink), which is something to
> brainstorm on. 
> 2 - ambassador related activities around promoting fedora as a platform for
> data engineers to use. 
>
> If interested, i'm on this telegram group: https://t.me/fedora_dataengineering

While I'm not directly in data engineering/science, this is now a very
very important component of the general scientific pipeline. So, I'd be
interested to help out, and I'm sure the rest of the NeuroFedora team
would be too. It'll probably also be beneficial to rope in the sci-tech
folks, and any other folks that were part of the data science/machine
learning SIGS (not sure if the SIGs are active). I also expect the
Astronomy SIG would be interested in some tools, since they also do deal
with rather large amounts of data. If we can bring more of these SIGs
together to help each other, we'll get a lot more work done :)

https://fedoraproject.org/wiki/Category:SciTech_SIG
https://fedoraproject.org/wiki/SIGs/ML
https://fedoraproject.org/wiki/Category:Astronomy_SIG?rd=Astronomy_SIG
https://docs.fedoraproject.org/en-US/neurofedora/overview/


yeah i agree .. my exp dealing with the tools highlights one tricky problem we might face -> avoiding statically / bundled dependencies , while on the same time getting all the tools to work .. most of them locks to specific dependency version, and wont work on other.. and at the moment i'm not sure whats the right approach to this, so really need input from others. My current workflow is to install each tool and all their dependencies in their own virtualenvs or vendor directory. 
 
We're meeting at Flock to discuss how we can leverage the excellent
resources that the Fedora community provides to enable Free Science.
Please do join the discussion too:

https://pagure.io/flock/issue/220


Unfortunately i dont think i can reach Flock, flying to budapest at the moment is beyond my budget and time availability. But if you can bring the topic on how to handle such dependency conflict / bundled dependencies, that'll be really helpful. 

--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


--
Mohd Izhar Firdaus Bin Ismail / KageSenshi
Fedora Malaysia Contributor & Ambassador
http://blog.kagesenshi.org
92C2 B295 B40B B3DC 6866  5011 5BD2 584A 8A5D 7331