On Thu, Jul 4, 2019 at 7:45 PM Ankur Sinha sanjay.ankur@gmail.com wrote:
On Thu, Jul 04, 2019 12:41:26 +0800, Mohd Izhar Firdaus Ismail wrote:
Hi all,
Hello!
Long time no see!!
yeah its been a while ^^
I'm wondering if there are people who are working on data engineering is interested in working on a SIG focusing on DE.
Currently my idea on this SIG would be: 1 - packaging data engineering related softwares into Fedora, and make
them
easy to install, covering from workflow tools (eg: airflow, luigi), data processing engines (eg: apache spark, flink), visualization tools
(superset,
redash) and make life easier for that. I'm not sure how much these tools
can
fit into fedora packaging guidelines (lots of bundled jars, and users
expects
upstream binaries, esp on engines such as spark/flink), which is
something to
brainstorm on. 2 - ambassador related activities around promoting fedora as a platform
for
data engineers to use.
If interested, i'm on this telegram group:
https://t.me/fedora_dataengineering
While I'm not directly in data engineering/science, this is now a very very important component of the general scientific pipeline. So, I'd be interested to help out, and I'm sure the rest of the NeuroFedora team would be too. It'll probably also be beneficial to rope in the sci-tech folks, and any other folks that were part of the data science/machine learning SIGS (not sure if the SIGs are active). I also expect the Astronomy SIG would be interested in some tools, since they also do deal with rather large amounts of data. If we can bring more of these SIGs together to help each other, we'll get a lot more work done :)
https://fedoraproject.org/wiki/Category:SciTech_SIG https://fedoraproject.org/wiki/SIGs/ML https://fedoraproject.org/wiki/Category:Astronomy_SIG?rd=Astronomy_SIG https://docs.fedoraproject.org/en-US/neurofedora/overview/
yeah i agree .. my exp dealing with the tools highlights one tricky problem we might face -> avoiding statically / bundled dependencies , while on the same time getting all the tools to work .. most of them locks to specific dependency version, and wont work on other.. and at the moment i'm not sure whats the right approach to this, so really need input from others. My current workflow is to install each tool and all their dependencies in their own virtualenvs or vendor directory.
We're meeting at Flock to discuss how we can leverage the excellent resources that the Fedora community provides to enable Free Science. Please do join the discussion too:
Unfortunately i dont think i can reach Flock, flying to budapest at the moment is beyond my budget and time availability. But if you can bring the topic on how to handle such dependency conflict / bundled dependencies, that'll be really helpful.