Dne 26. 02. 19 v 23:42 Ryan Lerch napsal(a):

On Wed, 27 Feb 2019 at 05:39, Clement Verna <cverna@fedoraproject.org mailto:cverna@fedoraproject.org> wrote:

Hi all,

fedora-packages [0] code base is showing its age. The code base and
the technology stack  (Turbogears2 [1] web framework and the Moksha
[2] middleware) is currently not ready for Python3 and I am not
planning to do the work required to make it Python3 compatible, so the
application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of
Concept (fedora-search [3]) to replace the backend of the application.

Backend of fedora-packages is mdapi, isn't it? What would be the difference?

Fedora-search would be a REST API service offering full test search
API. Such a service would then be available for other application to
use, fedora-packages would then become a frontend only application
using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that
we should be proceeding that way, for the simple reason that this add
yet another code base to maintain, I think we should use this
opportunity to consider using Elasticsearch instead of maintaining our
own "search engine".

I think that Elasticsearch offers quite a few advantages :
  - Powerful Query language
  - Python bindings
  - Javascript bindings
  - Can be deployed in our infrastructure or used as a service
  - Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our
infrastructure ? Should we look at deploying a cluster in our infra /
Should we approach the Council to see if we can get founding to have
this service hosted by Elastic ?

Thanks
Clément

From an information point of view, the package-centric view of Fedora-packages is quite similar to the pagure dist-git. Would it worth investgating merging the front-end functionality of packages (lists of builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora packages front end?

+1

V.

—ry

[0] - https://apps.fedoraproject.org/packages/
[1] - http://www.turbogears.org/
[2] - https://mokshaproject.github.io/mokshaproject.net/
[3] - https://github.com/fedora-infra/fedora-search
_______________________________________________
infrastructure mailing list --
infrastructure@lists.fedoraproject.org
<mailto:infrastructure@lists.fedoraproject.org>
To unsubscribe send an email to
infrastructure-leave@lists.fedoraproject.org
<mailto:infrastructure-leave@lists.fedoraproject.org>
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

---

infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

On Tue, Feb 26, 2019 at 5:43 PM Ryan Lerch rlerch@redhat.com wrote:

On Wed, 27 Feb 2019 at 05:39, Clement Verna cverna@fedoraproject.org wrote:

...

Hi all,

fedora-packages [0] code base is showing its age. The code base and the technology stack (Turbogears2 [1] web framework and the Moksha [2] middleware) is currently not ready for Python3 and I am not planning to do the work required to make it Python3 compatible, so the application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of Concept (fedora-search [3]) to replace the backend of the application. Fedora-search would be a REST API service offering full test search API. Such a service would then be available for other application to use, fedora-packages would then become a frontend only application using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that we should be proceeding that way, for the simple reason that this add yet another code base to maintain, I think we should use this opportunity to consider using Elasticsearch instead of maintaining our own "search engine".

I think that Elasticsearch offers quite a few advantages :

• Powerful Query language
• Python bindings
• Javascript bindings
• Can be deployed in our infrastructure or used as a service
• Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our infrastructure ? Should we look at deploying a cluster in our infra / Should we approach the Council to see if we can get founding to have this service hosted by Elastic ?

Thanks Clément

From an information point of view, the package-centric view of Fedora-packages is quite similar to the pagure dist-git. Would it worth investgating merging the front-end functionality of packages (lists of builds, bugs, updates, etc) into pagure dist-git, and retiring the Fedora packages front end?

How would you propose we'd allow people to search by binary packages and seeing contents, changelog, and seeing which versions are shipped in releases?

I don't think it would be a good idea to merge the two views, because it really doesn't make this any easier. The package search is a user-focused view into the package collection, and is a helpful reference (when it works!). I think it still serves a purpose to have a dedicated frontend.

On Wed, 2019-02-27 at 10:48 +0100, Pierre-Yves Chibon wrote:

We look at Elasticsearch at one point (I believe Aurélien was even looking at setting up an instance in our cloud) but it came down to two aspects:

• setting it up is another application to maintain
• an application we have no expertise on

On the other side, fedora-search is:

• setting up another application to maintain
• an application we have expertise on

Well, fedora-search is also:

- an application we have to create. - an application we have to maintain the code and the deployment, where elasticsearch is just maintaining the deployment.

On Wed, 27 Feb 2019 at 13:27, Stephen John Smoogen smooge@gmail.com wrote:

On Tue, 26 Feb 2019 at 14:39, Clement Verna cverna@fedoraproject.org wrote:

...

Hi all,

fedora-packages [0] code base is showing its age. The code base and the technology stack (Turbogears2 [1] web framework and the Moksha [2] middleware) is currently not ready for Python3 and I am not planning to do the work required to make it Python3 compatible, so the application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of Concept (fedora-search [3]) to replace the backend of the application. Fedora-search would be a REST API service offering full test search API. Such a service would then be available for other application to use, fedora-packages would then become a frontend only application using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that we should be proceeding that way, for the simple reason that this add yet another code base to maintain, I think we should use this opportunity to consider using Elasticsearch instead of maintaining our own "search engine".

The main issues to getting elasticsearch working in the past was the following:

1 The number of systems needed to make it work. There is a large difference from their 'proof-of-concept see how great this is' to 'ok you want to do anything with load' setups in everything from storage to number of search nodes to network speeds. [The number of hardware for the data we have was to start with 5-8 dedicated Dell systems, some amount of shared fast storage, and N virtual machines with a 10-40GB backbone.. or throwing all of Fedora Infrastructure at once into the cloud.. because the feed it from PHX2 to the cloud is expensive.]

2. Packaging of elasticsearch was a mess. At the time we had rules

that all packages needed to be packaged in Fedora and follow Fedora packaging rules. [This one has been relaxed.]

3. Running of elasticsearch was a large service in itself. It doesn't

take care of itself and we would need one or more people who know it well to keep it running. [This goes down the ladder.. the logstash backends are also full services.. ] Most of that was written in Java which no one on the team at the time had good experiences with.

4. A kibana/elasticsearch query expert. Just like any database, most

of the queries you can make are the worse kind. They will take a lot more CPU/memory/time than they should making just grepping for data faster.

However that is 3-5 years ago.. so a lot has changed since then.

Thanks smooge for feeling my knowledge gap on what was looked at previously, I feel that a lot of the issues would actually be solved if we were to outsource the service and have someone else maintain the cluster for us. If that's the case I would be happy to open a council ticket to start the conversation about this possibility .

...

I think that Elasticsearch offers quite a few advantages :

• Powerful Query language
• Python bindings
• Javascript bindings
• Can be deployed in our infrastructure or used as a service
• Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our infrastructure ? Should we look at deploying a cluster in our infra / Should we approach the Council to see if we can get founding to have this service hosted by Elastic ?

Thanks Clément

[0] - https://apps.fedoraproject.org/packages/ [1] - http://www.turbogears.org/ [2] - https://mokshaproject.github.io/mokshaproject.net/ [3] - https://github.com/fedora-infra/fedora-search _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

-- Stephen J Smoogen. _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

I looked at the version of Elasticsearch via sources and spec file, and I know that Elasticsearch is on 6.0 right now. It is true that we do need an expert in elasticsearch in order to move forward on using elasticsearch as an api if agreed on using elasticsearch.

I am currently working on Elasticsearch on another project and learning more about Elasticsearch, but this is my first time actually using it. Are there any plans to upgrade Elasticsearch to an updated version?

On Wed, Feb 27, 2019 at 5:56 PM Stephen John Smoogen smooge@gmail.com wrote:

On Wed, 27 Feb 2019 at 14:36, Mikolaj Izdebski mizdebsk@redhat.com wrote:

...

On Wed, Feb 27, 2019 at 1:20 PM Stephen John Smoogen smooge@gmail.com wrote:

...

2. Packaging of elasticsearch was a mess. At the time we had rules

that all packages needed to be packaged in Fedora and follow Fedora packaging rules. [This one has been relaxed.]

I just want to point out that Elasticsearch has been packaged [1] in Fedora for more than 4 years.

https://src.fedoraproject.org/rpms/elasticsearch

The version I am seeing there is 1.7.5 and the version on github is 6.6.1 .. i didn't know if that was a 'dont even think about using that'

Upstream skipped a bunch of versions. They went from 1.x to 2.x to 5.x and now 6.x with 7.x in development.

I forgot the exact reason for this, but it had something to do with aligning the versions across all of Elastic's software.

I would like to take on maintaining the package, however, I do not have adequate experience in creating and maintaining as such. I have looked at the rpm packaging guidelines, and I can see what to do, but actually doing it is a little bit different. I understand everyone has different methods in completing the same outcome, which is creating and maintaining packages.

I am learning, however, within a mock environment, but I do not sufficient permissions to push the package anyway. If it is possible to go forward with Elasticsearch, I would be interested in maintaining but I would need a co-maintainer for assistance.

On Wed, 27 Feb 2019 at 16:05, Jim Perrin jperrin@redhat.com wrote:

How much heresy is involved in us using Amazon's elasticsearch service for this, so that we don't have yet-another-thing to maintain?

I was wondering how much data are we looking to shove there, does that data need to be 'protected', and how fast do we need it to be for us to talk back and forth to the cloud. The heresy side I don't have any say in..

On 2/27/19 4:19 AM, Stephen John Smoogen wrote:

...

On Tue, 26 Feb 2019 at 14:39, Clement Verna cverna@fedoraproject.org wrote:

...

Hi all,

fedora-packages [0] code base is showing its age. The code base and the technology stack (Turbogears2 [1] web framework and the Moksha [2] middleware) is currently not ready for Python3 and I am not planning to do the work required to make it Python3 compatible, so the application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of Concept (fedora-search [3]) to replace the backend of the application. Fedora-search would be a REST API service offering full test search API. Such a service would then be available for other application to use, fedora-packages would then become a frontend only application using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that we should be proceeding that way, for the simple reason that this add yet another code base to maintain, I think we should use this opportunity to consider using Elasticsearch instead of maintaining our own "search engine".

The main issues to getting elasticsearch working in the past was the following:

1 The number of systems needed to make it work. There is a large difference from their 'proof-of-concept see how great this is' to 'ok you want to do anything with load' setups in everything from storage to number of search nodes to network speeds. [The number of hardware for the data we have was to start with 5-8 dedicated Dell systems, some amount of shared fast storage, and N virtual machines with a 10-40GB backbone.. or throwing all of Fedora Infrastructure at once into the cloud.. because the feed it from PHX2 to the cloud is expensive.]

2. Packaging of elasticsearch was a mess. At the time we had rules

that all packages needed to be packaged in Fedora and follow Fedora packaging rules. [This one has been relaxed.]

3. Running of elasticsearch was a large service in itself. It doesn't

take care of itself and we would need one or more people who know it well to keep it running. [This goes down the ladder.. the logstash backends are also full services.. ] Most of that was written in Java which no one on the team at the time had good experiences with.

4. A kibana/elasticsearch query expert. Just like any database, most

of the queries you can make are the worse kind. They will take a lot more CPU/memory/time than they should making just grepping for data faster.

However that is 3-5 years ago.. so a lot has changed since then.

...

I think that Elasticsearch offers quite a few advantages :

• Powerful Query language
• Python bindings
• Javascript bindings
• Can be deployed in our infrastructure or used as a service
• Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our infrastructure ? Should we look at deploying a cluster in our infra / Should we approach the Council to see if we can get founding to have this service hosted by Elastic ?

Thanks Clément

[0] - https://apps.fedoraproject.org/packages/ [1] - http://www.turbogears.org/ [2] - https://mokshaproject.github.io/mokshaproject.net/ [3] - https://github.com/fedora-infra/fedora-search _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

---

infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

On Wed, 27 Feb 2019 at 22:11, Jim Perrin jperrin@redhat.com wrote:

How much heresy is involved in us using Amazon's elasticsearch service for this, so that we don't have yet-another-thing to maintain?

If we want to proceed with elasticsearch I think that this is indeed a good way forward. What would it take to use our current AWS account to setup a test cluster ?

On 2/27/19 4:19 AM, Stephen John Smoogen wrote:

...

On Tue, 26 Feb 2019 at 14:39, Clement Verna cverna@fedoraproject.org wrote:

...

Hi all,

fedora-packages [0] code base is showing its age. The code base and the technology stack (Turbogears2 [1] web framework and the Moksha [2] middleware) is currently not ready for Python3 and I am not planning to do the work required to make it Python3 compatible, so the application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of Concept (fedora-search [3]) to replace the backend of the application. Fedora-search would be a REST API service offering full test search API. Such a service would then be available for other application to use, fedora-packages would then become a frontend only application using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that we should be proceeding that way, for the simple reason that this add yet another code base to maintain, I think we should use this opportunity to consider using Elasticsearch instead of maintaining our own "search engine".

The main issues to getting elasticsearch working in the past was the following:

1 The number of systems needed to make it work. There is a large difference from their 'proof-of-concept see how great this is' to 'ok you want to do anything with load' setups in everything from storage to number of search nodes to network speeds. [The number of hardware for the data we have was to start with 5-8 dedicated Dell systems, some amount of shared fast storage, and N virtual machines with a 10-40GB backbone.. or throwing all of Fedora Infrastructure at once into the cloud.. because the feed it from PHX2 to the cloud is expensive.]

2. Packaging of elasticsearch was a mess. At the time we had rules

that all packages needed to be packaged in Fedora and follow Fedora packaging rules. [This one has been relaxed.]

3. Running of elasticsearch was a large service in itself. It doesn't

take care of itself and we would need one or more people who know it well to keep it running. [This goes down the ladder.. the logstash backends are also full services.. ] Most of that was written in Java which no one on the team at the time had good experiences with.

4. A kibana/elasticsearch query expert. Just like any database, most

of the queries you can make are the worse kind. They will take a lot more CPU/memory/time than they should making just grepping for data faster.

However that is 3-5 years ago.. so a lot has changed since then.

...

I think that Elasticsearch offers quite a few advantages :

• Powerful Query language
• Python bindings
• Javascript bindings
• Can be deployed in our infrastructure or used as a service
• Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our infrastructure ? Should we look at deploying a cluster in our infra / Should we approach the Council to see if we can get founding to have this service hosted by Elastic ?

Thanks Clément

[0] - https://apps.fedoraproject.org/packages/ [1] - http://www.turbogears.org/ [2] - https://mokshaproject.github.io/mokshaproject.net/ [3] - https://github.com/fedora-infra/fedora-search _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

---

infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...

On Wed, Feb 27, 2019 at 01:04:57PM -0800, Jim Perrin wrote:

How much heresy is involved in us using Amazon's elasticsearch service for this, so that we don't have yet-another-thing to maintain?

I don't think this is heresy at all. Hosted open source services are great, and the Amazon web services team has been very generous to the project.

On Sat, Mar 2, 2019 at 3:31 PM Kevin Fenzi kevin@scrye.com wrote:

On 2/26/19 11:38 AM, Clement Verna wrote:

...

Hi all,

fedora-packages [0] code base is showing its age. The code base and the technology stack (Turbogears2 [1] web framework and the Moksha [2] middleware) is currently not ready for Python3 and I am not planning to do the work required to make it Python3 compatible, so the application will stop working when Fedora 29 is EOL.

In order to keep the service running, I have started a Proof Of Concept (fedora-search [3]) to replace the backend of the application. Fedora-search would be a REST API service offering full test search API. Such a service would then be available for other application to use, fedora-packages would then become a frontend only application using the service provided by fedora-search.

While the POC shows that this is a viable solution, I don't think that we should be proceeding that way, for the simple reason that this add yet another code base to maintain, I think we should use this opportunity to consider using Elasticsearch instead of maintaining our own "search engine".

I think that Elasticsearch offers quite a few advantages :

• Powerful Query language
• Python bindings
• Javascript bindings
• Can be deployed in our infrastructure or used as a service
• Can be useful for other applications ( docs.fp.o, pagure, ??)

So what is the general feeling about using Elasticsearch in our infrastructure ? Should we look at deploying a cluster in our infra / Should we approach the Council to see if we can get founding to have this service hosted by Elastic ?

Well, I am not sure this could really work, but I don't know enough about ES to say. What would the frontend look like?

I think users currently use packages to look at information about packages in a structured way. A "homepage" for the package if you will. I am sure ES could pull the rpm data and let people search it, but could it show a nice page with all the various types of information about a particular package? Or would it just say "here's the first page of your search results" and let users have to try and craft a search result that will find the info they are looking for? I think that would be a pretty big step back.

I agree with the points smooge brought up eariler about us running it (rpm difficult, etc), but now we do have openshift and it does ship with ES and might be easy to add to an app. But I am not sure if that helps if we still need a app to use it, xiapan isn't the issue here right?

Finally, I wonder if we could look at what other distros do? Could we work with say debian on theirs and add ability to theme it and handle rpm vs deb data?

openSUSE has a package search system[1] that can leverage AppStream data too, but I'm not sure how its backend works. I think it currently leverages OBS specific APIs.

As for Debian's package search system[2], it looks like a nightmare... :(

Mageia's App DB[3] is similar to openSUSE's system, but I can't quite tell if it's just only Mageia branded or relies on Mageia-specific things. It seems to have some aspects of Bodhi in it too...

As insane as it is, it looks like our existing system is the most distro agnostic approach I've seen! It's relatively easy to use it for any arbitrary set of repositories, and the branding is easy enough to swap out. It's mainly a matter of disabling the code to talk to Bodhi and simplify the package view page to remove the Fedora-specific bodhi and fedmsg integration stuff.

If it had the ability to handle Debian packages, it'd probably be a decent replacement for a number of these. But then again, our tool doesn't necessarily need to be aware of the difference between RPMs and DEBs, because it uses mdapi, and an alternative implementation for Debian repos would be sufficient to make it work.

[1]: https://github.com/openSUSE/software-o-o [2]: https://salsa.debian.org/webmaster-team/packages [3]: https://github.com/agallou/mageia-app-db

On 3/3/19 11:28 PM, Clement Verna wrote:

The frontend will be an independent application that we will need to develop or as suggested by Ryan we may explore using src.fp.o. Having a separated backend for search capabilities makes a lot of sense to me since that makes this service easy to reuse by other apps and let people write their custom client if they wish to do so.

So, really this is about replacing xiapan with ES, right? Since we will still need a app to interface with those results..

With the idea to have a separate search backend with HTTP API, xapian is a problem since we have to write the HTTP wrapper around xapian's API to make it available to all, there is xapiand (https://kronuz.io/Xapiand/) but the project seems fairly recent and not yet mature. The other benefits of Elasticsearch are a maintained Python client, a maintained Javascript client, better documentation, largely adopted solution (easier to get people interested in).

Yeah, all good points.

kevin

On 3/27/19 3:50 AM, Clement Verna wrote:

So the fun fact is that we are already running elasticsearch with kibana and logstash [0] in our Openshift clusters. So maybe we can leverage on how this is deployed to run a dedicated instance for fedora-packages.

Yeah.

However, it's pretty tailored to logs of pods. It would likely be easier to start with a dedicated elastic pod for searching packages, but then you would need to teach it how to gather the info and write a frontend for it. :(

kevin

On 3/28/19 8:54 AM, Clement Verna wrote:

On Thu, 28 Mar 2019 at 16:27, Kevin Fenzi kevin@scrye.com wrote:

...

On 3/27/19 3:50 AM, Clement Verna wrote:

...

So the fun fact is that we are already running elasticsearch with kibana and logstash [0] in our Openshift clusters. So maybe we can leverage on how this is deployed to run a dedicated instance for fedora-packages.

Yeah.

However, it's pretty tailored to logs of pods. It would likely be easier to start with a dedicated elastic pod for searching packages, but then you would need to teach it how to gather the info and write a frontend for it. :(

Yes, it does not look like we can easily reuse this. I think our best chance is to work on making src.fp.o the new central place for packages, but we need to improve the search functionality of Pagure. I personally don't think we will have enough time to work on making a fedora-packages 2.0 except if someone else is interested in working on it.

Do you think this could be a good project for outreachy or the like if you have time to mentor? Or is it too big for that kind of timeframe?

kevin