On Fri, Dec 06, 2019 at 04:49:15PM +0000, Tom Hughes wrote:
On 06/12/2019 16:41, Martin Kolman wrote:
On Fri, 2019-12-06 at 08:38 -0600, Chris Adams wrote:
I also installed the Fedora 31 GeoIP packages and ran the geoipupdate, and that DB has the correct info.
IIRC the infra team mentioned some issues with the new geoip database being incompatible with how the service is currently implemented, resulting in being stuck with an outdated database until this is resolved.
Sounds like it maybe doesn't have support for GeoLite2 and is using the old MaxMind GeoLite Legacy databases which haven't been updated since the start of this year.
Yes, that is exactly the case.
We recently looked at this to see if we could retire the service, but it looks like it's still needed, so we need to figure out how to get cycles to update it.
If someone wants to work on this, let us know!
kevin
I’m interested in looking at this. Is this the script?
https://github.com/fedora-infra/geoip-city-wsgi/blob/master/geoip-city.wsgi
Assuming that’s the one, it looks like we need to replace GeoIP with geoip2 and wrap the geoip2 data in a structure to match what is currently being returned. Let me know if I’m missing anything.
-- kevin
On Dec 6, 2019, at 6:39 PM, Kevin Fenzi kevin@scrye.com wrote:
On Fri, Dec 06, 2019 at 04:49:15PM +0000, Tom Hughes wrote:
On 06/12/2019 16:41, Martin Kolman wrote:
On Fri, 2019-12-06 at 08:38 -0600, Chris Adams wrote: I also installed the Fedora 31 GeoIP packages and ran the geoipupdate, and that DB has the correct info.
IIRC the infra team mentioned some issues with the new geoip database being incompatible with how the service is currently implemented, resulting in being stuck with an outdated database until this is resolved.
Sounds like it maybe doesn't have support for GeoLite2 and is using the old MaxMind GeoLite Legacy databases which haven't been updated since the start of this year.
Yes, that is exactly the case.
We recently looked at this to see if we could retire the service, but it looks like it's still needed, so we need to figure out how to get cycles to update it.
If someone wants to work on this, let us know!
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
-- kevin
On Dec 6, 2019, at 7:13 PM, Kevin Sandy kevin@digitallotus.com wrote:
I’m interested in looking at this. Is this the script?
https://github.com/fedora-infra/geoip-city-wsgi/blob/master/geoip-city.wsgi
Assuming that’s the one, it looks like we need to replace GeoIP with geoip2 and wrap the geoip2 data in a structure to match what is currently being returned. Let me know if I’m missing anything.
-- kevin
On Dec 6, 2019, at 6:39 PM, Kevin Fenzi kevin@scrye.com wrote:
On Fri, Dec 06, 2019 at 04:49:15PM +0000, Tom Hughes wrote:
On 06/12/2019 16:41, Martin Kolman wrote:
On Fri, 2019-12-06 at 08:38 -0600, Chris Adams wrote:
I also installed the Fedora 31 GeoIP packages and ran the geoipupdate, and that DB has the correct info.
IIRC the infra team mentioned some issues with the new geoip database being incompatible with how the service is currently implemented, resulting in being stuck with an outdated database until this is resolved.
Sounds like it maybe doesn't have support for GeoLite2 and is using the old MaxMind GeoLite Legacy databases which haven't been updated since the start of this year.
Yes, that is exactly the case.
We recently looked at this to see if we could retire the service, but it looks like it's still needed, so we need to figure out how to get cycles to update it.
If someone wants to work on this, let us know!
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Do you need some IP's with known location to test?
I went ahead and published the updated copy on a test site, so feel free to hit it and test it out - http://geoip.digitallotus.com/. It’s also accepting the X-Forwarded-For header and doing the lookup on that if set, so I’ve been testing it with curl, e.g.:
curl -s -H 'X-Forwarded-For: 1.180.10.10' http://geoip.digitallotus.com/ | jq .
-- kevin
On Dec 7, 2019, at 1:10 PM, Tomasz Torcz tomek@pipebreaker.pl wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Do you need some IP's with known location to test?
-- Tomasz Torcz “If you try to upissue this patchset I shall be seeking tomek@pipebreaker.pl an IP-routable hand grenade.” — Andrew Morton (LKML) _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
Thanks again for working on this!
kevin --
-- kevin
On Dec 6, 2019, at 7:13 PM, Kevin Sandy kevin@digitallotus.com wrote:
I’m interested in looking at this. Is this the script?
https://github.com/fedora-infra/geoip-city-wsgi/blob/master/geoip-city.wsgi
Assuming that’s the one, it looks like we need to replace GeoIP with geoip2 and wrap the geoip2 data in a structure to match what is currently being returned. Let me know if I’m missing anything.
-- kevin
On Dec 6, 2019, at 6:39 PM, Kevin Fenzi kevin@scrye.com wrote:
On Fri, Dec 06, 2019 at 04:49:15PM +0000, Tom Hughes wrote:
On 06/12/2019 16:41, Martin Kolman wrote:
On Fri, 2019-12-06 at 08:38 -0600, Chris Adams wrote:
I also installed the Fedora 31 GeoIP packages and ran the geoipupdate, and that DB has the correct info.
IIRC the infra team mentioned some issues with the new geoip database being incompatible with how the service is currently implemented, resulting in being stuck with an outdated database until this is resolved.
Sounds like it maybe doesn't have support for GeoLite2 and is using the old MaxMind GeoLite Legacy databases which haven't been updated since the start of this year.
Yes, that is exactly the case.
We recently looked at this to see if we could retire the service, but it looks like it's still needed, so we need to figure out how to get cycles to update it.
If someone wants to work on this, let us know!
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
Thanks again for working on this!
kevin
Sounds good. I submitted the PR on GitHub yesterday. I’ve also included the git patch below.
--- geoip-city.wsgi | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
diff --git a/geoip-city.wsgi b/geoip-city.wsgi index 9435820..4f9d4d7 100755 --- a/geoip-city.wsgi +++ b/geoip-city.wsgi @@ -13,12 +13,13 @@
from string import zfill, atoi, strip, replace from paste.wsgiwrappers import * -import GeoIP +from iso3166 import countries +import geoip2.database +import geoip2.errors import json
global gi -gi = GeoIP.open("/usr/share/GeoIP/GeoLiteCity.dat", GeoIP.GEOIP_STANDARD) -gi.set_charset(GeoIP.GEOIP_CHARSET_UTF8) +gi = geoip2.database.Reader("/usr/share/GeoIP/GeoLite2-City.mmdb")
def real_client_ip(xforwardedfor): @@ -44,6 +45,7 @@ def get_client_ip(environ, request): def application(environ, start_response): request = WSGIRequest(environ) response = WSGIResponse() + results = {} code = 500
try: @@ -51,15 +53,39 @@ def application(environ, start_response): if client_ip is None: code = 400 raise Exception - results = gi.record_by_addr(client_ip) - if results is None: + data = gi.city(client_ip) + if data is None: code = 404 raise Exception + except geoip2.errors.AddressNotFoundError: + response.status_code = 404 + return response(environ, start_response) except: response.status_code=code return response(environ, start_response)
results['ip'] = client_ip + + # map geoip2 data to a structure that matches the prior geoip format + results['city'] = data.city.name + results['region_name'] = data.subdivisions.most_specific.name + results['region'] = data.subdivisions.most_specific.iso_code + results['postal_code'] = data.postal.code + results['country_name'] = data.country.name + results['country_code'] = data.country.iso_code + results['time_zone'] = data.location.time_zone + results['latitude'] = data.location.latitude + results['longitude'] = data.location.longitude + results['metro_code'] = data.location.metro_code + results['dma_code'] = data.location.metro_code + + # geoip2 no longer includes country_code3, so it has to be pulled + # from iso3166.countries + if data.country.iso_code in countries: + results['country_code3'] = countries[data.country.iso_code].alpha3 + else: + results['country_code3'] = None + results = json.dumps(results) response.headers['Content-Length'] = str(len(results)) response.write(results)
On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
I had a look at the PR (https://github.com/fedora-infra/geoip-city-wsgi/pull/1) and it works and the output is almost the same. Maybe one of the known data consumers can have a short look if those subtle changes in the output are not a problem.
Adrian
On Fri, Dec 13, 2019 at 08:00:01AM +0100, Adrian Reber wrote:
On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
I had a look at the PR (https://github.com/fedora-infra/geoip-city-wsgi/pull/1) and it works and the output is almost the same. Maybe one of the known data consumers can have a short look if those subtle changes in the output are not a problem.
The main user I know of is anaconda and it sounds like the changes work for them.
I guess I'd say we should pick a date (likely after the new year now), announce we are going to be changing it then a few times, and then on that date cut over to the new system.
kevin
On Fri, 2019-12-13 at 09:21 -0800, Kevin Fenzi wrote:
On Fri, Dec 13, 2019 at 08:00:01AM +0100, Adrian Reber wrote:
On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
I had a look at the PR (https://github.com/fedora-infra/geoip-city-wsgi/pull/1) and it works and the output is almost the same. Maybe one of the known data consumers can have a short look if those subtle changes in the output are not a problem.
The main user I know of is anaconda and it sounds like the changes work for them.
I guess I'd say we should pick a date (likely after the new year now), announce we are going to be changing it then a few times, and then on that date cut over to the new system.
Sounds good to me (as an Anaconda developer)! :)
BTW, will the new API be available somewhere before that (stagin, etc.) ?
We could patch Anaconda to talk to the new API & do some smoke testing, just in case. :)
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Wed, Dec 18, 2019 at 01:29:03PM +0100, Martin Kolman wrote:
On Fri, 2019-12-13 at 09:21 -0800, Kevin Fenzi wrote:
On Fri, Dec 13, 2019 at 08:00:01AM +0100, Adrian Reber wrote:
On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Thanks for diving in and getting this going.
In another email thread someone, Adrian said he had some code to do this, but if you already have it working, perhaps he could review your code/setup and we could then push it into ansible?
I had a look at the PR (https://github.com/fedora-infra/geoip-city-wsgi/pull/1) and it works and the output is almost the same. Maybe one of the known data consumers can have a short look if those subtle changes in the output are not a problem.
The main user I know of is anaconda and it sounds like the changes work for them.
I guess I'd say we should pick a date (likely after the new year now), announce we are going to be changing it then a few times, and then on that date cut over to the new system.
Sounds good to me (as an Anaconda developer)! :)
BTW, will the new API be available somewhere before that (stagin, etc.) ?
We could patch Anaconda to talk to the new API & do some smoke testing, just in case. :)
We finally pushed it to Fedora's staging environment. Please try it using https://geoip.stg.fedoraproject.org/city if you have a chance.
Adrian
On Thu, Apr 09, 2020 at 04:55:20PM +0200, Adrian Reber wrote:
We finally pushed it to Fedora's staging environment. Please try it using https://geoip.stg.fedoraproject.org/city if you have a chance.
It seems to have only IPv4 address. It provides correct city in my case, but IPv6 testing would be nice, too.
On Thu, 9 Apr 2020 at 12:35, Tomasz Torcz tomek@pipebreaker.pl wrote:
On Thu, Apr 09, 2020 at 04:55:20PM +0200, Adrian Reber wrote:
We finally pushed it to Fedora's staging environment. Please try it using https://geoip.stg.fedoraproject.org/city if you have a chance.
It seems to have only IPv4 address. It provides correct city in my case, but IPv6 testing would be nice, too.
staging is ipv4 only.
-- Tomasz Torcz “(…) today's high-end is tomorrow's embedded processor.” tomek@pipebreaker.pl — Mitchell Blank on LKML _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
I have an updated copy working with geoip2, with the exception of area code data (which isn’t included in the new database). But if I’m understanding it’s purpose correctly, telephone area code isn’t needed. I need to do some testing with IPs around the world to ensure that the data is sane outside the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
Wow. Awesome work!
Awesome indeed. Please enjoy the Blessing of the FPL badge. :)
https://badges.fedoraproject.org/user/ks3
infrastructure@lists.fedoraproject.org
-
Adrian Reber -
Kevin Fenzi -
Kevin Sandy -
Martin Kolman -
Matthew Miller -
Stephen John Smoogen -
Tomasz Torcz