[PATCH 1/4] Add geolocation module

Martin Kolman mkolman at gmail.com
Wed Apr 3 15:45:58 UTC 2013


The geolocation module exposes an API that makes it possible to get current territory code using either GeoIP or alternativelly nearby WiFi accesspoints. The module has multiple location backends, the default backend is using the Fedora MirrorManager.
The geolocation module is a singleton, that is instantiated
by the init_geolocation() call and refreshes geolocation info
using a thread after refresh() is called.

Signed-off-by: Martin Kolman <mkolman at gmail.com>
---
 pyanaconda/constants.py |   1 +
 pyanaconda/geoloc.py    | 715 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 716 insertions(+)
 create mode 100644 pyanaconda/geoloc.py

diff --git a/pyanaconda/constants.py b/pyanaconda/constants.py
index bb59afc..fe570a7 100644
--- a/pyanaconda/constants.py
+++ b/pyanaconda/constants.py
@@ -119,3 +119,4 @@ THREAD_CHECK_SOFTWARE = "AnaCheckSoftwareThread"
 THREAD_SOURCE_WATCHER = "AnaSourceWatcher"
 THREAD_INSTALL = "AnaInstallThread"
 THREAD_CONFIGURATION = "AnaConfigurationThread"
+THREAD_GEOLOCATION_REFRESH = "AnaGeolocationRefreshThread"
diff --git a/pyanaconda/geoloc.py b/pyanaconda/geoloc.py
new file mode 100644
index 0000000..0ededfb
--- /dev/null
+++ b/pyanaconda/geoloc.py
@@ -0,0 +1,715 @@
+#
+# Copyright (C) 2013  Red Hat, Inc.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# the GNU General Public License v.2, or (at your option) any later version.
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY expressed or implied, including the implied warranties of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General
+# Public License for more details.  You should have received a copy of the
+# GNU General Public License along with this program; if not, write to the
+# Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.  Any Red Hat trademarks that are incorporated in the
+# source code or documentation are not subject to the GNU General Public
+# License and may only be used or replicated with the express permission of
+# Red Hat, Inc.
+#
+# Red Hat Author(s): Martin Kolman <mkolman at redhat.com>
+#
+
+"""
+A GeoIP and WiFi location module - location detection based on IP address
+
+How to use the geolocation module
+
+First call init_geolocation() - this creates the LocationInfo singleton and
+you can also use it to set what geolocation provider should be used.
+To actually look up current position, call refresh() - this will trigger
+the actual online geolocation query, which runs in a thread.
+After the look-up thread finishes, the results are stored in the singleton
+and can be retrieved using the get_territory_code() and get_result() methods.
+If you call these methods without calling refresh() first or if the look-up
+is currently in progress, both return None.
+
+Geolocation backends
+
+This module currently supports three geolocation backends:
+* Fedora MirrorManager
+* Hostip GeoIP
+* Google WiFi
+
+Fedora MirrorManager backend
+This is the default backend. It queries the Fedora mirror manager API for
+a list of most optimal mirrors and parses the result for country code.
+For production, it would be probably good to ask the MirrorManager people
+to provide a simple GeoIP API Anaconda can use, as the currently used one
+isn't originally meant for this usecase (so it might not be guaranteed to
+contain the country code in the future) and returns quite a lot of useless
+data (mirror URLs) that has to be computed by MirrorManager but is of no
+use to Anaconda.
+
+Hostip backend
+A GeoIP look-up backend that can be used to determine current country code
+from current public IP address. The public IP address is determined
+automatically when calling the API.
+GeoIP results from Hostip can contain more granularity than the results
+from MirrorManager (territory code only). They contain the current public IP
+and an approximate address. To get this detail location info, use the
+get_result() method to get an instance of the LocationResult class,
+wrapping the result.
+
+Google WiFi backend
+This backend is probably the most accurate one, at least as long as the
+computer has a working WiFi hardware and there are some WiFi APs nearby.
+It sends data about nearby APs (ssid, MAC address & signal strength)
+acquired from Network Manager to a Google API to get approximate
+geographic coordinates. If there are enough AP nearby (such as in a
+normal city) it can be very accurate, even up to currently determining
+which building is the computer currently in.
+But this only returns current geographic coordinates, to get country code
+the Nominatim reverse-geocoding API is called to convert the coordinates
+to an address, which includes a country code.
+While having many advantages, this backend also has some severe disadvantages:
+* needs working WiFi hardware
+* tells your public IP address & possibly quite precise geographic coordinates
+  to two external entities (Google and Nominatim)
+This could have severe privacy issues and should be carefully considered before
+enabling it to be used by default.
+* the Google WiFi geolocation API seems to lack official documentation
+As a result its long-term stability might not be guarantied.
+
+
+
+Possible issues with GeoIP
+
+"I'm in Switzerland connected to corporate VPN and anaconda tells me
+I'm in Netherlands."
+The public IP address is not directly mapped to the physical location
+of a computer. So while your world visible IP address is registered to
+an IP block assigned to an ISP in Netherlands, it is just the external
+address of the Internet gateway of  your corporate network.
+As VPNs and proxies can connect two computers anywhere on Earth,
+this issue is unfortunately probably unsolvable.
+
+
+Backends that could possibly be used in the future
+* GPS geolocation
++ doesn't leak your coordinates to a third party
+(not entirely true for assisted GPS)
+- unassisted cold GPS startup can take tens of minutes to acquire a GPS fix
++ assisted GPS startup (as used in most smartphones) can acquire a fix
+in a couple seconds
+* cell tower geolocation
+
+"""
+
+import urllib
+import urllib2
+import json
+import re
+import dbus
+import threading
+
+MIRRORMANAGER_GEOIP_PROVIDER = 1
+GOOGLE_WIFI_LOCATION = 2
+HOSTIP_GEOIP_PROVIDER = 3
+
+GEOCODER_NOMINATIM = 1
+
+DEFAULT_GEOCODER = GEOCODER_NOMINATIM
+DEFAULT_PROVIDER = MIRRORMANAGER_GEOIP_PROVIDER
+
+import gettext
+_ = lambda x: gettext.ldgettext("anaconda", x)
+P_ = lambda x, y, z: gettext.ldngettext("anaconda", x, y, z)
+
+import logging
+log = logging.getLogger("anaconda")
+
+from pyanaconda import constants
+from pyanaconda.threads import AnacondaThread, threadMgr
+from pyanaconda import nm
+
+location_info_instance = None
+
+
+def init_geolocation(provider=DEFAULT_PROVIDER):
+    """
+    Prepare the geolocation module for handling geolocation queries.
+    This method sets-up the GeoLocation instance with the given
+    geolocation_provider (or using the default one if no provider
+    is given. Please note that calling this method doesn't actually
+    execute any queries by itself, you need to call refresh()
+    to do that.
+
+    @provider specifies what geolocation backend to use
+    """
+
+    global location_info_instance
+    location_info_instance = LocationInfo(provider_id=provider)
+
+
+def refresh():
+    """
+    Refresh information about current location using the currently specified
+    geolocation provider.
+    """
+    if location_info_instance:
+        location_info_instance.refresh()
+    else:
+        raise GeolocationError("refresh() called before init_geolocation()")
+
+
+def get_territory_code():
+    """
+    Returns the current country code or None if it is not known.
+    None might also be returned even if refresh was called, but
+    the look-up is still in progress.
+
+    @return current country code or None if not known
+    """
+    if location_info_instance:
+        return location_info_instance.get_territory_code()
+    else:
+        raise GeolocationError("init_geolocation() not called")
+        return None
+
+
+def get_result():
+    """
+    Returns the current geolocation result wrapper or None
+    if no information about current location is known.
+    None might also be returned even if refresh was called, but
+    the look-up is still in progress.
+
+    @return LocationResult instance or None if location is unknown
+    """
+    if location_info_instance:
+        return location_info_instance.get_result()
+    else:
+        raise GeolocationError("init_geolocation() not called!")
+
+
+def _get_provider(provider):
+    """
+    Return GeoIP provider instance based on the provider ID
+    """
+    if provider == MIRRORMANAGER_GEOIP_PROVIDER:
+        return MirrorManagerGeoIPProvider()
+    elif provider == GOOGLE_WIFI_LOCATION:
+        return GoogleWiFiLocationProvider()
+    elif provider == HOSTIP_GEOIP_PROVIDER:
+        return HostipGeoIPProvider()
+
+
+class GeolocationError(Exception):
+    """Exception class for geolocation related errors"""
+    pass
+
+
+class LocationInfo(object):
+    """
+    Determines current location based on IP address or
+    nearby WiFi access points (depending on what backend is used)
+    """
+
+    def __init__(self, provider_id=DEFAULT_PROVIDER, refresh_now=False):
+        """
+        @param provider_id: GeoIP provider id specified by module constant
+        @param refresh_now: if True, a GeoIP information refresh will be
+                    done once the class is initialized
+        """
+        self._provider = None
+        self._provider = _get_provider(provider_id)
+        if refresh_now:
+            self.refresh()
+
+    def refresh(self):
+        """
+        Refresh location info
+        """
+        # first check if a provider is available
+        if self._provider is None:
+            raise GeolocationError("can't refresh - no provider")
+            return
+
+        # then check if a refresh is already in progress
+        if threadMgr.get(constants.THREAD_GEOLOCATION_REFRESH):
+            log.debug("Geoloc: refresh already in progress")
+        else:  # wait for Internet connectivity
+            if self._wait_for_connectivity():
+                threadMgr.add(AnacondaThread(
+                    name=constants.THREAD_GEOLOCATION_REFRESH,
+                    target=self._provider.refresh))
+            else:
+                log.error(_("Geolocation refresh failed"
+                          " - no connectivity"))
+
+    def _wait_for_connectivity(self):
+        """
+        Wait for Internet connectivity to become available.
+        @return True is connectivity is available, False otherwise
+        """
+        # wait for the thread that waits for NM to connect
+        threadMgr.wait(constants.THREAD_WAIT_FOR_CONNECTING_NM)
+        # then check if NM connected successfully
+        return nm.nm_is_connected()
+
+    def get_result(self):
+        """
+        Get result from the provider
+        @return the result object,
+        returns None if no results are available
+        """
+        return self._provider.get_result()
+
+    def get_territory_code(self):
+        """
+        a convenience function for getting the territory code,
+        returns None if no results are available
+        """
+        result = self._provider.get_result()
+        if result:
+            return result.territory_code
+        else:
+            return None
+
+    def get_public_ip_address(self):
+        """
+        a convenience function for getting current public IP,
+        returns None if no results are available
+        """
+        result = self._provider.get_result()
+        if result:
+            return result.public_ip_address
+        else:
+            return None
+
+
+class LocationResult(object):
+    def __init__(self, territory_code=None, public_ip_address=None, city=None):
+        """
+        Encapsulates the result from GeoIP lookup.
+
+        @param territory_code the territory code from GeoIP lookup
+        @param public_ip_address current public IP address
+        @param city current city
+        """
+        self._territory_code = territory_code
+        self._public_ip_address = public_ip_address
+        self._city = city
+
+    @property
+    def territory_code(self):
+        return self._territory_code
+
+    @property
+    def public_ip_address(self):
+        return self._public_ip_address
+
+    @property
+    def city(self):
+        return self._city
+
+    def __str__(self):
+        if self.territory_code:
+            result_string = _("territory: %s") % self.territory_code
+            if self.public_ip_address:
+                result_string += _("\npublic IP address: "
+                                   "%s") % self.public_ip_address
+            if self.city:
+                result_string += _("\ncity: %s") % self.city
+            return result_string
+        else:
+            return _("Position unknown")
+
+
+class GeolocationBackend(object):
+    """
+    Base class for GeoIP backends.
+    """
+    def __init__(self):
+        self._result = None
+        self._result_lock = threading.Lock()
+
+    def get_name(self):
+        """
+        @return name of the backend
+        """
+        pass
+
+    def refresh(self, force=False):
+        # check if refresh is needed
+        if force is True or self._result is None:
+            log.info(_("Starting GeoIP lookup"))
+            self._refresh()  # refresh even if a result is available
+            log.info(_("GeoIP lookup finished"))
+            result = self.get_result()
+            if result:
+                log.info("%s" % result)
+            else:
+                log.info(_("no results"))
+
+    def _refresh(self):
+        pass
+
+    def _set_result(self, result):
+        """Set current location"""
+        # As the value is set from a thread but read from
+        # the main thread, use a lock when acessing it
+        with self._result_lock:
+            self._result = result
+
+    def get_result(self):
+        """Get current location"""
+        with self._result_lock:
+            return self._result
+
+    def __str__(self):
+        return self.get_name()
+
+
+class MirrorManagerGeoIPProvider(GeolocationBackend):
+    """
+    The Fedora GeoIP service provider
+    """
+
+    API_URL = "https://mirrors.fedoraproject.org/" \
+              "mirrorlist?repo=fedora-18&arch=i386"
+
+    def __init__(self):
+        GeolocationBackend.__init__(self)
+
+    def get_name(self):
+        return "MirrorManager"
+
+    def _refresh(self):
+        try:
+            reply = urllib2.urlopen(self.API_URL)
+            if reply:
+                territory = re.findall("country = ([A-Z]*)", reply.readline())
+                if territory:
+                    self._set_result(LocationResult(
+                        territory_code=territory[0]))
+        except urllib2.URLError as e:
+            log.debug("Geoloc: URLError during FMM lookup:\n%s" % e)
+
+
+class HostipGeoIPProvider(GeolocationBackend):
+    """
+    The Hostip GeoIP service provider
+    """
+
+    API_URL = "http://api.hostip.info/get_json.php"
+
+    def __init__(self):
+        GeolocationBackend.__init__(self)
+
+    def get_name(self):
+        return "Hostip"
+
+    def _refresh(self):
+        try:
+            reply = urllib2.urlopen(self.API_URL)
+            if reply:
+                reply_dict = json.load(reply)
+                territory = reply_dict.get("country_code", None)
+
+                # unless at least country_code is available,
+                # we don't return any results
+                if territory is not None:
+                    self._set_result(LocationResult(
+                        territory_code=territory,
+                        public_ip_address=reply_dict.get("ip", None),
+                        city=reply_dict.get("city", None)
+                    ))
+        except urllib2.URLError as e:
+            log.debug("Geoloc: URLError during Hostip lookup:\n%s" % e)
+
+
+class GoogleWiFiLocationProvider(GeolocationBackend):
+    """
+    The Google WiFi location service provider
+    """
+
+    API_URL = "https://maps.googleapis.com/" \
+              "maps/api/browserlocation/json?browser=firefox&sensor=true"
+
+    def __init__(self):
+        GeolocationBackend.__init__(self)
+
+    def get_name(self):
+        return "Google"
+
+    def _refresh(self):
+        log.info(_("Scanning for WiFi access points."))
+        scanner = WifiScanner(scan_now=True)
+        access_points = scanner.get_results()
+        if access_points:
+            try:
+                url = self._get_url(access_points)
+                reply = urllib2.urlopen(url)
+                result_dict = json.load(reply)
+                status = result_dict.get('status', 'NOT OK')
+                if status == 'OK':
+                    lat = result_dict['location']['lat']
+                    lon = result_dict['location']['lng']
+                    log.info(_("Found current location."))
+                    coords = Coordinates(lat=lat, lon=lon)
+                    geocoder = Geocoder()
+                    geocoding_result = geocoder.reverse_geocode_coords(coords)
+                    # for compatibility, return GeoIP result instead
+                    # of GeocodingResult
+                    t_code = geocoding_result.territory_code
+                    self._set_result(LocationResult(territory_code=t_code))
+                else:
+                    log.info(_("Service couldn't find current location."))
+            except urllib2.URLError as e:
+                log.debug("Geoloc: URLError during Google"
+                          "  Wifi lookup:\n%s" % e)
+        else:
+            log.info(_("No WiFi access points found - can't detect location."))
+
+    def _get_url(self, access_points):
+        """
+        generate Google API URL for the given access points
+        @param access_points: a list of WiFiAccessPoint objects
+        @return Google WiFi location API URL
+        """
+        url = self.API_URL
+        for ap in access_points:
+            url += self._describe_access_point(ap)
+        return url
+
+    def _describe_access_point(self, access_point):
+        """
+        Describe an access point in a format compatible with the API call
+
+        @param access_point: an WiFiAccessPoint instance
+        @return API compatible AP description
+        """
+        quoted_ssid = urllib.quote_plus(access_point.ssid)
+        return "&wifi=mac:%s|ssid:%s|ss:%d" % (access_point.bssid,
+                                               quoted_ssid, access_point.rssi)
+
+
+class Geocoder(object):
+    """
+    Provides online geociding services (only reverse geocoding at the moment).
+    """
+
+    # MapQuest Nominatim instance without (?) rate limiting
+    NOMINATIM_API_URL = "http://open.mapquestapi.com/" \
+                        "nominatim/v1/reverse.php?format=json"
+    # Alternative OSM hosted Nominatim instance (with rate limiting):
+    # http://nominatim.openstreetmap.org/reverse?format=json
+
+    def __init__(self, geocoder=DEFAULT_GEOCODER):
+        self._geocoder = geocoder
+
+    def reverse_geocode_coords(self, coordinates):
+        """
+        Turn geographic coordinates to address
+
+        @param coordinates: Coordinates (geographic coordinates)
+        @return: GeocodingResult if the lookup succeeds or None if it fails
+        """
+        if self._geocoder == GEOCODER_NOMINATIM:
+            return self._reverse_geocode_nominatim(coordinates)
+        else:
+            log.error(_("Wrong Geocoder specified!"))
+            return None  # unknown geocoder specified
+
+    def _reverse_geocode_nominatim(self, coordinates):
+        """reverse geocoding using the Nominatim API"""
+        url = "%s&addressdetails=1&lat=%f&lon=%f" % (
+            self.NOMINATIM_API_URL,
+            coordinates.latitude,
+            coordinates.longitude)
+        try:
+            reply = urllib2.urlopen(url)
+            if reply:
+                reply_dict = json.load(reply)
+                territory_code = reply_dict['address']['country_code'].upper()
+                return GeocodingResult(coordinates=coordinates,
+                                       territory_code=territory_code)
+            else:
+                return None
+        except urllib2.URLError as e:
+            log.debug("Geoloc: URLError during Nominatim reverse geocoding"
+                      " :\n%s" % e)
+
+
+class GeocodingResult(object):
+    """A result from geocoding lookup"""
+
+    def __init__(self, coordinates=None, territory_code=None, address=None):
+        """
+        @param coords: geographic coordinates
+        @param territory_code: teritory code of the result
+        @param address: a (street) address string
+        """
+        self._coords = coordinates
+        self._territory_code = territory_code
+        self._address = address
+
+    @property
+    def coordinates(self):
+        return self._coords
+
+    @property
+    def territory_code(self):
+        return self._territory_code
+
+    @property
+    def address(self):
+        return self._address
+
+
+class Coordinates(object):
+    """
+    A set of geographic coordinates.
+    """
+    def __init__(self, lat=None, lon=None):
+        """
+        @param lat: WGS84 latitude
+        @param lon: WGS84 longitude
+        """
+        self._lat = lat
+        self._lon = lon
+
+    @property
+    def latitude(self):
+        return self._lat
+
+    @property
+    def longitude(self):
+        return self._lon
+
+    def __str__(self):
+        return "lat,lon: %f,%f" % (self.latitude, self.longitude)
+
+
+class WifiScanner(object):
+    """
+    Uses the Network Manager DBUS API to provide information
+    about nearby WiFi access points
+    """
+
+    NETWORK_MANAGER_DEVICE_TYPE_WIFI = 2
+
+    def __init__(self, scan_now=True):
+        """
+        @param scan_now: if an initial scan should be done
+        """
+        self._scan_results = []
+        if scan_now:
+            self.scan()
+
+    def scan(self):
+        """
+        Scan for WiFi access points
+        """
+        devices = ""
+        access_points = []
+        # connect to network manager
+        try:
+            bus = dbus.SystemBus()
+            network_manager = bus.get_object('org.freedesktop.NetworkManager',
+                                             '/org/freedesktop/NetworkManager')
+            devices = network_manager.GetDevices()
+        except Exception as e:
+            log.debug("Exception caught during WiFi AP scan: %s" % e)
+        # iterate over all devices
+        for device_path in devices:
+            device = bus.get_object('org.freedesktop.NetworkManager',
+                                    device_path)
+            # get type of the device
+            device_type = device.Get("org.freedesktop.NetworkManager.Device",
+                                     'DeviceType')
+            if device_type == self.NETWORK_MANAGER_DEVICE_TYPE_WIFI:
+                # iterate over all APs
+                for ap_path in device.GetAccessPoints():
+                    network = bus.get_object('org.freedesktop.NetworkManager',
+                                             ap_path)
+                    network_properties = dbus.Interface(
+                        network,
+                        dbus_interface='org.freedesktop.DBus.Properties')
+                    bssid = str(network_properties.Get(
+                        "org.freedesktop.NetworkManager.AccessPoint",
+                        "HwAddress"))
+                    essid = str(network_properties.Get(
+                                "org.freedesktop.NetworkManager.AccessPoint",
+                                "Ssid", byte_arrays=True))
+                    rssi = int(network_properties.Get(
+                        "org.freedesktop.NetworkManager.AccessPoint",
+                        "Strength"))
+                    #print bssid, essid, rssi
+                    ap = WiFiAccessPoint(bssid=bssid, ssid=essid, rssi=rssi)
+                    access_points.append(ap)
+        self._scan_results = access_points
+
+    def get_results(self):
+        """
+        @return: returns a list of WiFiAccessPoint objects,
+        an empty list if no APs were found or the scan failed
+        """
+        return self._scan_results
+
+
+class WiFiAccessPoint(object):
+    """Encapsulates information about WiFi access point"""
+
+    def __init__(self, bssid, ssid=None, rssi=None):
+        """
+        @param bssid: MAC address of the access point
+        @param ssid: name of the access point
+        @param rssi: signal strength
+        """
+        self._bssid = bssid
+        self._ssid = ssid
+        self._rssi = rssi
+
+    @property
+    def bssid(self):
+        return self._bssid
+
+    @property
+    def ssid(self):
+        return self._ssid
+
+    @property
+    def rssi(self):
+        return self._rssi
+
+    def __str__(self):
+        return "bssid (MAC): %s ssid: %s rssi " \
+               "(signal strength): %d" % (self.bssid, self.ssid, self.rssi)
+
+if __name__ == "__main__":
+    print "GeoIP directly started"
+
+    print "trying the default backend"
+    location_info = LocationInfo()
+    location_info.refresh()
+    print "  provider used: %s" % location_info._provider
+    print "  territory code: %s" % location_info.get_territory_code()
+
+    print "trying the Fedora MirrorManager backend"
+    location_info = LocationInfo(provider_id=MIRRORMANAGER_GEOIP_PROVIDER)
+    location_info.refresh()
+    print "  provider used: %s" % location_info._provider
+    print "  territory code: %s" % location_info.get_territory_code()
+
+    print "trying the Google WiFi location backend"
+    location_info = LocationInfo(provider_id=GOOGLE_WIFI_LOCATION)
+    location_info.refresh()
+    print "  provider used: %s" % location_info._provider
+    print "  territory code: %s" % location_info.get_territory_code()
+
+    print "trying the Hostip backend"
+    location_info = LocationInfo(provider_id=HOSTIP_GEOIP_PROVIDER)
+    location_info.refresh()
+    print "  provider used: %s" % location_info._provider
+    print "  territory code: %s" % location_info.get_territory_code()
-- 
1.8.1.4



More information about the anaconda-patches mailing list