Giter VIP home page Giter VIP logo

scala-maxmind-iplookups's Introduction

Scala MaxMind IP Lookups

Build Status Maven Central Coverage Status Gitter

Introduction

This is a Scala wrapper for the MaxMind Java Geo-IP2 library. The main benefits of using this wrapper over directly calling the Java library from Scala are:

  1. Provides a common interface to four MaxMind databases - it works with MaxMind's databases for looking up geographic location, ISP, domain, and connection type from an IP address
  2. Better type safety - the MaxMind Java library is somewhat null-happy. This wrapper uses Option-boxing wherever possible
  3. Better performance - as well as or instead of using MaxMind's own caching (CHMCache), you can also configure an LRU (Least Recently Used) cache of variable size

Installation

The latest version of scala-maxmind-iplookups is 0.8.1 and is compatible with Scala 2.13.

Add this to your SBT config:

val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0.8.1"

Retrieve the GeoLite2-City.mmdb file from the MaxMind downloads page (direct link).

MaxMind also has databases for looking up ISPs, domain names, and connection types from IP addresses. Scala MaxMind IP Lookups supports all of these.

Usage

See the scaladoc for the full API reference.

Here is a simple usage example, performing just a geographic lookup and not the ISP, domain, or connection type lookups:

import cats.effect.IO
import com.snowplowanalytics.maxmind.iplookups.IpLookups

val result = (for {
  ipLookups <- CreateIpLookups[IO].createFromFilenames(
    geoFile = Some("/opt/maxmind/GeoLite2-City.mmdb"),
    ispFile = None,
    domainFile = None,
    connectionTypeFile = None,
    memCache = false,
    lruCacheSize = 20000
  )
  lookup <- ipLookups.performLookups("175.16.199.0")
} yield lookup).unsafeRunSync()

result.ipLocation match {
  case Some(Right(loc)) =>
    println(loc.countryCode)   // => "CN"
    println(loc.countryName)   // => "China"
  case _ =>
    println("Lookup failed")
}

cats.Id is also supported:

import cats.Id

val idResult: IpLookupResult = {
  val ipLookups = CreateIpLookups[Id].createFromFilenames(
    geoFile = Some("/opt/maxmind/GeoLite2-City.mmdb")
    ispFile = None,
    domainFile = None,
    connectionTypeFile = None,
    memCache = false,
    lruCacheSize = 20000
  )
  ipLookups.performLookups("175.16.199.0")
}

Note that GeoLite2-City.mmdb is updated by MaxMind each month.

For further usage examples for Scala MaxMind IP Lookups, please see the tests in IpLookupsTest.scala. The test suite uses test databases provided by MaxMind.

Implementation details

IpLookups constructor

The signature is as follows:

final case class IpLookups(
  geoFile: Option[File],
  ispFile: Option[File],
  domainFile: Option[File],
  connectionTypeFile: Option[File],
  memCache: Boolean = true,
  lruCache: Int = 10000
)

CreateIpLookups proposes an alternative constructor which takes Option[String] as file paths to the databases instead:

def createFromFilenames(
  geoFile: Option[String],
  ispFile: Option[String],
  domainFile: Option[String],
  connectionTypeFile: Option[String],
  memCache: Boolean = true,
  lruCache: Int = 10000
)

The first four arguments are the MaxMind databases from which the lookup should be performed. geoFile, ispFile, domainFile, and connectionTypeFile refer respectively to MaxMind's databases for looking up location, ISP, domain, and connection type based on an IP address. They are all wrapped in Option, so if you don't have access to all of them, just pass in None as in the example above.

In both signatures, the memCache flag is set to true by default. This flag enables MaxMind's own caching (CHMCache).

The lruCache value defaults to 10000 - meaning Scala MaxMind IP Lookups will maintain an LRU cache of 10,000 values, which it will check prior to making a MaxMind lookup. To disable the LRU cache, set its size to zero, i.e. lruCache = 0.

Returned value

The performLookups(ip) method returns a:

final case class IpLookupResult(
  ipLocation: Option[Either[Throwable, IpLocation]],
  isp: Option[Either[Throwable, String]],
  organization: Option[Either[Throwable, String]],
  domain: Option[Either[Throwable, String]],
  connectionType: Option[Either[Throwable, String]]
)

The first element is the result of the geographic location lookup. It is either None (if no geographic lookup database was provided) or Some(ipLocation), where ipLocation is an instance of the IpLocation case class described below. The other three elements in the tuple are Options wrapping the results of the other four possible lookups: ISP, organization, domain, and connection type.

Note that enabling providing an ISP database will return an organization in addition to an isp.

IpLocation case class

The geographic lookup returns an IpLocation case class instance with the following structure:

final case class IpLocation(
  countryCode: String,
  countryName: String,
  region: Option[String],
  city: Option[String],
  latitude: Float,
  longitude: Float,
  timezone: Option[String],
  postalCode: Option[String],
  metroCode: Option[Int],
  regionName: Option[String],
  isInEuropeanUnion: Boolean,
  continent: String,
  accuracyRadius: Int
)

An example using multiple databases

This example shows how to do a lookup using all four databases.

import com.snowplowanalytics.maxmind.iplookups.IpLookups

val lookupResult = (for {
  ipLookups <- CreateIpLookups[IO].createFromFilenames(
    geoFile = Some("/opt/maxmind/GeoLite2-City.mmdb"),
    ispFile = Some("/opt/maxmind/GeoIP2-ISP.mmdb"),
    domainFile = Some("/opt/maxmind/GeoIP2-Domain.mmdb"),
    connectionType = Some("/opt/maxmind/GeoIP2-Connection-Type.mmdb"),
    memCache = false,
    lruCache = 10000
  )
  lookupResult <- ipLookups.performLookups("70.46.123.145")
} yield lookupResult).unsafeRunSync()

// Geographic lookup
println(lookupResult.ipLocation).map(_.countryName) // => Some(Right("United States"))
println(lookupResult.ipLocation).map(_.regionName)  // => Some(Right("Florida"))

// ISP lookup
println(lookupResult.isp) // => Some(Right("FDN Communications"))

// Organization lookup
println(lookupResult.organization) // => Some(Right("DSLAM WAN Allocation"))

// Domain lookup
println(lookupResult.domain) // => Some(Right("nuvox.net"))

// Connection type lookup
println(lookupResult.connectionType) // => Some(Right("Dialup"))

LRU cache

We recommend trying different LRU cache sizes to see what works best for you.

Building etc

Assuming you already have SBT installed:

$ git clone git://github.com/snowplow/scala-maxmind-iplookups.git
$ cd scala-maxmind-iplookups
$ sbt test
<snip>
[info] Passed: : Total 276, Failed 0, Errors 0, Passed 276, Skipped 0

GeoLite Legacy discontinuation

Maxmind are discontinuing updates for the GeoLite Legacy databases in April 2018.

We will be discontinuing updates to the GeoLite Legacy databases as of April 1, 2018. You will still be able to download the April 2018 release until January 2, 2019. GeoLite Legacy users will need to update their integrations in order to switch to the free GeoLite2 or commercial GeoIP databases by April 2018.

For more information, please visit our Support Center.

In addition, in 2019, latitude and longitude coordinates in the GeoLite2 databases will be removed.* Latitude and longitude coordinates will continue to be provided in GeoIP2 databases. Please check back for updates.

As such we recommend upgrading to version 0.4.0 as soon as possible

Copyright and license

Copyright 2012-2022 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

scala-maxmind-iplookups's People

Contributors

alexanderdean avatar andrusha avatar benfradet avatar chuwy avatar csaltos avatar dilyand avatar fblundun avatar hiiamok avatar istreeter avatar lilja avatar lilja-at-funnel avatar limansky avatar miike avatar mukundananthu avatar pondzix avatar three avatar tmacedo avatar yeonhoo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scala-maxmind-iplookups's Issues

Update to GeoLite2

For some IP's (e.g. in São Paulo, Brazil), MaxMind's GeoLite correctly identifies the lat/lon coordinates, but fails to provide the city name. I've run some tests with GeoLite2 and that problem seems to be solved.

There is a Scala wrapper based on Snowplow's implementation at https://github.com/Sanoma-CDA/maxmind-geoip2-scala. Implementation seems pretty straightforward - do you see any downsides of updating to GeoLite2?

Update: running the following query returns 95% of results from São Paulo's coordinates - but city name is null

SELECT geo_latitude, geo_longitude, count(*) FROM atomic.events WHERE geo_city IS NULL GROUP BY 1,2 ORDER BY 3 DESC

Update README as per v0.3.0

Scala MaxMind IP Lookups v0.3.0

Improved thread safety : Thread safety

Switched from LruMap to SynchronizedLruMap (#28)

Build: Added dedicated Vagrant setup (#30)

Publish scaladoc

Scaladoc can be published through 3 plugins:

  • sbt-unidoc
  • sbt-site
  • sbt-ghpages

Add Maxmind discontinuation notice to the README

We will be discontinuing updates to the GeoLite Legacy databases as of April 1, 2018. You will still be able to download the April 2018 release until January 2, 2019. GeoLite Legacy users will need to update their integrations in order to switch to the free GeoLite2 or commercial GeoIP databases by April 2018.

Full notice is here and blog post here - it looks like this will primarily impact those using .dat files.

Set up cross-build

, crossScalaVersions := Seq("2.9.2", "2.9.3", "2.10.0", "2.10.1")

And add into .travis.yml

Remove dependency resolvers

[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#util-collection;5.3.10: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.twitter:util-collection:5.3.10 (/code_disk/scala-maxmind-iplookups/project/ScalaMaxmindIpLookupsBuild.scala#L30)
[warn] +- com.snowplowanalytics:scala-maxmind-iplookups_2.9.3:0.3.0
[trace] Stack trace suppressed: run last :update for the full output.
[error] (
:update) sbt.ResolveException: unresolved dependency: com.twitter#util-collection;5.3.10: not found
[error] Total time: 14 s, completed Jan 25, 2018 3:09:57 PM

Lookup timeout config

It seems that there's a hard-coded 4 seconds timeout on lookups in performLookupsWithoutLruCache(). This can cause possible I/O bottlenecks and should be at least configurable in the IpLookups class.

Add Bintray credentials to .travis.yml

  • BINTRAY_SNOWPLOW_MAVEN_USER
  • BINTRAY_SNOWPLOW_MAVEN_API_KEY

Additionally, there is no scala-maxmind-iplookups repo in snowplow-maven so that needs to be created.
Also, mirroring to central will need to be enabled.

  • AD: add creds
  • AD: create repo in Bintray
  • BF: push first M1
  • AD: add to JCenter
  • AD: activate Maven Central sync

0.4.0 not found

Hello, i have followed the instruction and got this

sbt.ResolveException: unresolved dependency: com.snowplowanalytics#scala-maxmind-iplookups_2.11;0.4.0: not found

Refactor test suite

Not urgent. This just describes the inner loop of the tests - still want to perform the outer loops as-are

Clarify and update README

  • The dependencies syntax
  • getLocation - what format for ip, it returns an Option(...) not a raw IpLocation

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.