An Osmosis plugin that inserts OpenStreetMap data into an elasticsearch cluster

Osmosis installation is really easy and should not take you more than 5 minutes. Just follow/adapt the few shell commands below :-) You could also be interested in the osmosis-chef-cookbook that automates the osmosis installation on any Chef-managed node.

Untar the wanted build (look at the Osmosis builds page) into the /opt directory, create the /etc/osmosis file and set the $OSMOSIS_HOME and $PATH environment variable accordingly.

5-shell-command procedure:

# Osmosis 0.40.1 installation
wget -P /tmp http://dev.openstreetmap.org/~bretth/osmosis-build/osmosis-0.40.1.tgz
tar -zxvf /tmp/osmosis-0.40.1.tgz -C /opt
echo "JAVACMD_OPTIONS=\"-server -Xmx2G\"" > /etc/osmosis
export OSMOSIS_HOME=/opt/osmosis-0.40.1
export PATH=$PATH:$OSMOSIS_HOME/bin

Put the latest jar (see the downloads section) into $OSMOSIS_HOME/lib/default directory and add the org.openstreetmap.osmosis.plugin.elasticsearch.elasticsearchWriterPluginLoader line into the $OSMOSIS_HOME/config/osmosis-plugins.conf file (create it if necessary).

3-shell-command procedure:

# elasticsearch-osmosis-plugin 0.0.2 installation
wget -P /tmp https://github.com/downloads/ncolomer/elasticsearch-osmosis-plugin/elasticsearch-osmosis-plugin-1.0.2.jar
cp /tmp/elasticsearch-osmosis-plugin-1.0.2.jar $OSMOSIS_HOME/lib/default/
echo "org.openstreetmap.osmosis.plugin.elasticsearch.elasticsearchWriterPluginLoader" > $OSMOSIS_HOME/config/osmosis-plugins.conf

You must have an elasticsearch cluster up and running and reachable to make this plugin running.

To enable the plugin, append the following to your Osmosis command:

--write-elasticsearch (--wes)

Available options are:

Connect to cluster elasticsearch as TransportClient through localhost:9300:

osmosis \
    --read-pbf ~/osm/extract/ile-de-france.osm.pbf \
    --write-elasticsearch

Connect to cluster openstreetmap as NodeClient through 10.0.0.1:9300 and (re)create index osm prior to insert data:

osmosis \
	--read-pbf ~/osm/extract/ile-de-france.osm.pbf \
	--wes isNodeClient="true" host="10.0.0.1" clustername="openstreetmap" createIndex="true"

OSM data is organized in a relational model composed of data primitives - mainly node, way and relation - linked each other by their osmid. As relational, this model fits well in a RDBMS (commonly PostgreSQL + Postgis) and is exportable. Even though XML is the official representation, OpenStreetMap also supports other compressed formats such as PBF (Protocol Buffers) or BZ2 (compressed XML). These files can be easily found on the Internet (see 4.1. Get some OSM test data).

The Osmosis tool is able to read both XML and PBF formats: it deserializes data into Java objects that can be processed through plugins. In our case, the elasticsearch-osmosis-plugin will convert these Java objects into their JSON equivalent prior to be inserted into elasticsearch.

Please note that all user and version metadata are not inserted into elasticsearch for the moment.

Given the following sample.osm file:

<?xml version="1.0" encoding="UTF-8"?>
<osm xmlns:xapi="http://jxapi.openstreetmap.org/" version="0.6" 
  generator="Osmosis SNAPSHOT-r26564" xapi:planetDate="2012-03-30T00:17:05Z">
  <node id="343866517" version="9" timestamp="2009-09-25T21:37:24Z" uid="149399" 
    user="awikatchikaen" changeset="2627737" lat="48.6752901" lon="2.379928"/>
  <node id="497017646" version="1" timestamp="2009-09-15T12:39:40Z" uid="149399" 
    user="awikatchikaen" changeset="2491394" lat="48.675636" lon="2.3795092"/>
  <node id="497017647" version="1" timestamp="2009-09-15T12:39:40Z" uid="149399" 
    user="awikatchikaen" changeset="2491394" lat="48.6755728" lon="2.3795936"/>
  <way id="40849832" version="1" timestamp="2009-09-15T12:39:40Z" uid="149399" 
    user="awikatchikaen" changeset="2491394">
    <nd ref="497017646"/>
    <nd ref="497017647"/>
    <nd ref="343866517"/>
    <tag k="highway" v="residential"/>
    <tag k="name" v="Avenue Marc Sangnier"/>
  </way>
</osm>

The elasticsearch-osmosis-plugin will convert and insert these data into elasticsearch using two different index's types:

• All nodes will be stored into the node type, with its osmid as elasticsearch id

{"id":343866517,"location":[2.379928,48.6752901],"tags":{}}
{"id":497017646,"location":[2.3795092,48.675636],"tags":{}}
{"id":497017647,"location":[2.3795936,48.6755728],"tags":{}}

Execute the following command to retrieve the first node above:

curl -XGET 'http://localhost:9200/osm/node/343866517'

• All ways will be store into the way type, with its osmid as elasticsearch id

{"id":40849832,"tags":{"highway":"residential","name":"Avenue Marc Sangnier"},"nodes":[497017646,497017647,343866517]}

Execute the following command to retrieve the way above:

curl -XGET 'http://localhost:9200/osm/way/40849832'

• All relations and bounds (not present in this exmaple) are ignored because not yet implemented.

You can get OSM files (planet, extract) from various location. OpenStreetMap have some listed on their dedicated Planet.osm wiki page. Here is an example for the ile-de-france.osm.pbf extract by Geofabrik.de:

mkdir -p ~/osm/extract ~/osm/planet ~/osm/output
wget -P ~/osm/extract http://download.geofabrik.de/osm/europe/france/ile-de-france.osm.pbf

# Reset the whole osm index created by this plugin
curl -XDELETE 'http://localhost:9200/osm/'

• Osmosis detailed usage wiki page
• Self-Updating Local OpenStreetMap Extract tutorial
• Other Osmosis plugins (largely inspirated from)
  • Mapsforge's mapsforge-map-writer plugin
  • Neo4j's neo4j-osmosis-plugin plugin

• elasticsearch Chef cookbook by karmi
• Documentation about geospatial capabilities
  • Geo Location and Search article
  • Geo Distance Filter guide
  • Geo Point Type guide