osm2pgsql-dev / osm2pgsql Goto Github PK
View Code? Open in Web Editor NEWOpenStreetMap data to PostgreSQL converter
Home Page: https://osm2pgsql.org
License: GNU General Public License v2.0
OpenStreetMap data to PostgreSQL converter
Home Page: https://osm2pgsql.org
License: GNU General Public License v2.0
Hello again :),
so now i tried to import oberbayern-latest.osm.bz2 into my db using the latest version of osm2pgsql (with fix from #31), and now i have the polygon called "Stadtbezirk 04 Schwabing-West". Thank you very much for the fix. But now the problem occured again. So what did i do:
First i filtered out admin boundaries using osmosis:
bzcat oberbayern-latest.osm.bz2 | ./osmosis/bin/osmosis --read-xml enableDateParsing=no file=- --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --write-xml file=- | bzip2 > oberbayern-adminbounds.xml.bz2
I used: Osmosis Version 0.43.1
After it i checked if the relation is inside the file "oberbayern-adminbounds.xml.bz2" and found this:
<relation id="56389" version="25" timestamp="2013-05-31T16:52:28Z" uid="130472" user="fx99" changeset="14290051">
<member type="way" ref="28823678" role="outer"/>
<member type="way" ref="30033834" role="outer"/>
<member type="way" ref="195085889" role="outer"/>
<member type="way" ref="196473438" role="outer"/>
<member type="way" ref="194705249" role="outer"/>
<member type="way" ref="195085890" role="outer"/>
<member type="way" ref="69685291" role="outer"/>
<member type="way" ref="69685244" role="outer"/>
<member type="way" ref="30101361" role="outer"/>
<member type="way" ref="196473436" role="outer"/>
<member type="way" ref="45048519" role="outer"/>
<member type="way" ref="194705245" role="outer"/>
<member type="way" ref="196167289" role="outer"/>
<member type="way" ref="196473435" role="outer"/>
<member type="way" ref="42941597" role="outer"/>
<member type="way" ref="30033065" role="outer"/>
<member type="way" ref="30033066" role="outer"/>
<member type="way" ref="196328310" role="outer"/>
<member type="way" ref="196328308" role="outer"/>
<member type="relation" ref="66998" role=""/>
<member type="relation" ref="66999" role=""/>
<member type="relation" ref="67000" role=""/>
<tag k="admin_level" v="9"/>
<tag k="boundary" v="administrative"/>
<tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
<tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
<tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
<tag k="type" v="multipolygon"/>
</relation>
Then i migrate it to my db using this:
./osm2pgsql/osm2pgsql --cache 6000 --number-processes 7 -S ./default.style --latlong -H localhost -U michael -P 9201 -d osm oberbayern-adminbounds.xml.bz2
Now i check the db and Schwabing is not there anymore. I checked the db in several ways, but exemplary i show you a short shell dialog:
osm=# SELECT * from planet_osm_polygon where name = 'Stadtbezirk 04 Schwabing-West';
osm_id | access | addr:housename | addr:housenumber | addr:interpolation | admin_level | aerialway | aeroway | amenity | area | barrier | bicycle | brand | bridge | boundary | building | construction | covered | culvert | cutting | denomination | disused | embankment | foot | generator:source | harbour | highway | historic | horse | intermittent | junction | landuse | layer | leisure | lock | man_made | military | motorcar | name | natural | office | oneway | operator | place | population | power | power_source | public_transport | railway | ref | religion | route | service | shop | sport | surface | toll | tourism | tower:type | tracktype | tunnel | water | waterway | wetland | width | wood | z_order | way_area | timezone | way
--------+--------+----------------+------------------+--------------------+-------------+-----------+---------+---------+------+---------+---------+-------+--------+----------+----------+--------------+---------+---------+---------+--------------+---------+------------+------+------------------+---------+---------+----------+-------+--------------+----------+---------+-------+---------+------+----------+----------+----------+------+---------+--------+--------+----------+-------+------------+-------+--------------+------------------+---------+-----+----------+-------+---------+------+-------+---------+------+---------+------------+-----------+--------+-------+----------+---------+-------+------+---------+----------+----------+-----
(0 rows)
Thanks very much for your help. You guys are really very responsive, i appreciate this.
PS: Regarding the default.style i just added a timezone line. Now it looks like this:
# This is the style file that matches the old version of osm2pgsql, which
# did not make distinctions between tags for nodes and for ways. There are a
# number of optimisations that can be applied here. Firstly, certain tags
# only apply to only nodes or only ways. By fixing this we reduce the amount
# of useless data loaded into the DB, which is a good thing. Possible
# optimisations for the future:
#1. Generate this file directly from the mapnik XML config, so it's always
# optimal
#2. Extend it so it can understand that highway=tertiary is for ways and
# highway=bus_stop is for nodes
# Flags field isn't used much yet, expect if it contains the text "polygon"
# it indicates the shape is candidate for the polygon table. In the future I
# would like to be able to add directives like "nocache" which tells
# osm2pgsql that it is unlikely this node will be used by a way and so it
# doesn't need to be stored (eg coastline nodes). While in essence an
# optimisation hack, for --slim mode it doesn't matter if you're wrong, but
# in non-slim you might break something!
# Also possibly an ignore flag, for things like "note" and "source" which
# can simply be deleted. (In slim mode this is, does not apply to non-slim
# obviously)
# OsmType Tag DataType Flags
node,way note text delete # These tags can be long but are useless for rendering
node,way source text delete # This indicates that we shouldn't store them
node,way created_by text delete
#node,way note text polygon # These tags can be long but are useless for rendering
#node,way source text polygon # This indicates that we shouldn't store them
#node,way created_by text polygon
node,way access text linear
node,way addr:housename text linear
node,way addr:housenumber text linear
node,way addr:interpolation text linear
node,way admin_level text linear
node,way aerialway text linear
node,way aeroway text polygon
node,way amenity text nocache,polygon
node,way area text # hard coded support for area=1/yes => polygon is in osm2pgsql
node,way barrier text linear
node,way bicycle text nocache
node,way brand text linear
node,way bridge text linear
node,way boundary text linear
node,way building text polygon
node capital text linear
node,way construction text linear
node,way covered text linear
node,way culvert text linear
node,way cutting text linear
node,way denomination text linear
node,way disused text linear
node ele text linear
node,way embankment text linear
node,way foot text linear
node,way generator:source text linear
node,way harbour text polygon
node,way highway text linear
node,way historic text polygon
node,way horse text linear
node,way intermittent text linear
node,way junction text linear
node,way landuse text polygon
node,way layer text linear
node,way leisure text polygon
node,way lock text linear
node,way man_made text polygon
node,way military text polygon
node,way motorcar text linear
node,way name text linear
node,way natural text polygon # natural=coastline tags are discarded by a hard coded rule in osm2pgsql
node,way office text polygon
node,way oneway text linear
node,way operator text linear
node,way place text polygon
node poi text
node,way population text linear
node,way power text polygon
node,way power_source text linear
node,way public_transport text polygon
node,way railway text linear
node,way ref text linear
node,way religion text nocache
node,way route text linear
node,way service text linear
node,way shop text polygon
node,way sport text polygon
node,way surface text linear
node,way toll text linear
node,way tourism text polygon
node,way tower:type text linear
way tracktype text linear
node,way tunnel text linear
node,way water text polygon
node,way waterway text polygon
node,way wetland text polygon
node,way width text linear
node,way wood text linear
node,way z_order int4 linear # This is calculated during import
way way_area real # This is calculated during import
# mic specific
node,way timezone text polygon
# If you're interested in bicycle routes, you may want the following fields
# To make these work you need slim mode or the necessary data won't be remembered.
#way lcn_ref text linear
#way rcn_ref text linear
#way ncn_ref text linear
#way lcn text linear
#way rcn text linear
#way ncn text linear
#way lwn_ref text linear
#way rwn_ref text linear
#way nwn_ref text linear
#way lwn text linear
#way rwn text linear
#way nwn text linear
#way route_pref_color text linear
#way route_name text linear
# The following entries can be used with the --extra-attributes option
# to include the username, userid, version & timstamp in the DB
#node,way osm_user text
#node,way osm_uid text
#node,way osm_version text
#node,way osm_timestamp text
Difference found: ways in a MP where inner, outer and relation are tagged with landuse=forest. I don't have a problem with lua's handling, the tagging is unclear. Also, a broken import.
Way in question: http://www.openstreetmap.org/browse/way/42234367
i have a problem when trying to compile unde Centos 6.3:
$ ./autogen.sh
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
/usr/bin/m4:configure.ac:37: recursion limit of 1024 exceeded, use -L to change it
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: autom4te failed with exit status: 1
autoreconf: aclocal failed with exit status: 1
$ uname -rvmpio
2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ aclocal --version
aclocal (GNU automake) 1.11.1
$ autoconf --version
autoconf (GNU Autoconf) 2.63
$ m4 --version
m4 (GNU M4) 1.4.13
below there are the required libraries installed
$ rpm -qa | grep -i boost
boost-date-time-1.41.0-11.el6_1.2.x86_64
boost-wave-1.41.0-11.el6_1.2.x86_64
boost-test-1.41.0-11.el6_1.2.x86_64
boost-thread-1.41.0-11.el6_1.2.x86_64
boost-filesystem-1.41.0-11.el6_1.2.x86_64
boost-graph-1.41.0-11.el6_1.2.x86_64
boost-python-1.41.0-11.el6_1.2.x86_64
boost-serialization-1.41.0-11.el6_1.2.x86_64
boost-iostreams-1.41.0-11.el6_1.2.x86_64
boost-1.41.0-11.el6_1.2.x86_64
boost-system-1.41.0-11.el6_1.2.x86_64
boost-regex-1.41.0-11.el6_1.2.x86_64
boost-signals-1.41.0-11.el6_1.2.x86_64
boost-program-options-1.41.0-11.el6_1.2.x86_64
boost-devel-1.41.0-11.el6_1.2.x86_64
$ rpm -qa | grep libxml
libxml2-devel-2.7.6-8.el6_3.4.x86_64
libxml2-2.7.6-8.el6_3.4.x86_64
$ rpm -qa | grep proj
proj-4.8.0-2.rhel6.x86_64
proj-devel-4.8.0-2.rhel6.x86_64
$ rpm -qa | grep geos
geos-3.3.6-4.rhel6.x86_64
geos-devel-3.3.6-4.rhel6.x86_64
$ rpm -qa | grep bzip2
bzip2-libs-1.0.5-7.el6_0.x86_64
bzip2-devel-1.0.5-7.el6_0.x86_64
bzip2-1.0.5-7.el6_0.x86_64
$ rpm -qa | grep zlib
zlib-1.2.3-27.el6.x86_64
zlib-devel-1.2.3-27.el6.x86_64
$ rpm -qa | grep postgres
postgresql92-devel-9.2.2-1PGDG.rhel6.x86_64
postgresql92-9.2.2-1PGDG.rhel6.x86_64
postgresql92-server-9.2.2-1PGDG.rhel6.x86_64
postgresql92-libs-9.2.2-1PGDG.rhel6.x86_64
postgresql92-contrib-9.2.2-1PGDG.rhel6.x86_64
$ rpm -qa | grep gis
postgis2_92-2.0.2-1.rhel6.x86_64
$ rpm -qa | grep -i gettext
gettext-devel-0.17-16.el6.x86_64
gettext-0.17-16.el6.x86_64
gettext-libs-0.17-16.el6.x86_64
When i try to import data from the following file:
http://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.bz2
(not very large 144MB, so you could try)
i don't get the (for me very import) administrative boundaries in the polygons table (perhaps there are one or two meaningless ones).
Example: There is no "Stadtbezirk 04 Schwabing-West" polygon.
But even the Boundary of the capital München is missing. All the administrative boundaries are missing. I did not check for other polygons or other database files yet, but i try to make a concrete example here, that you could trace it down.
I still had an old database, and there is everything well imported. I checked the ids of the missing polygons and there was something interesting. all these polygons had negative ids in my old database. In the xml file there were only positive ids (i guess you know this well, but this is all new for me). The ids of the xml file were the same like in the database but just positive ones.
I checked the xml file and i seem to miss this one (next to all the others):
<relation id="56389" version="25" timestamp="2012-12-16T09:21:11Z" changeset="14290051" uid="130472" user="fx99">
<member type="way" ref="28823678" role="outer"/>
<member type="way" ref="30033834" role="outer"/>
<member type="way" ref="195085889" role="outer"/>
<member type="way" ref="196473438" role="outer"/>
<member type="way" ref="194705249" role="outer"/>
<member type="way" ref="195085890" role="outer"/>
<member type="way" ref="69685291" role="outer"/>
<member type="way" ref="69685244" role="outer"/>
<member type="way" ref="30101361" role="outer"/>
<member type="way" ref="196473436" role="outer"/>
<member type="way" ref="45048519" role="outer"/>
<member type="way" ref="194705245" role="outer"/>
<member type="way" ref="196167289" role="outer"/>
<member type="way" ref="196473435" role="outer"/>
<member type="way" ref="42941597" role="outer"/>
<member type="way" ref="30033065" role="outer"/>
<member type="way" ref="30033066" role="outer"/>
<member type="way" ref="196328310" role="outer"/>
<member type="way" ref="196328308" role="outer"/>
<member type="relation" ref="66998" role=""/>
<member type="relation" ref="66999" role=""/>
<member type="relation" ref="67000" role=""/>
<tag k="admin_level" v="9"/>
<tag k="boundary" v="administrative"/>
<tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
<tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
<tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
<tag k="type" v="multipolygon"/>
</relation>
The file can be found on:
http://download.geofabrik.de/europe/germany/bayern.html
I use the following command:
./osm2pgsql/osm2pgsql --cache 5000 --number-processes 7 -S osm2pgsql/default.style --latlong -H localhost -U michael -P 9201 -d osm oberbayern-latest.osm.bz2
My version of osm2pgsql:
./osm2pgsql/osm2pgsql --version
osm2pgsql SVN version 0.83.0 (64bit id space)
I tried with native ubuntu 12.04 libraries, but now i got the latest ones as you can see at the end of my post (ldd command).
I tried with postgis 1.5 and 2.0.
I use postgres 8.4.
I configured osm2pgsql with 32 bit space and 64 bit space. No difference. Before I switchted to this version of osm2pgsql i could import everything, but had other problems (with appending spain to germany database). I don't know this old version any more now.
here are the used libraries
ldd osm2pgsql
linux-vdso.so.1 => (0x00007fff7b1ff000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fce528bb000)
libpq.so.5 => /usr/lib/libpq.so.5 (0x00007fce5268f000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fce52333000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fce52123000)
libgeos-3.4.0dev.so => /usr/local/lib/libgeos-3.4.0dev.so (0x00007fce51d89000)
libproj.so.0 => /usr/lib/libproj.so.0 (0x00007fce51b46000)
libprotobuf-c.so.0 => /usr/lib/libprotobuf-c.so.0 (0x00007fce51936000)
liblua5.2.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.2.so.0 (0x00007fce51705000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fce51404000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fce51108000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fce50ef2000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fce50cd4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce50915000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fce506b7000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fce502db000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fce5000d000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fce4fe09000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fce4fbca000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fce4f97b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fce4f777000)
/lib64/ld-linux-x86-64.so.2 (0x00007fce52ae3000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fce4f54e000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fce4f346000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fce4f141000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fce4ef25000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fce4ed17000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fce4eafb000)
libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007fce4e8bd000)
libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007fce4e601000)
libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007fce4e382000)
libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007fce4e17b000)
libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007fce4def4000)
libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fce4dc54000)
libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fce4da20000)
libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fce4d80a000)
libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007fce4d5f9000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fce4d3e7000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fce4d1e2000)
libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007fce4cfb9000)
libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fce4cdaa000)
libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007fce4cb5f000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fce4c8bc000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fce4c683000)
please help, i am really stuck here.
Thanks, Michael
The Readme says:
Depending on the command-line switches you can select which projection you
want the database in. You have three choices:4326: The standard lat/long coordinates
900913: The spherical Mercator projection, used by TileCache, Google Earth etc.
3395: The legacy (broken) WGS84 Mercator projection
It is unclear why EPSG:3395 is considered "broken" relative to EPSG:4326. One should clarify that point.
One should clarify this point.
When a multipolygon is deleted by diff updates, its polygon is correctly deleted from planet_osm_polygon, but if the outer member ways are still there in the DB, their polygons are not recreated and thus missing after the update because they were not created because of the multipolygon.
It looks like delete_rels query may set pending back true on members to workaround this, but I'm not familiar enough with osm2pgsql code to offer a patch.
On a Debian Squeeze box, the master branch of osm2pgsql imports everything as expected from a planet.osm file. However, the same repository on an ubuntu 12.04 install misses some polygon features like forests and the rest of a river after a damn, some islands etc.
I tested osm2pgsql on a debian box, pointing at the database on my new ubuntu machine. So, it's not the postgres server, but probably one of the libraries that osm2pgsql depends on for parsing. Same problem happens on the PBF or XML file.
Where do I start to get this figured out? I have a new shiny new dedicated tile server and need to migrate off the old server. Due to dependency hell I have not been able to get the exact versions of libgeos and postgis running on the new ubuntu server.
I want to generate line for type=waterway in addition to already existing types. I have added the corresponding code to style.lua, but was blocked by the following code:
/* Only a limited subset of type= is supported, ignore other */
if ( (strcmp(type, "route") != 0) && (strcmp(type, "multipolygon") != 0) && (strcmp(type, "boundary") != 0))
return 0;
This is already handled by style.lua, but it doesn't seem to be present in default tagtransform.
I'm hitting an issue with some tags that are not correctly imported in table planet_osm_polygons, and I think it might be because of the tags order in the .osm source.
For example, these two osm files only differ on the tag ordering of type=boundary:
http://osm7.openstreetmap.fr/~osm2pgsql/alsace.osm.gz
http://osm7.openstreetmap.fr/~osm2pgsql/alsace-fail.osm.gz
--- alsace.osm 2013-06-01 00:38:25.182404624 +0200
+++ alsace-fail.osm 2013-06-01 01:29:07.841660538 +0200
@@ -31910,6 +31910,7 @@
<member type="way" ref="48683073" role="outer"/>
<member type="way" ref="48565263" role="outer"/>
<member type="way" ref="23301554" role="outer"/>
+ <tag k="type" v="boundary"/>
<tag k="admin_level" v="4"/>
<tag k="alt_name:la" v="Elisatia"/>
<tag k="boundary" v="administrative"/>
@@ -31933,7 +31934,6 @@
<tag k="ref:INSEE" v="42"/>
<tag k="ref:ISO 3166-2" v="A"/>
<tag k="ref:NUTS" v="FR42"/>
- <tag k="type" v="boundary"/>
<tag k="wikipedia" v="fr:Alsace"/>
</relation>
With the second file, admin_level column of planet_osm_polygon is not filled with value 4.
Must the tags be sorted before importing with osm2pgsql ?
configure line 15698 should be
as_fn_error $? "$ac_word does not exist or it is not an executable file" "$LINENO" 5
or xml2-config name is not printed in error msg.
package lua5.2 is not suggested nor by configure and neither in readme.
in readme is not suggested to do make install
900913 is included in PostGIS 2.0.1:
duplicate key value violates unique constraint "spatial_ref_sys_pkey"
DETAIL: Key (srid)=(900913) already exists.
a link to
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
could be useful. checkpoint_completion_target is also interesting, even if I'm not sure I've understood what it does.
It's not very clear of the purpose of not having --melts-geometry , is it a problem with old versions of pg / postgis? Furthermore on this wiki page:
http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks
it's stated that pbf reader is 2x and (bugged) primitive is 30% faster. IMHO it's useful to document both of them in readme and man.
PS: thank you very much for your program and work :)
I'm trying to create a nominatim server and attempting to load osm file using osm2pgsql gives segfault as soon as it gets to a 64bit id.
You can get a copy of the small (~15k) datafile from http://www.4x4falcon.com/osm_data/node_0002.osm which shows the problem.
The version of osm2pgsql I'm using is the latest (1 May 2013) from git.
The problem appears to be in the function ram_cache_nodes_set_dense in file node-ram-cache.c
Command to load using nominatim is
./utils/setup.php --osm-file node_0002.osm --all
which then uses osm2pgsql with the following command
./osm2pgsql -lsc -O gazetteer --hstore -C 4250 -d nominatim node_0002.osm
Even if i enter the osm2pgsql command directly osm2pgsql still segfaults.
Running on ubuntu 12.04 lts
Cheers
Ross
Hello,
is it normal behaviour, that the same osm_id is used multiple times in the database? I wanted to use the osm_id as primary key, but this makes it problematic now.
osm=# SELECT * from planet_osm_polygon where name = 'München';
osm_id | note | source | created_by | access | addr:housename | addr:housenumber | addr:interpolation | admin_level | aerialway | aeroway | amenity | area | barrier | bicycle | brand | bridge | boundary | building | construction | covered | culvert | cutting | denomination | disused | embankment | foot | generator:source | harbour | highway | historic | horse | intermittent | junction | landuse | layer | leisure | lock | man_made | military | motorcar | name | natural | office | oneway | operator | place | population | power | power_source | public_transport | railway | ref | religion | route | service | shop | sport | surface | toll | tourism | tower:type | tracktype | tunnel | water | waterway | wetland | width | wood | z_order | way_area | timezone | way
--------+------+--------+------------+--------+----------------+------------------+--------------------+-------------+-----------+---------+---------+------+---------+---------+-------+--------+----------------+----------+--------------+---------+---------+---------+--------------+---------+------------+------+------------------+---------+---------+----------+-------+--------------+----------+---------+-------+---------+------+----------+----------+----------+---------+---------+--------+--------+----------+-------+------------+-------+--------------+------------------+---------+-----+----------+-------+---------+------+-------+---------+------+---------+------------+-----------+--------+-------+----------+---------+-------+------+---------+----------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-62428 | | | | | | | | 6 | | | | | | | | | administrative | | | | | | | | | | | | | | | | | | | | | | | | München | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 | 0 | | 0103000020E6100000010000000A00000087E8C6AAF7FA2640FC766DCA700948408D367D2C33FB264092CF2B9E7A0948408B86318E36FB26406231EA5A7B094840DE23F66459FB2640D56E055E770948405473B9C150FB2640357D76C075094840E72ED3403DFB26401F8FCF1A72094840F28B5C8132FB2640BBCF961870094840E784758824FB2640BA2583ED71094840744694F606FB2640E6FE8FB86C09484087E8C6AAF7FA2640FC766DCA70094840
-62428 | | | | | | | | 6 | | | | | | | | | administrative | | | | | | | | | | | | | | | | | | | | | | | | München | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 | 1e-06 | | 0103000020E61000000100000006000000679EB70C931C274089FB7E202F0A48404B2B7414D61C27405ECA0A8F470A4840E79CE96F531D2740B62BF4C1320A48400675DBCF731D2740CB1E57D92D0A48404DF8A57EDE1C2740E1AD98B6240A4840679EB70C931C274089FB7E202F0A4840
(2 rows)
https://github.com/openstreetmap/osm2pgsql/blob/master/output-pgsql.c#L160 implies that wildcards (*
or ?
) can be used in delete style entries. However I can't actually find any code that handles wildcards in delete.
Do we accept wildcards in deletes? If not, should we? It would be nice to be able to drop source* rather than source, source:ja, source:en, source:name, ... etc
See #32 (comment)
The README included in the repository needs updating and improving.
Newer feature like e.g. lua tag transform need explaining and also the build dependencies updated.
Give example of typically used parameter combinations. The default settings usually aren't appropriate for full planet imports and can lead to very poor performance. So documenting typical parameter combinations is important.
time bzcat belgium-latest.osm.bz2 | osm2pgsql -r libxml2 -d osm2pgsqltest -l -O null -
takes 1m21s but time bzcat belgium-latest.osm.bz2 | osm2pgsql -r primitive -d osm2pgsqltest -l -O null -
takes 0.2s, which is clearly absurd. It also only parses 30k nodes.
When removing -O null, the resulting database is under 1MB.
If I bunzip2 to disk and give it the file, primitive does work and takes 46s (-O null) to libxml2's 71s. The pbf takes about 7.25 seconds.
If we aren't maintaining the primitive parser, I'd suggest removing it. PBF is absurdly faster than either XML parser for where speed matters.
I volunteer to build/provide binary packages of osm2pgsql for Mac OS X. My motivation is to make introductory tutorials for those new to OSM as easy as possible to follow (like this one).
But for me to be able to do this I need:
Questions? Thoughts?
Why not homebrew for OS X? Because osm2pgsql was kicked out of homebrew for always being broken. It was always broken because the formula pulled from svn head because of lacking releases. So, while homebrew would be the preferred method I think starting with binary packages is a good first step. If osm2pgsql, hopefully through @apmon's leadership, starts providing frequent stable, tagged releases then maybe we, as a community, could consider re-submitting a formula to homebrew.
When importing with --drop
or without --slim
the resulting tables will be static (i.e. not updated in the future).
The FILLFACTOR
parameter allows the creation of physically smaller indices. For static tables fillfactor 100 is best to minimize the physical size. This results in more page splits on updates, but if the tables are never updated this is not an issue.
Using a fillfactor of 100 if --drop
is specified or --slim
is not specified would reduce index size, slightly speeding up the database.
fillfactor appears to be backwards compatible to postgres 8.2
Would it be possible to select feature to import that do not match a column in the tables? Or is it already possible with lua style? I think this is would be very useful when having all tags in an indexed hstore.
I could look into implementing this with some pointers how it should be done
parse-pbf.c
uses mostly long int
to locally store OSM ids. That fails on 32bit systems for 64bit ids. It should be using the explicit osmid_t
instead.
Already changed for dense nodes in bbe0de0, rest of code needs to be adapted as well.
Originally the --number-processes
option defaulted to one because the multi-threaded code wasn't as well tested. At this point it should be well tested and unless they have a reason to do otherwise they should use the number of CPUs they have, particularly for an import. Larger servers might want to use less but they should be tuning their own osm2pgsql anyways.
The default for --number-processes
should be the number of cores.
Hi together,
I'd like to suggest to put a hint to the scripts /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql and /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql on the man page of osm2pgsql. I'm sure that it saves a lot of time for some people (besides me ;)).
All the best,
Kalle
Hi, when trying to
git clone https://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
./autogen.sh
./configure --with-proj=/opt/local
make
I get the following error:
node-persistent-cache.c: In function 'wait_for_outstanding_io':
node-persistent-cache.c:62:13: warning: implicit declaration of function 'aio_error64' [-Wimplicit-function-declaration]
node-persistent-cache.c:89:17: warning: implicit declaration of function 'aio_suspend64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'writeout_dirty_nodes':
node-persistent-cache.c:112:9: warning: implicit declaration of function 'lseek64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'persistent_cache_nodes_prefetch_async':
node-persistent-cache.c:295:28: error: invalid application of 'sizeof' to incomplete type 'struct aiocb64'
node-persistent-cache.c:296:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:297:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:301:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:303:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:305:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:307:13: warning: implicit declaration of function 'aio_write64' [-Wimplicit-function-declaration]
node-persistent-cache.c:318:62: error: invalid application of 'sizeof' to incomplete type 'struct aiocb64'
node-persistent-cache.c:320:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:321:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:324:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:326:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:328:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:331:9: warning: implicit declaration of function 'aio_read64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'init_node_persistent_cache':
node-persistent-cache.c:651:13: warning: implicit declaration of function 'posix_fallocate' [-Wimplicit-function-declaration]
make[2]: *** [node-persistent-cache.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Any help would be greatly appreciated.
Stefan
Currently osm2pgsql always creates geometry indexes and orders the tables by geometry, essentially clustering them. There are cases where this is not desirable.
5. Cases where you don't plan to consume diffs (e.g. imports with --drop
). You can be better off with a non-default FILLFACTOR
I'm running postgis 2.0.1, psql 9.1.9, master of osm2pgsql, ubuntu 12.04, 64bit.
Note this problem does not occur for people using 0.81.
SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2901483;
SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2915918;
SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2934273;
First query returned landuse=residential,
2nd query, blank on both,
both were expected behaviors, but in this third query,
for this relation, I received a landuse=residential and building=yes
even though this relation was a type=multipolygon relation.
Building=yes is attributed for all polygons in this relation although these member areas are not all buildings.
I realize these relations, created for the HOT task manager, are hackish, but they may be other use cases where this happens.
In addition to OS X (#15) the osm2pgsql community of developers should provide binaries for Windows.
Let's coordinate how to get this done via this ticket.
Tasks:
Background discussions: hotosm/tech#1
It's been said that you could almost use /dev/null for a flat-nodes file if you have a large enough -C value and don't plan to update the database. An option that allows you to use no file at all for flat nodes would be useful for when someone doesn't plan to update diffs and has enough memory. It would save on sequential writes and save on disk space. The latter is useful when using SSDs.
The code was moved in d32bad6 but the comment remains:
An increasing number of users are using hstore. Most of them won't want tags like the TIGER tags in their hstore tag columns. I suggest having the default.style not import those to the hstore by default. Note: errol's DB is one of the exceptions where we probably want every tag.
@Komzpa has a list of common tags that are unlikely to ever be used for rendering
The logic for what goes in the planet_osm_roads
table is highly tied to the default OSM Mapnik stylesheet. For rendering other styles it might make sense for this table to be different or not present at all.
Completely removing the creation of this table (since I don't need it) cuts 10-20% off my import times.
Tried to install following the directions at http://wiki.openstreetmap.org/wiki/Osm2pgsql#With_Homebrew but got the following error:
Error: No available formula for protobuf-c
Copying planet_osm_polygon to cluster by geometry finished
CREATE INDEX planet_osm_polygon_no_building_index ON planet_osm_polygon USING GIST (way) WHERE building is null TABLESPACE gisidx;
failed: ERROR: syntax error at or near "TABLESPACE"
LINE 1: ...m_polygon USING GIST (way) WHERE building is null TABLESPACE...
^
The TABLESPACE spec belongs before the WHERE clause.
This may not be a bug, but wiki here
http://wiki.openstreetmap.org/wiki/Osm2pgsql
says way_area is "calculated in Mercator meters".
Maybe this is a feature and way_area should really be calculated in decimal degrees when importing with -l flag, but then it shoud be pointed out, because some major osm styles use this field when filtering out data for rendering. Or could it be another flag for convenience?:)
Tried to import OSM data into local postgresql database. Excuted
./osm2pgsql -S default.style --slim -d gis -C 2048 --number-processes=1 --cache-strategy=dense japan-latest.osm.bz2
It went well at the first but was broke with an error after 1hr or so. I'm not sure if this is a bug due to my computer environment or something else. Just report it here, hoping someone can shed a light on this.
And the logcat goes like:
Reading in file: ../japan-latest.osm.bz2
Processing: Node(24210k 51.5k/s) Way(0k 0.00k/s) Relation(0 0.00/s)WARNING: Found Out of order node 1052693513 (34582453,9) - this will impact the cache efficiency
Processing: Node(93240k 56.6k/s) Way(0k 0.00k/s) Relation(0 0.00/s)get_node_list failed: ERROR: prepared statement "get_node_list" does not exist
(7)
Arguments were: {31236733,621545916,621545917,31236732,31236568,31236567,31236566,621545902,31236565,621545903,31236564,621545901,621545904,31236563,621545900,31236562,621545905,31236558,31236583,31236584,573323586,31236585,31236586,31236587,31236588,31236589,31236590,31236591,31236629,570755435,621545906,570755436,621545907,31236630,31236631,31236632,31236633,621545908,31236634,31236635,621545909,31236636,31236637,31252890,31252891,31252892},
Error occurred, cleaning up
And the system environment is like:
OS - Ubuntu 12.04
Disk - 40GB
RAM - 2GB
PostgreSQL - postgresql-9.1-postgis, postgresql-contrib-9.1
Fork() isn't supported by the Windows API:
osm2pgsql cannot build in the usual MingW environment without reverting back the changes which added fork for parallel processing.
middle-pgsql.o: In function `pgsql_iterate_relations':
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:1098:
undefined reference to `fork'
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:1184:
undefined reference to `wait'
middle-pgsql.o: In function `pgsql_iterate_ways':
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:782: u
ndefined reference to `fork'
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:893: u
ndefined reference to `wait'
Might want to consider using fork/exec so the Posix call can be easily replaced by spawn on windows:
http://en.wikipedia.org/wiki/Fork-exec
Microsoft Windows does not support the fork-exec model, as it does not have a system call
analogous to fork() The spawn() family of functions declared in process.h can replace it in
cases where the call to fork() is followed directly by exec().
http://stackoverflow.com/questions/12059143/unable-to-code-with-threading-as-an-alternative-to-fork
http://msdn.microsoft.com/en-us/library/y23kc048%28VS.80%29.aspx
One of the largest areas of difference is in the process model. UNIX has fork; Win32 does not.
Depending on the use of fork and the code base, Win32 has two APIs that can be used:
CreateProcess and CreateThread. A UNIX application that forks multiple copies of itself
can be reworked in Win32 to have either multiple processes or a single process with
multiple threads. If multiple processes are used, there are multiple methods of IPC that
can be used to communicate between the processes (and perhaps to update the code
and data of the new process to be like the parent, if the functionality that fork provides is
needed).
fastupdate does not work with osm2pgsql because with a high work_mem it builds up a large temporary part of the index it has to sequentially scan, slowing down updates significantly
It would be nice if we could use fastupdate with a smaller temporary part of the index.
The integration tests fail with
SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 );
failed: ERROR: AddGeometryColumn() - invalid SRID
CONTEXT: SQL statement "SELECT AddGeometryColumn('','',$1,$2,$3,$4,$5, $6)"
PL/pgSQL function "addgeometrycolumn" line 5 at SQL statement
because they do not add the 900913 SRID. Before doing an osm2pgsql import on a new database you have to set up postgis, hstore and the 900913 SRID
No import any nodes where id>type integer in table planet_osm_nodes.id and planet_osm_ways.nodes. After changed type INTEGER -> NUMERIC show error:
PREPARE node_changed_mark(int4) AS UPDATE planet_osm_ways SET pending = true WHERE nodes && ARRAY[$1] AND NOT pending;
failed: ERROR: operator does not exist: numeric[] && integer[]
LINE 1: ...TE planet_osm_ways SET pending = true WHERE nodes && ARRAY[$...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
While following the directions for installing osm2pgsql from source, I get a fatal error while running 'make':
build_geometry.Tpo -c -o build_geometry.o build_geometry.cpp
build_geometry.cpp:29:26: fatal error: geos/version.h: No such file or directory
compilation terminated.
I'm following the directions here:
http://wiki.openstreetmap.org/wiki/Osm2pgsql
and here:
http://wiki.openstreetmap.org/wiki/Osm2pgsql#From_source_.28generic.29
I'm building libprotobuf-c0-dev from source as well, because I'd like PBF support.
Thanks.
Currently we have a nocache flag. This flag sets FLAG_NOCACHE for the tag, but the only other occurrence of FLAG_NOCACHE is where it is defined to be 4, i.e. the flag is not actually used.
This is confusing, and probably would lead to errors if anyone actually tried to do anything with it.
The docs say
In the future I would like to be able to add directives like "nocache" which tells osm2pgsql that it is unlikely this node will be used by a way and so it doesn't need to be stored (eg coastline nodes). While in essence an optimisation hack, for --slim mode it doesn't matter if you're wrong, but in non-slim you might break something!
I suggest removing nocache from the default.style, to eventually allow the option of the removal of the FLAG_NOCACHE code.
Invalidation of tiles for large relations could lead to all the tiles in the US being invalidated by a tag change on a US border relation. It would be desirable to be able to not invalidate the interior of large boundary relations.
Build fails with following exception:
https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-44
autoreconf-2.68: Entering directory `.'
autoreconf-2.68: configure.ac: not using Gettext
autoreconf-2.68: running: aclocal --force -I m4
autoreconf-2.68: configure.ac: tracing
autoreconf-2.68: running: libtoolize --copy --force
autoreconf-2.68: running: /usr/bin/autoconf-2.68 --force
autoreconf-2.68: running: /usr/bin/autoheader-2.68 --force
autoreconf-2.68: running: automake --add-missing --copy --force-missing
configure.ac:41: installing './config.guess'
configure.ac:41: installing './config.sub'
configure.ac:10: installing './install-sh'
configure.ac:10: installing './missing'
Makefile.am: installing './depcomp'
autoreconf-2.68: Leaving directory `.'
getconf: Unrecognized variable `LFS_CFLAGS'
configure: error: cannot find suitable Lua interpreter
Introduced by the following change:
#32 mentions phstore
as a flag for .style files, but I'm sure there are others undocumented. These should be.
Hello,
Im using Postgres9.0.10 and PostGIS 1.5.5-1 witn CentOS5.5 32bits
I have installed these packages: geos-devel proj-devel postgresql-devel libxml2-devel bzip2-devel gcc-c++ autoconf automake libtool
But i had some problems with this: protobuf-c-devel (because im using centos 5.5)
./autogen.sh: seems to be ok
(at the end)
legacy/Makefile.am: installing ./depcomp' autoreconf: Leaving directory
.'
./configure:
(of course)
checking for protobuf-c 0.14...
checking for protobuf-c version >= 0.14...
checking for ProtobufCFieldDescriptor.packed... no
protobuf-c >= 0.14: no
checking for protobuf-c usability... no
(and)
checking for PostgreSQL libraries... yes
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
(to the end)
config.status: creating Makefile
config.status: creating legacy/Makefile
config.status: creating config.h
config.status: executing depfiles commands
make:
(a lot of references error of pgsql8.4 sources...)
(to the end)
/usr/lib/libpq.a(fe-auth.o): In function pg_krb5_sendauth': /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:220: undefined reference to
krb5_sname_to_principal'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:225: undefined reference to error_message' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:243: undefined reference to
krb5_free_principal'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:248: undefined reference to krb5_sendauth' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:278: undefined reference to
krb5_free_error'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:283: undefined reference to krb5_free_principal' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:273: undefined reference to
error_message'
collect2: ld returned 1 exit status
make[2]: *** [osm2pgsql] Error 1
make[2]: Leaving directory /home/covoit/workspace/osm2pgsql/osm2pgsql' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory
/home/covoit/workspace/osm2pgsql/osm2pgsql'
make: *** [all] Error 2
Currently osm2pgsql will create a planet_osm_nodes table even if --flat-nodes is specified. It would probably be best to not create this table.
It is also probably possible to screw up the DB by omitting --flat-nodes on diff processing and this could eliminate that possibility,
You should be able to cut out a bunch of the if statements this way.
Add before the function definitions:
polygon_keys = { 'building', 'landuse', 'amenity', 'harbour', 'historic', 'leisure',
'man_made', 'military', 'natural', 'office', 'place', 'power',
'public_transport', 'shop', 'sport', 'tourism', 'waterway',
'wetland', 'water', 'aeroway' }
then replace the if statements in filter_tags_way with:
for i,k in ipairs(polygon_keys) do
if keyvalues[k] then
poly=1
break
end
end
The parsing of tags for multipolygons seems broken - I'm getting man_made=pier on polygon created from relation 936128 (admin boundary for Poland). This is a very large multipolygon - out of several hundreds of way members only two (way 227911442 and 227911443) have this tag.
I'm using osm2pgsql build from git on 2013-09-16.
Relations with multiple members like http://www.openstreetmap.org/?lat=57.011&lon=-6.428&zoom=10&layers=M are rendered multiple times and are a bit of a cartographic disaster.
One possible fix is to import with osm2pgsql in -G mode to create a MULTIPOLYGON instead of multiple POLYGONs. @springmeyer informs me that mapnik should render the label on just the largest POLYGON within the MULTIPOLYGON by default.
Do we know anyone running with -G?
I am merging two planets from planet.openstreetmap.nl. The planets for Aruba and Curacao. Although both islands are pretty far apart, they share one common relations; the nautical boundary.
During import in regular mode, all goes well. But in slim mode, I am getting errors:
--2013-04-08 15:43:51-- http://planet.openstreetmap.nl/aruba/planet-aruba-130408.osm.pbf
Resolving planet.openstreetmap.nl (planet.openstreetmap.nl)... 2a00:d10:101::13:1, 93.186.179.161
Connecting to planet.openstreetmap.nl (planet.openstreetmap.nl)|2a00:d10:101::13:1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1069130 (1.0M)
Saving to: ‘planet-aruba-130408.osm.pbf’
100%[===============================================================================================================================================================================>] 1,069,130 1.83MB/s in 0.6s
2013-04-08 15:43:52 (1.83 MB/s) - ‘planet-aruba-130408.osm.pbf’ saved [1069130/1069130]
osm2pgsql SVN version 0.81.0 (64bit id space)
Using projection SRS 4326 (Latlong)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102401*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=800
Setting up table: planet_osm_nodes
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"
Reading in file: planet-aruba-130408.osm.pbf
Processing: Node(112k 112.2k/s) Way(8k 4.40k/s) Relation(50 50.00/s) parse time: 3s
Node stats: total(112174), max(2170360891) in 1s
Way stats: total(8792), max(196359215) in 2s
Relation stats: total(51), max(2323309) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending ways...
4885 ways are pending
Using 1 helper-processes
Helper process 0 out of 1 initialised
Process 0 finished processing 4885 ways in 4 sec
All child processes exited
4885 Pending ways took 4s at a rate of 1221.25/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Going over pending relations...
0 relations are pending
Using 1 helper-processes
Process 0 finished processing 0 relations in 0 sec
All child processes exited
node cache: stored: 112174(100.00%), storage efficiency: 50.92% (dense blocks: 18, sparse nodes: 100935), hit rate: 100.00%
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_point
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Building index on table: planet_osm_ways (fastupdate=off)
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels (fastupdate=off)
Stopped table: planet_osm_nodes in 0s
Stopped table: planet_osm_rels in 0s
Analyzing planet_osm_roads finished
Analyzing planet_osm_line finished
Analyzing planet_osm_polygon finished
Analyzing planet_osm_point finished
Stopped table: planet_osm_ways in 0s
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Creating osm_id index on planet_osm_roads
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Creating indexes on planet_osm_roads finished
Creating osm_id index on planet_osm_line
All indexes on planet_osm_roads created in 2s
Completed planet_osm_roads
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on planet_osm_polygon
Creating osm_id index on planet_osm_polygon
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 2s
Completed planet_osm_line
Creating indexes on planet_osm_polygon finished
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
All indexes on planet_osm_polygon created in 2s
Completed planet_osm_polygon
Creating osm_id index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 4s
Completed planet_osm_point
Osm2pgsql took 15s overall
Reading in file: planet-curacao-130408.osm.pbf
Processing: Node(34k 34.7k/s) Way(5k 5.02k/s) Relation(30 30.00/s) parse time: 1s
Node stats: total(34735), max(2229592357) in 1s
Way stats: total(5020), max(213219417) in 0s
Relation stats: total(33), max(2676830) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
COPY_END for COPY planet_osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint "planet_osm_rels_pkey"
DETAIL: Key (id)=(2323309) already exists.
CONTEXT: COPY planet_osm_rels, line 30: "2323309 1 354 {268396336,202476411,202476412,86298925,202487355,202487361,202487362,86297905,8629810..."
I don't think it belongs to this utility granting permissions on tables, instead output_pgsql.c is granting SELECT to PUBLIC (any database user) on anything that's imported.
On CartoDB we have a special user representing "the public", and that user is NOT "PUBLIC", so this assumption of osm2pgsql is wrong. What's the rationale for setting permissions from the importer ?
node_persistent_cache.c tests for a non-zero return value from posix_fallocate but then relies on errno being set; posics_fallocate however does not set errno; instead the return value must be interpreted for determining the cause of error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.