Giter VIP home page Giter VIP logo

osmosis's Introduction

Osmosis

Build Status

Overview

Osmosis is a command line Java application for processing Open Street Map data.

The tool consists of a series of pluggable components that can be chained together to perform a larger operation. For example, it has components for reading from database and from file, components for writing to database and to file, components for deriving and applying change sets to data sources, components for sorting data, etc. It has been written so that it is easy to add new features without re-writing common tasks such as file or database handling.

The main point of entry for documentation is the project wiki page, although some information is included below.

Status

Osmosis is in light-maintenance mode. As of 2018 we’ve stopped active development and transitioned to periodic acceptance of pull requests with tests and minor version releases. Keep an eye on osmosis-dev list for any updates.

Usage

Installation

It is recommended to use a pre-built distribution archive rather than compile from source. The location of the latest builds are specified on the project wiki. These archives may be extracted to a location of your choice. The bin sub-directory should either be added to your PATH, or in the case of UNIX-like environments the "osmosis" script may be symlinked into an existing directory already on the PATH.

Development

See Development for details.

Issue Tracking

See https://github.com/openstreetmap/osmosis/issues

osmosis's People

Contributors

alfh avatar brettch avatar chuanyezaici avatar frodrigo avatar gagern avatar iandees avatar ivanttt avatar jeffreyeriksondg avatar jocelynj avatar mazdermind avatar mbouchard8915 avatar migurski avatar pnorman avatar ppawel avatar priteau avatar ralucamartinescu avatar rhansen avatar scrosby avatar sebastic avatar sebkur avatar siliconfiend avatar simonpoole avatar stefanb avatar stephankn avatar toebee avatar tomhughes avatar tsmock avatar vitalymil avatar woodpeck avatar zerebubuth avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

brettch scrosby podolsir toebee jean-emile iboyko ralucamartinescu morb-au ppawel jmarca tosseto mayfourth zuiwanting priteau kieranshaw conected jtheuer iosphere regogranlund cappelaere mbouchard8915 zerebubuth yiqingj olicarbo aashish24 openlawrence cstenac rootkit007 arunkmp gagern h4ck3rm1k3 gschmidtee jeffreyerikson woodpeck xiaokai0512 sebastic liqianggao oztc vince-from-nice gi-santana ratomiclab branches-cc amandasaurus evizbiz lincomatic kkdd tomcatguru ivanttt anitatailor constantineg1 m-rozycki willtemperley baditaflorin fvogler hwbllmnn sobolsigizmund bwitham carvalho7976 jocelynj longfeilove7 justin2061 wangyoo bright1992 guillaumerose lidi100 slachiewicz armandobs14 andreynovikov gdg windyfashion patlmi mantj podlom caspain mojodna hallahan onfla jaswellnitz andrew-aladev james-gould connectthefuture diffblue-benchmarks nubiofs cellington1 ljqian demonrem jburgess777 mobiwize simonpoole ianmadlenya someoneelseosm noblige abyrd rtroilo nakaner aaron22 sebkur shine0514 vitalymil sksundaram-learning

osmosis's Issues

Crash when exporting world-wide dataset

I did import a planet.osm file using osmosis, but only streets. I then did some modification to the data and now I'd like to export that data again for further processing. I did so before for a smaller dataset and it worked fine, but now that I am using it on the whole world, it always fails about an hour or so in.

To export the data I use the following command:

osmosis --read-pgsql host="localhost:5432" database="my-db" user="my-user" password="my-password" outPipe.0=pg --dd inPipe.0=pg outPipe.0=dd --write-pbf inPipe.0=dd file=output.osm.pbf

Then as mentioned it works fine for about an hour and then throws some exceptions and fails.
Did anybody export a whole-planet dataset again using osmosis? My system should be more than capable. It hast about 300GB of RAM, 50 Core-CPU and plenty of fast storage.

Any tipps on making it go through would be greatly appreciated :)

Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Mar 18, 2023 10:23:06 AM org.openstreetmap.osmosis.core.database.DatabaseLocker <init>
WARNING: Locking functions do not exist in database. Disabling locking.
Mar 18, 2023 11:17:03 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pgsql failed
java.lang.RuntimeException: java.lang.NullPointerException
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:59)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
        at java.base/java.io.DataOutputStream.writeUTF(DataOutputStream.java:347)
        at java.base/java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeString(DataOutputStoreWriter.java:108)
        at org.openstreetmap.osmosis.core.domain.v0_6.Tag.store(Tag.java:60)
        at org.openstreetmap.osmosis.core.domain.v0_6.TagCollectionImpl.store(TagCollectionImpl.java:70)
        at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:230)
        at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:154)
        at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:201)
        at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
        at org.openstreetmap.osmosis.core.store.SegmentedObjectStore.add(SegmentedObjectStore.java:130)
        at org.openstreetmap.osmosis.core.store.ChunkedObjectStore.add(ChunkedObjectStore.java:78)
        at org.openstreetmap.osmosis.core.sort.common.FileBasedSort.flushAddBuffer(FileBasedSort.java:93)
        at org.openstreetmap.osmosis.core.sort.common.FileBasedSort.add(FileBasedSort.java:118)
        at org.openstreetmap.osmosis.core.database.SortingStoreRowMapperListener.process(SortingStoreRowMapperListener.java:38)
        at org.openstreetmap.osmosis.core.database.SortingStoreRowMapperListener.process(SortingStoreRowMapperListener.java:17)
        at org.openstreetmap.osmosis.pgsnapshot.common.RowMapperRowCallbackListener.processRow(RowMapperRowCallbackListener.java:46)
        at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1575)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:482)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.getFeaturelessEntity(EntityDao.java:208)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:253)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:279)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.PostgreSqlDatasetContext.iterate(PostgreSqlDatasetContext.java:214)
        at org.openstreetmap.osmosis.dataset.v0_6.DumpDataset.process(DumpDataset.java:46)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:57)
        ... 1 more

Mar 18, 2023 11:17:03 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:321)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:234)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

I am also running on the latest version of osmosis: INFO: Osmosis Version 0.48.3

Export data from openstreetmap-website

I ran osm website on my server.
Now I want to export the whole data form it.
I use this command: osmosis --read-apidb host="*" database="openstreetmap" user="root" password="your_password" --write-xml file="Test.osm" validateSchemaVersion="no"
But I get this error:
Nov 19, 2023 10:26:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Nov 19, 2023 10:26:44 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Nov 19, 2023 10:26:44 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Argument validateSchemaVersion for task 2-write-xml was not recognised.
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:65)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:51)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:321)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:234)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

[Question] Is there any way to monitor the progress of a pipeline programatically?

Hello,

I am using Osmosis pipelines programmatically in Java as part of a larger GIS processing application, and was wondering if there is any way to poll or receive updates on the progress of a running pipeline.

As most of the documentation is focused on the command line tool, I was not able to find any resources to help me figure out how to do this -- and looking at some of the basic interfaces like Sink and OsmosisSerializer I was not able to find anything pertinent myself.

Is this currently possible to do in Osmosis?

AuthZ Query is not implemented

I want to query AuthZ.
but It is not implemented.

I queryed under link.
https://lcd.osmosis.zone/cosmos/authz/v1beta1/grants/granter/osmo1wf87ctwh5pqwugr0x9vemcllyn7vu8q4dug7hu

cosmos hub is working.
https://lcd-cosmoshub.keplr.app/cosmos/authz/v1beta1/grants/granter/cosmos1wf87ctwh5pqwugr0x9vemcllyn7vu8q498mwpw

Cannot read .bz2 files

Bug

Osmosis attempts to read .bz2 files as gzip, and fails

Steps to reproduce

On a debian linux machine:

wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/planet/planet-latest.osm.bz2

check the md5 hash against https://planet.openstreetmap.org/ to ensure it's correct.

wget https://github.com/openstreetmap/osmosis/releases/download/0.48.3/osmosis-0.48.3.tgz
tar zxvf osmosis-0.48.3.tgz
bin/osmosis --read-xml file="planet-latest.osm.bz2" compressionMethod="bzip2"    --write-xml file="copy.osm.bz2" compressionMethod="bzip2"

This produces the error:

Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to unzip gz file planet-latest.osm.bz2.
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.unzipParse(BaseXMLReader.java:109)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.handleXML(BaseXMLReader.java:84)
	at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:52)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.zip.ZipException: Not in GZIP format
	at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:166)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:80)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:92)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.unzipParse(BaseXMLReader.java:101)
	... 3 more

Oct 08, 2021 8:15:53 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

`enableKeepPartialLinestring` on partial ways may convert closed ways to open ways.

Background
With the parameter enableKeepPartialLinestring enabled, the linestring of a way is generated even if the way is incomplete. This geometry is - as per the definition - different from the original geometry. The implementation is (shortly said) that every non-existing node is simply skipped. This normally doesn't change the nature of the way, it is just "shortened" or "shrunk" a little bit. (In rare cases the linestring can become an invalid geometry. All as expected)

Problem
However, for a closed way where the first (= last) node is missing, this changes the geometry such that a closed way (which can be polygonized) suddenly becomes an open way. An example would be to take way https://www.openstreetmap.org/way/1059958114 from the extract of Zeeland generated here: http://download.openstreetmap.fr/extracts/europe/netherlands/ . This partially downloaded (closed) way generates the following (open) linestring

LINESTRING(3.60106 51.643859500000005,3.6032414 51.6446471,3.6053391 51.645518300000006,3.6073448 51.646469700000004,3.6092506 51.6474976,3.611049 51.648597800000005,3.6127328 51.6497661,3.6142955 51.6509979,3.6157309 51.6522882,3.6170333 51.653632,3.6181975 51.655024100000006,3.619219 51.6564588,3.6200937 51.6579306,3.6208181 51.6594337,3.6213894 51.660962000000005,3.6218053 51.6625097,3.6220641 51.6640705,3.6221648 51.665638400000006,3.6221071 51.667207100000006)

Request
When calling Osmosis with enableKeepPartialLinestring enabled, verify whether the id of the first node of the original way was identical to the id of the last node of the original way. (The references to the node ID should still be in the way object, even if some of the nodes themselves are missing). If this is the case, close the partial line string by "adding" the first node to the end of the linestring.

Maybe this should be the default for enableKeepPartialLinestring=yes, or maybe it's better to add a third option for enableKeepPartialLinestring (like enableKeepPartialLinestring="keepClosed").

cc @frodrigo

Error when extracting place from osm.pbf

The flags I run osmosis with: --read-pbf file="C:\europe-latest.osm.pbf" --tf accept-nodes "place=country,state,region,province,district,county,municipality,city,town,village,hamlet,isolated_dwelling,farm,allotments" --tf reject-ways --tf reject-relations --sort --read-pbf file="C:\europe-latest.osm.pbf" --tf accept-ways "place=country,state,region,province,district,county,municipality,city,town,village,hamlet,isolated_dwelling,farm,allotments" --tf reject-relations --used-node --sort --read-pbf file="C:\europe-latest.osm.pbf" --tf accept-relations "place=country,state,region,province,district,county,municipality,city,town,village,hamlet,isolated_dwelling,farm,allotments" --used-way --used-node --sort --merge --merge --write-pbf granularity=10000 "europe-places.osm.pbf"

apr 01, 2022 10:00:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
apr 01, 2022 10:00:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
apr 01, 2022 10:00:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
apr 01, 2022 10:00:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error has occurred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors(DataPostbox.java:162)
        at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue(DataPostbox.java:218)
        at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java:305)
        at org.openstreetmap.osmosis.set.v0_6.impl.DataPostboxSink.process(DataPostboxSink.java:45)
        at org.openstreetmap.osmosis.core.sort.v0_6.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:68)
        at org.openstreetmap.osmosis.core.sort.v0_6.EntitySorter.complete(EntitySorter.java:80)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:150)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:150)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:150)
        at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
        at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:37)
        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
        at java.lang.Thread.run(Unknown Source)

apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 6-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to write long 1529723894 to the store.
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeLong(DataOutputStoreWriter.java:86)
        at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:215)
        at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:154)
        at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:201)
        at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.store(NodeContainer.java:49)
        at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
        at org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:105)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:92)
        at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:75)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
        at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
        at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: There is not enough space on the disk
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(Unknown Source)
        at java.util.zip.DeflaterOutputStream.deflate(Unknown Source)
        at java.util.zip.DeflaterOutputStream.write(Unknown Source)
        at java.util.zip.GZIPOutputStream.write(Unknown Source)
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
        at java.io.BufferedOutputStream.write(Unknown Source)
        at java.io.DataOutputStream.writeLong(Unknown Source)
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeLong(DataOutputStoreWriter.java:84)
        ... 18 more

apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 11-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to write integer 527535939 to the store.
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeInteger(DataOutputStoreWriter.java:74)
        at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:203)
        at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.store(NodeContainer.java:49)
        at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
        at org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:105)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedWayFilter.process(UsedWayFilter.java:90)
        at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedWayFilter.process(UsedWayFilter.java:73)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.process(TagFilter.java:141)
        at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)
        at org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
        at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
        at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: There is not enough space on the disk
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(Unknown Source)
        at java.util.zip.DeflaterOutputStream.deflate(Unknown Source)
        at java.util.zip.DeflaterOutputStream.write(Unknown Source)
        at java.util.zip.GZIPOutputStream.write(Unknown Source)
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
        at java.io.BufferedOutputStream.write(Unknown Source)
        at java.io.DataOutputStream.writeInt(Unknown Source)
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeInteger(DataOutputStoreWriter.java:72)
        ... 15 more

apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 16-merge failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An input error has occurred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForInputErrors(DataPostbox.java:174)
        at org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue(DataPostbox.java:244)
        at org.openstreetmap.osmosis.core.store.DataPostbox.hasNext(DataPostbox.java:441)
        at org.openstreetmap.osmosis.set.v0_6.EntityMerger.nextOrNull(EntityMerger.java:290)
        at org.openstreetmap.osmosis.set.v0_6.EntityMerger.run(EntityMerger.java:126)
        at java.lang.Thread.run(Unknown Source)

apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 17-merge failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An input error has occurred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForInputErrors(DataPostbox.java:174)
        at org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue(DataPostbox.java:244)
        at org.openstreetmap.osmosis.core.store.DataPostbox.hasNext(DataPostbox.java:441)
        at org.openstreetmap.osmosis.set.v0_6.EntityMerger.nextOrNull(EntityMerger.java:290)
        at org.openstreetmap.osmosis.set.v0_6.EntityMerger.run(EntityMerger.java:126)
        at java.lang.Thread.run(Unknown Source)

apr 01, 2022 10:54:29 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

Options to process Notes

I would like to know if Osmosis provides a tool to process the Map Notes, for example to insert the notes dump from Planet and insert into Postgres.

[Question] Can one use tag transform to concatenate tag values?

Regarding the mapping of bus stops: I would like to combine the value of the "name" tag of a bus stop with the value of the "route_ref" tag into one new tag. The result should look like "name (route_ref)". Does anyone know if and how this would be possible by a simple tag-transform rule?

The question regards to my "mapsforge" map-project "OpenOutdoorMap".

Update "How to install Osmosis (Windows)" Troubleshooting section to use the correct shell variable

I found an outdated piece of documentation when I was following the Install Osmosis on Windows.

In the troubleshooting section, one of the ways recommended is by manually finding and setting the %JAVACMD% variable to point to the main Java directory. But as of osmosis version 0.49.2 I found that the variable has been changed to %JAVA_EXE% which should be manually overridden, an update in the documentation would be useful for the new users that are trying to install osmosis.

Behind a proxy

I have this question in S.O. from 2016

https://gis.stackexchange.com/questions/203905/how-to-setting-proxy-for-osmosis

And still havin problem to use osmosis behind a proxy.

opened another question here

https://gis.stackexchange.com/questions/439702/osmosis-behind-a-proxy

but no one are able to answer.

so.. how to make osmosis work behind a proxy ?

I've installed a OpenStreetMap tile server from overv/openstreetmap-tile-server Docker.

Here are some options to run ( after install using same options ):


    docker run --name openstreetmap --hostname=openstreetmap --network=mynetwork --security-opt seccomp=unconfined \
    -v /srv/osm-data/tiles/:/data/tiles/ \
    -v /srv/osmfiles/some_merged_data.osm.pbf:/data/region.osm.pbf \
    -v /srv/osm-data:/data/database/ \
    -v /srv/osmfiles/update-area.poly:/data/region.poly \
    -e https_proxy=http://MY_PROXY_IP.61:6160/ \
    -e HTTPS_PROXY=http://MY_PROXY_IP.61:6160/ \
    -e http_proxy=http://MY_PROXY_IP.61:6160/ \
    -e HTTP_PROXY=http://MY_PROXY_IP.61:6160/ \
    -e no_proxy="blah,blah,blah" \
    -e REPLICATION_URL=https://planet.openstreetmap.org/replication/day \
    -e UPDATES=enabled \
    -e THREADS=24 \
    -e ALLOW_CORS=enabled \
    -e AUTOVACUUM=off \
    -e EXPIRY_MINZOOM=13 \
    -e EXPIRY_TOUCHFROM=13 \
    -e EXPIRY_DELETEFROM=19 \
    -e EXPIRY_MAXZOOM=20 \
    --shm-size="500m" \
    -e "OSM2PGSQL_EXTRA_ARGS=-C 16384" \
    -p 80:80 \
    --restart=always \
    -d overv/openstreetmap-tile-server run

I've noticed the install process was worked fine without any error including the state file download.

After this run command, the tile server was up and running (I can access its tiles without any problem) but the log file shows this error:


    Sep 01, 2022 5:19:01 PM org.openstreetmap.osmosis.core.Osmosis run
    INFO: Osmosis Version 0.48.3
    Sep 01, 2022 5:19:02 PM org.openstreetmap.osmosis.core.Osmosis run
    INFO: Preparing pipeline.
    Sep 01, 2022 5:19:02 PM org.openstreetmap.osmosis.core.Osmosis run
    INFO: Launching pipeline execution.
    Sep 01, 2022 5:19:02 PM org.openstreetmap.osmosis.core.Osmosis run
    INFO: Pipeline executing, waiting for completion.
    [2022-09-01 17:19:01] 5458 start import from seq-nr 3639, replag is 235141s
    [2022-09-01 17:19:01] 5458 downloading diff
    tail: /var/log/tiles/osmosis.log: file truncated
    Sep 01, 2022 5:19:12 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
    SEVERE: Thread for task 1-read-replication-interval failed
    org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read the state from the server.
    	at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:95)
    	at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:46)
    	at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:273)
    	at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:372)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: java.net.UnknownHostException: planet.openstreetmap.org
    	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
    	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    	at java.base/java.net.Socket.connect(Socket.java:609)
    	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
    	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
    	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
    	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
    	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
    	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373)
    	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)
    	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
    	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
    	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)
    	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
    	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
    	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
    	at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:91)
    	... 4 more
    Sep 01, 2022 5:19:12 PM org.openstreetmap.osmosis.core.Osmosis main
    SEVERE: Execution aborted.

Seems it can't reach planet.openstreetmap.org even when it was able to do it before at install stage.

What I've been done wrong?

EDIT : Forgot to say I can download the state file from terminal using

wget "https://planet.openstreetmap.org/replication/day/state.txt"

Osmosis read apidb not complete

Hi,

I'm using osmosis ver 0.48.3, and I'm trying to read my osm postgres api db then convert it to xml.bz2 for overpass API.
The osm postgres db schema migrations version is 20210511104518
ref: https://github.com/openstreetmap/openstreetmap-website/blob/6aed84bf7b72dde6ee1fc9069b63c537bcdd5c95/db/structure.sql#L3451
on postgreSQL 13.6.
Here is my question:
My command: bin/osmosis --read-apidb host="xxx" database="xxx" user="xxx" password="xxx" validateSchemaVersion="no" --write-xml file="xxx.osm.bz2"
However, the size of the output file xxx.osm.bz2 is different between day time and night time.
At day time, there are some user accessing the api db from website-api, and the xxx.osm.bz2 is smaller and not complete(Some POI points are missing).
At night time, there is no user accessing the api db, and the xxx.osm.bz2 is clearly bigger and complete.
If the api db is busy, does the --read-apidb get incomplete results?
Is there a way to check if this --read-apidb is getting the complete result?
Thanks in advance!

osmosis export corrupted

Not sure where the issue is here, but when running an osmosis export on the rails port database, sometimes it's successful and sometimes it gets corrupted. The full export is normally 8.6gb for North America, but sometimes when it exports it is only 6gb.

sudo osmosis --read-apidb-current host="localhost" database="openstreetmap" user="openstreetmap" password="openstreetmap" validateSchemaVersion="no" --write-pbf file="/var/www/html/export.osm.pbf"

It finishes without errors thrown, but it isn't a full export. If I run the command over again eventually it will finish correctly.

DEL in keys not preserved in replication diffs

When a change is made to create an object that has keys which contain DEL U+007F control characters, Osmosis strips the control characters. One place this shows up is if two keys differ only by DEL characters, then Osmosis emits two <tag/> elements with the same k value.

As an example, in changeset 90486873 a node was introduced with the tags

   <tag k="shelter_type" v="weather_shelter"/>
   <tag k="shelter_type^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?" v="lean_to"/>

where the ^? are the DEL character (U+007F).

This can be seen with the shell command curl -s https://www.openstreetmap.org/api/0.6/node/7881523719/3 | less, but most browsers will hide the characters.

This was transmitted out in the minutely replication feed with sequence number 004183898. If this is examined with curl -s https://planet.openstreetmap.org/replication/minute/004/183/898.osc.gz | zcat | less -p 90486873
this reveals the XML

      <tag k="shelter_type" v="weather_shelter"/>
      <tag k="shelter_type" v="lean_to"/>

This XML does not contain the DEL characters, and has two keys which are the same. We found this because it was resulting in a duplicate key error, but aside from that Osmosis is producing results inconsistent with what is in the database.

Daily update is very slow

When I use --write-pgsql-change to write daily osc file into Postgres database. I found it is very slow. Only 20MB data of indonesia may take 2-3 hours, did I miss anything?

Error: Could not find or load main class org.codehaus.classworlds.Launcher

Hi
Latest .zip download which extracts to osmosis-main\osmosis-main\package\bin returns:
Error: Could not find or load main class org.codehaus.classworlds.Launcher when running osmosis.bat

The previous version (osmosis-0.48.3\lib) runs as expected.

Any ideas for a solution?

Using Windows 10 Pro 22H2

Edit: To check Is the .zip file behind the green code button not the correct place to download the latest version? (It's where I get all programs for other repositories) Ít extracts to a total of over a thousand files in 429 folders!

OsmosisRuntimeException: Task type "mapfile-writer" already exists.

I've been trying to filter .OSM data as a part of SciGRID on MacOS v. 11.6. I'm a beginner in terminal commands though. Every time I'm trying to run osmosis in terminal I get the following error:

Okt. 31, 2021 12:32:50 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.48.3
Okt. 31, 2021 12:32:50 PM org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task type "mapfile-writer" already exists.
    at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactoryRegister.register(TaskManagerFactoryRegister.java:44)
    at org.openstreetmap.osmosis.core.TaskRegistrar.loadPluginClass(TaskRegistrar.java:363)
    at org.openstreetmap.osmosis.core.TaskRegistrar.loadPlugin(TaskRegistrar.java:319)
    at org.openstreetmap.osmosis.core.TaskRegistrar.loadBuiltInPlugins(TaskRegistrar.java:135)
    at org.openstreetmap.osmosis.core.TaskRegistrar.initialize(TaskRegistrar.java:80)
    at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:81)
    at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

I've downloaded osmosis via Homebrew and tried following solutions like adding jar files to the /lib/default folder. But nothing seems to work and I don't quite understand the error message or the need/use of mapfile-writer or mapsforge. I'm trying to run the basic command:

osmosis --read-pbf file="path"
Any help is appreciated.

Error java.lang.NegativeArraySizeException with a simple read-pbf

Hi!

I have downloaded a fresh copy of .osm.pbf and .poly files from Geofabrik.de - Europe. After long time of execution of osmosis I get this error.

Error

FINE: Waiting for task 1-read-pbf to complete.
May 26, 2021 4:15:36 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pbf failed
java.lang.NegativeArraySizeException
	at org.openstreetmap.osmosis.osmbinary.file.FileBlockHead.readHead(FileBlockHead.java:53)
	at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:130)
	at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
	at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)
	at java.lang.Thread.run(Thread.java:748)

Executed command

For test --read-pbf:

  --read-pbf file="europe-latest.osm.pbf" \
  --write-null

Osmosis

Version: 0.48.2
Java options: JAVACMD_OPTIONS=-server -Xmx8G

I have tested different command options but without success. The error is the same.

Any idea?

Where is PbfWriter?

Note: I am using osmosis version 0.48.3, with the xml and pbf2 packages installed.

I am building up a pipeline programmatically in my application using the osmosis library, and I would like to both be able to read from, and write to pbf format. However, whereas there is a class PbfReader, there seems to be no corresponding PbfWriter to be found.

The osmosis command line tool can export to pbf format, so presumably this functionality exists somewhere, but I can not for the life of me find it in the osmosis library. Why is there no PbfWriter class for this functionality? How can I create a pipeline that writes to pbf format in the current version of the library?

For now I am having to make do with the XmlWriter class.

Request for a new release of Osmosis package

The current release of the osmosis package on GitHub was back in September 2020 wit version 0.48.3, and there hasn't been any new release since then. As it has been over two years since the last release, the current released version has outdated dependencies, which are already updated in the main branch of the package (e.g. org.springframework:spring-core:5.1.0.RELEASE -> dependencyVersionSpring=5.3.20).

Are any plans for a new release of the package? Thank you!

Msg error in osmosis when bboxing with bounding-polygon

Hi,

I would like to make an extract with Osmosis for the Bahia state in Brazil from the brasil.osm.bz2 file that can be found on http://download.geofabrik.de/osm/south-america/

osmosis --read-xml file="D:/OSM/Salvador/brazil.osm" --bounding-polygon file="D:/OSM/Salvador/Bahia_contour.poly" --write-xml file="D:/OSM/Salvador/bahia.osm"

My specs are :

OS Vista SP1
Osmosis 0.40.1 (installed by running a frist osmosis as indicated here http://wiki.openstreetmap.org/wiki/Osmosis/Installation )
I get this error message :

D:\OSM\osmosis-0.40.1\bin>osmosis --read-xml
file="D:/OSM/Salvador/brazil.osm" -
-bounding-polygon file="D:/OSM/Salvador/Bahia_contour.poly" --write-xml
file="D:
/OSM/Salvador/bahia.osm"
6 mai 2012 11:28:36 org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.40.1
6 mai 2012 11:28:36 org.java.plugin.registry.xml.ManifestParser
INFO: got SAX parser factory -
org.apache.xerces.jaxp.SAXParserFactoryImpl at 45a87
7
6 mai 2012 11:28:36 org.java.plugin.registry.xml.PluginRegistryImpl
configure
INFO: configured, stopOnError=false, isValidating=true
6 mai 2012 11:28:37 org.java.plugin.registry.xml.PluginRegistryImpl register
INFO: plug-in and fragment descriptors registered - 1
6 mai 2012 11:28:37 org.java.plugin.standard.StandardPluginManager
activatePlugi
n
INFO: plug-in started - org.openstreetmap.osmosis.core.plugin.Core at 0.40.1
6 mai 2012 11:28:37 org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
6 mai 2012 11:28:37 org.openstreetmap.osmosis.core.Osmosis main
GRAVE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: The following named
pipe
s () and 1 default pipes have not been terminated with appropriate output
sinks.

    at

org.openstreetmap.osmosis.core.pipeline.common.Pipeline.connectTasks(
Pipeline.java:94)
at
org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipel
ine.java:116)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun
cher.java:329)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:239)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:409)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
352)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Does anyone have an idea where is my mistake ?

Sincerely,

Severin

Error OsmosisRuntimeException: Cannot represent 89962 a s a char.

Hello

I have got an error while trying to merge two osm data files as follows:

org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent 89962 as a char. at org.openstreetmap.osmosis.core.util.IntAsChar.intToChar(IntAsChar.java:32) at org.openstreetmap.osmosis.core.domain.v0_6.Way.store(Way.java:200)

In the mail archive of openstreetmaps stored similar issues, but link to patches is broken and last osmosis build 0.40.1 and build from github sources reproduce error as well
http://www.mail-archive.com/[email protected]/msg09679.html
http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-March/000961.html

command line for merging is

D:\temp\osmosis_latest\osmosis_src\package\bin\osmosis.bat --read-xml enableDateParsing=no file=map_track_klimec.osm outPipe.0=1 --read-xml enableDateParsing=no file=strm_klimec.osm outPipe.0=2 --sort type="TypeThenId" inPipe.0=1 outPipe.0=3 --sort type="TypeThenId" inPipe.0=2 outPipe.0=4 --merge conflictResolutionMethod=version inPipe.0=3 inPipe.1=4 outPipe.0=5 --write-xml file=all_klimec.osm inPipe.0=5

first file is osm part of country map and second osm, that represent rtsm data

Potential version mismatch in "simple" schema

When I tried executing osmosis v0.48.3 in my MacOS, I've got:

SEVERE: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: The database schema version of 5 does not match the expected version of 6.
	at org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:67)

I believe that's because I've been using "simple" schema from /usr/local/Cellar/osmosis/0.48.3/libexec/script/pgsimple_schema_0.6.sql (homebrew macos installation) which contains:

INSERT INTO schema_info (version) VALUES (5);

The "snapshot" schema definition SQL file contains version 6.

https_proxy under ubuntu 20.04

Not sure if this is an osmosis bug or not, i am having trouble setting the https_proxy for java/osmosis in ubuntu 20.04.
I successfully used JAVACMD_OPTIONS="-Dhttps.proxyHost=xxx -Dhttps.proxyPort=xxx -Dhttps.proxyUser=xxx -Dhttps.proxyPassword=xxx".

java -Dhttps.proxyHost=xxx -Dhttps.proxyPort=xxx -Dhttps.proxyUser=xxx -Dhttps.proxyPassword=xxx -cp /srv/osm-replicate/osmosis/lib/default/plexus-classworlds-2.5.2.jar -Dapp.home=/srv/osm-replicate/osmosis -Dclassworlds.conf=/srv/osm-replicate/osmosis/config/plexus.conf org.codehaus.classworlds.Launcher --rri workingDirectory=/srv/osm-replicate/osmosis-working-dir/ --simc --wxc /srv/osm-replicate/var/replicate-1604582744.osc.gz
Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-rri failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read the state from the server.
        at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:95)
        at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:46)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:273)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:372)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2177)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
        at org.openstreetmap.osmosis.replication.common.ServerStateReader.getServerState(ServerStateReader.java:91)
        ... 4 more

Nov 05, 2020 1:26:37 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

I also tried changing jdk.http.auth.tunneling.disabledSchemes="" in /etc/java-11-openjdk/net.properties without any change.

Write-XML vs Write-pbf unexplained differences

Hi there,
I tried Osmosis to filter and export highways from a french administrative region (burgundy) :
link to data on geofabrik

To compare Osmosis Perf and output sizes (write-xml versus write-pbf) I did 2 runs :
-> osmosis --read-pbf bourgogne-latest.osm.pbf --tf accept-ways highway=* --tf reject-relations --used-node --write-pbf bourg_highways.osm.pbf

-> osmosis --read-pbf bourgogne-latest.osm.pbf --tf accept-ways highway=* --tf reject-relations --used-node --write-xml bourg_highways.osm

Then I inserted both data using osm2pgsql into 2 different PG schema and compared the data (polygon / line / point / roads).
My osm2pgsql command was :
osm2pgsql -H localhost -P 5432 -U postgres -W -d osm bourg_highways.osm.pbf -m -S default.style

The number of rows are :
PBF :
37 710 points
303 485 lines
752 polygons
23 176 roads

XML :
37 710 points
303 501 lines
755 polygons
23 176 roads

3 polygons are missing in PBF schema and I dont know why !? (geom seems normal)
Osm_ids are : 23343996 / 204932791 / 566226558

16 linestring are missing in PBF schema and all of them are made with only 2 points.
osm_ids are :
309887308 / 727264867 / 358671237 / 361604076 / 669483224 / 693945286 / 779260526 / 831193716
695850492 / 648308083 / 497272938 / 444292828 / 832501799 / 490653698 / 769772000 / 137663695

I think there is a bug in the write-pbf algorithm of Osmosis.

IOException: No space left on device on Linux

Issue is resolved, but wanted to register the issue and resolution here to help anyone else who ran into this issue.

Repeatedly ran into the following error when using Osmosis on Linux*:

Jun 23, 2023 6:02:30 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Jun 23, 2023 6:02:30 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 23, 2023 6:02:31 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 23, 2023 6:02:31 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Jun 23, 2023 6:12:26 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to write String (jackbus) to the store.
	at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeString(DataOutputStoreWriter.java:110)
	at org.openstreetmap.osmosis.core.domain.v0_6.OsmUser.store(OsmUser.java:86)
	at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:226)
	at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:154)
	at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:201)
	at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.store(NodeContainer.java:49)
	at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
	at org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:105)
	at org.openstreetmap.osmosis.areafilter.v0_6.AreaFilter.process(AreaFilter.java:157)
	at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
	at org.openstreetmap.osmosis.areafilter.v0_6.AreaFilter.process(AreaFilter.java:125)
	at org.openstreetmap.osmosis.xml.v0_6.impl.NodeElementProcessor.end(NodeElementProcessor.java:139)
	at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107)
	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
	at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.parseXML(BaseXMLReader.java:65)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.handleXML(BaseXMLReader.java:77)
	at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:52)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:326)
	at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
	at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
	at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:401)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
	at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeString(DataOutputStoreWriter.java:108)
	... 28 more

Jun 23, 2023 6:12:26 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

The issue didn't make sense to me because the directory had terabytes of space left. The solution was to set export _JAVA_OPTIONS=-Djava.io.tmpdir=/path/to/disk/with/lots/of/space (helpful StackOverflow answers here). With is variable set, Osmosis should pick up on the environmental variable (first line of Osmosis output will look like Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/data/keeling/a/michels9/scratch/tmp) and use it so you no longer run into the issue.

* OS information:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Hope this helps someone and thank you devs for all the hard work!

How to register plugins (after Java Plugin Framework was removed with v0.49.0)

Hi!
I registered plugins (concrete: Map-Writer plugin) with v0.48.3 on macOS by copying the file to ~/.openstreetmap/osmosis/plugins as suggested in the official documentation.

After the Java Plugin Framework was removed with v0.49.0 (95a380b) as listed in the release notes, this does not work anymore.
How is the "official" way of registering plugins now?

Is this way still valid?
Where are these paths on macOS? In the home-directory?

These links are depreciated as of v0.49.0 in my opinion and need to be adapted:

would be happy to get a respond (and then help out to adjust these documentations) or is there a alternative to the Map-Writer plugin used by osmosis?

Osmosis not working in ubuntu 22.04

Installed osmosis on a ubuntu 22.04 using this command

apt install osmosis

Jun 19, 2022 7:41:23 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.47
Jun 19, 2022 7:41:23 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins
FINE: Searching for JPF plugins.
Jun 19, 2022 7:41:23 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins
FINE: Registering the core plugin.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins
FINE: Registering the extension plugins.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare
FINE: Building tasks.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks
FINE: Created task "1-rbf"
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks
FINE: Created task "2-wp"
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare
FINE: Connecting tasks.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask
FINE: Task "1-rbf" produced unnamed pipe stored at level 1 in the default pipe stack.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks
FINE: Connected task "1-rbf"
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask
FINE: Task "2-wp" consumed unnamed pipe stored at level 1 in the default pipe stack.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks
FINE: Connected task "2-wp"
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager execute
FINE: Launching task 1-rbf in a new thread.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager execute
FINE: Task 2-wp is passive, no execution required.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
FINE: Waiting for task 1-rbf to complete.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-rbf failed
java.lang.NoClassDefFoundError: com/google/common/util/concurrent/MoreExecutors
	at org.openstreetmap.osmosis.pbf2.v0_6.PbfReader.run(PbfReader.java:87)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.MoreExecutors
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
	... 2 more

Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager waitForCompletion
FINE: Task 2-wp is passive, no completion wait required.
Jun 19, 2022 7:41:24 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:321)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:234)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

Allow tee to spawn a thread by outPipe

I tried to use --tee to create several pbfs from an single pbf source. Currently all outputs are handled sequentially but for independant pipelines it could be worth to proccess them in parallel.

What do you think?

Windows | Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/classwor lds/Launcher

Osmosis is running on Windows by executing osmosis.bat

Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/classwor
lds/Launcher
Caused by: java.lang.ClassNotFoundException: org.codehaus.classworlds.Launcher
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.codehaus.classworlds.Launcher.  Program will
exit.

Duplicate work being done on pbf parsing and writing

@simon04 @brettch @migurski @joto

Noticed this while updating the dependency for googles protobuf lib for Vespuccii, but I would have screamed before if I had noticed that actual work was being done on https://github.com/openstreetmap/OSM-binary, which iirc was only moved to the openstreetmap org for archival purposes more than half a decade ago.

@brettch pls correct me if I'm wrong, but back when PBF support was integrated in to osmosis, a copy of the code in OSM-binary was integrated instead of adding it as a dependency. So what has happened now is that @simon04 has been working on the code in OSM-binary, with independent parallel now and then changes to the source code here.

Given that OSM doesn't have enough dev time to go around as is, duplicating work makes no sense at all. Unluckily due to the nature of how OSM-binary was integrated there doesn't seem to be a simple solution for projects that have either one or the other integrated, but maybe somebody has a better idea on how to resolve this?

update all films / shows crashes osmisis

dont know when this exactly happened. i do not often update all films / shows.

2020-10-25 10:20:16.589 T:4932   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.TypeError'>
                                            Error Contents: expected string or buffer
                                            Traceback (most recent call last):
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\default.py", line 78, in <module>
                                                strm_update(actor=int(params.get('updateActor', 0)))
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\resources\lib\updateAll.py", line 83, in strm_update
                                                fillPluginItems(url, strm=True, strm_name=name, strm_type=cType, name_orig=name_orig, pDialog=pDialog)
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\resources\lib\create.py", line 76, in fillPluginItems
                                                addMovies(details, strm_name, strm_type, name_orig, pDialog)
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\resources\lib\create.py", line 491, in addMovies
                                                m_title = cleanLabels(contentList[0].get('name'), keep_year=settings.KEEP_MOVIE_YEAR)
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\resources\lib\stringUtils.py", line 49, in cleanLabels
                                                text = multiple_reSub(text, dictresub)
                                              File "C:\Users\TV\AppData\Roaming\Kodi\addons\plugin.video.osmosis\resources\lib\utils.py", line 69, in multiple_reSub
                                                text = re.sub(i, j, text)
                                              File "C:\Program Files\Kodi\system\python\Lib\re.py", line 155, in sub
                                                return _compile(pattern, flags).sub(repl, string, count)
                                            TypeError: expected string or buffer
                                            -->End of Python script error report<--

log4j in osmosis 0.47

I have an elder installation (osmosis --version 0.47) of osmosis on my desktop.
I can see that it uses log4j as there is a config in /etc/osmosis/log4j.properties
Are there plans to upgrade to the latest log4j 2.16.0 or is this handled by the JRE/JDK alone ?

org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter issues SQL commit even when not needed

It appears that the "flushxxxxx()" methods (flushNodes(), flushNodeTags(), flushWays(), etc.) all issue a SQL commit() at the end of the method, even when the logic to actually do SQL inserts of data is not executed.

For example, in flushNodes() the while (nodeBuffer.size() >= INSERT_BULK_ROW_COUNT_NODE) loop (starting around line 586) that inserts data into the DB is only executed every 100 times the method is called (100 being the value of INSERT_BULK_ROW_COUNT_NODE), and yet the dbctx.commit() call is always executed at the end of the method. In addition, the 'complete' flag is false far more often than not and so the cleanup logic in that block is not executed and a dbctx.commit() call is not needed.

In addition, there is a commit near line 607 inside the while (nodeBuffer.size() ... loop that is executed every 100K records (with 100,000 being the value of TRANSACTION_SIZE:

if (transactionSizeCount % TRANSACTION_SIZE == 0) {
 	dbCtx.commit();
}

dbCtx.commit() simply checks to see if a DB connection is present (which appears to be always the case) and then issues the SQL commit() call.

Redundant dbctx.commit() calls are also made when sub-functions such as addNodeTags() (which calls flushNodeTags()) are called, such as near line 616. Committing of node tag inserts (or way tag inserts, or whatever) can be done in the same commit that handles the node or way as long as it's done on the same database connection.

I would propose that we either conditionally execute dbctx.commit() only if inserts have actually been executed, or just move dbctx.commit() calls to just outside the logic that actually does inserts. When the DB connection is created 'autoCommit' is set to 'false', so since we're doing batch inserts it's conceivably an option to just turn autocommit on and let the database handle this.

Updating these methods to restructure the logic and only issue dbctx.commit() requests when actually needed would likely speed up database insert processing by a non-trivial amount.

ERROR: duplicate key value violates unique constraint "way_tags_pkey" Detail: Key (way_id, version, k)=(1571192, 1, SEMANTIC) already exists

When I use osmosis to upload xml to postgre db. There is an error as the title described :

$ osmosis --read-xml file=tmp/tmp_upload_934962040 --write-apidb database=my_db host=mydb.com:5432 user=pguser password= validateSchemaVersion=no
Jun 11, 2021 6:55:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Jun 11, 2021 6:55:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 11, 2021 6:55:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 11, 2021 6:55:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.


Jun 11, 2021 6:58:29 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to bulk insert way tags into the database.
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.flushWayTags(ApidbWriter.java:777)
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.addWayTags(ApidbWriter.java:1252)
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.flushWays(ApidbWriter.java:724)
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.process(ApidbWriter.java:1238)
	at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(WayContainer.java:58)
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.process(ApidbWriter.java:1172)
	at org.openstreetmap.osmosis.xml.v0_6.impl.WayElementProcessor.end(WayElementProcessor.java:117)
	at org.openstreetmap.osmosis.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107)
	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
	at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.parseXML(BaseXMLReader.java:65)
	at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.handleXML(BaseXMLReader.java:77)
	at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:52)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "way_tags_pkey"
 ​Detail: Key (way_id, version, k)=(1571192, 1, SEMANTIC) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:120)
	at org.openstreetmap.osmosis.apidb.v0_6.ApidbWriter.flushWayTags(ApidbWriter.java:771)
	... 22 more

Jun 11, 2021 6:58:29 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
	at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

What puzzled me is that way_id=1571192 do not exist in way_tags even after this error occurs.

Why?

my_db=> insert into way_tags(way_id,version,k,v) values(1571192, 1, 'SEMANTIC','');
INSERT 0 1
my_db=>
my_db=> insert into way_tags(way_id,version,k,v) values(1571192, 1, 'SEMANTIC','');
ERROR:  duplicate key value violates unique constraint "way_tags_pkey"
DETAIL:  Key (way_id, version, k)=(1571192, 1, SEMANTIC) already exists.
my_db=> \d way_tags;
                  Table "public.way_tags"
Column  |       Type        | Collation | Nullable | Default
---------+-------------------+-----------+----------+---------
way_id  | bigint            |           | not null | 0
k       | character varying |           | not null |
v       | character varying |           | not null |
version | bigint            |           | not null |
Indexes:
   "way_tags_pkey" PRIMARY KEY, btree (way_id, version, k)
Foreign-key constraints:
   "way_tags_id_fkey" FOREIGN KEY (way_id, version) REFERENCES ways(way_id, version)

use Double.NaN as default value at WayNode coordinates to avoid confusion

org.openstreetmap.osmosis.core.domain.v0_6.WayNode.WayNode(long) initializes nodeId only.
latitude and longitude is initializes implicitly, resulting in 0.0

Since 0.0 is a valid value, there is no explicit pointing on incomplete initialization.

To make incomplete initialization visible, the default values should be Double.NaN

[Question] Is it always safe to restart osmosis when it crashes?

Currently I'm experiencing the occasional timeout on the server I fetch the changes from ( Unable to read the state from the server.)

Usage is:
osmosis --read-replication-interval workingDirectory=rri --simplify-change --write-xml-change file=changes.osc
and only if that succeeds the osm2pgsql starts

In the ideal world I would just want to restart it on failure so it can try again, but I do not know the risk of missing one update in some edgecase.
I guess it boils down to not being able to crash in any way after updating the state.txt

How to run and debug osmosis?

Hello, I couldn't figure how to debug certain code of osmosis. For example writing and reading PBF file. Is there a documentation about this? In best case for IntelliJ?

When I run osmosis.osmosis-core.main with

-p
crosby.binary.osmosis.BinaryPluginLoader
--read-pbf
/tmp/data_original.pbf
--write-pbf
/tmp//out.pbf

in IntelliJ. I get

image

I already added all modules as compilation dependency to osmosis-core

image

I followed https://github.com/openstreetmap/osmosis/blob/master/doc/development.md ... but almost at the end it fails with
image

Also, I'm not sure if the docker approach is capable of debugging?

Any ideas how to debug using IntelliJ is greatly appreciated!

ERROR: insert or update on table "current_way_nodes" violates foreign key constraint "current_way_nodes_node_id_fkey"

While using the following osmosis command to sync data from the planet dump,

osmosis --read-xml-change file=so-and-so.osc --write-apidb-change host=$HOSTNAME database=$DATABASE user=$USERNAME password=$PASSWORD validateSchemaVersion=no

but we are getting the following error:

org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to insert current way node with way id=6510672 and node id=9092467749.
45 at org.openstreetmap.osmosis.apidb.v0_6.impl.ChangeWriter.write(ChangeWriter.java:706)
46 at org.openstreetmap.osmosis.apidb.v0_6.impl.ActionChangeWriter.process(ActionChangeWriter.java:56)
47 at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(WayContainer.java:58)
48 at org.openstreetmap.osmosis.apidb.v0_6.ApidbChangeWriter.process(ApidbChangeWriter.java:81)
49 at org.openstreetmap.osmosis.xml.v0_6.impl.ChangeSourceElementProcessor$ChangeSinkAdapter.process(ChangeSourceElementProcessor.java:144)
50 at org.openstreetmap.osmosis.xml.v0_6.impl.WayElementProcessor.end(WayElementProcessor.java:117)
51 at org.openstreetmap.osmosis.xml.v0_6.impl.OsmChangeHandler.endElement(OsmChangeHandler.java:94)
52 at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
53 at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
54 at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
55 at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
56 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
57 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
58 at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
59 at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
60 at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
61 at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
62 at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
63 at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.parseXML(BaseXMLReader.java:65)
64 at org.openstreetmap.osmosis.xml.v0_6.impl.BaseXMLReader.handleXML(BaseXMLReader.java:77)
65 at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:51)
66 at java.base/java.lang.Thread.run(Thread.java:829)
67 Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table "current_way_nodes" violates foreign key constraint "current_way_nodes_node_id_fkey"
68 Detail: Key (node_id)=(9092467749) is not present in table "current_nodes".
69 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
70 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
71 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
72 at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
73 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
74 at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
75 at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132)
76 at org.openstreetmap.osmosis.apidb.v0_6.impl.ChangeWriter.write(ChangeWriter.java:702)
77 ... 21 more

granularity doesn work as expected

I want to reduce the filesize for a very big Map. I want to use the "granularity"-Tag in the --write-pbf Command. But always when i use an other value than "100" the file in Maperitive seems to be empty. the Filesize is almost the same Size as the read-file.
I use this command:

osmosis --read-pbf file="highway2_05.osm.pbf" --write-pbf granularity=10000 file="highway3.osm.pbf"

How to merge two OSM files in Kotlin?

I am trying to figure out how to merge two OSM files using the java library as opposed to the command line tool. The command line command I am trying to implement in Kotlin is "osmosis --rx file1.osm --rx file2.osm --merge --wx merged.osm" which should merge the two OSM PBF files.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.