Giter VIP home page Giter VIP logo

Comments (7)

orangejulius avatar orangejulius commented on July 19, 2024

Hey @Binternet,
That's unrelated to the openaddresses download, fortunately. It will be fixed very soon with pelias/whosonfirst#324

from dockerfiles.

Binternet avatar Binternet commented on July 19, 2024

Not sure I understand since the error occurs in openaddresses and not whosonfirst

from dockerfiles.

orangejulius avatar orangejulius commented on July 19, 2024

The downloads for all data sources are performed simultaneously, so the error/info messages are interleaved. It's definitely confusing, but faster :)

This issue is well known because it's due to the Mapzen Shutdown and was also reported in #50

If you have a minute, try the workaround mentioned in #50 (comment). A report of how well it works would be much appreciated.

from dockerfiles.

Binternet avatar Binternet commented on July 19, 2024

Just tried it, i'm still getting this error:

> [email protected] download /code/pelias/openstreetmap
> node util/download_data.js

[download]       https://dist.whosonfirst.org/sqlite/whosonfirst-data-latest.db.bz2
info: [download] Attempting to download selected data files: http://s3.amazonaws.com/data.openaddresses.io/openaddr-collected-global.zip
info: [download] Downloading http://s3.amazonaws.com/data.openaddresses.io/openaddr-collected-global.zip
info: [download] Downloading sources: http://download.bbbike.org/osm/planet/planet-latest.osm.pbf
error: [download] Failed to download data Error: Command failed: unzip -o -qq -d /data/openaddresses /code/pelias/openaddresses/tmp-172zUU7skhrllF.zip
[/code/pelias/openaddresses/tmp-172zUU7skhrllF.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /code/pelias/openaddresses/tmp-172zUU7skhrllF.zip or
        /code/pelias/openaddresses/tmp-172zUU7skhrllF.zip.zip, and cannot find /code/pelias/openaddresses/tmp-172zUU7skhrllF.zip.ZIP, period.

    at ChildProcess.exithandler (child_process.js:199:12)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:920:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:230:5)

npm ERR! Linux 4.4.0-1049-aws
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download"
npm ERR! node v6.13.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] download: `node utils/download_data.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] download script 'node utils/download_data.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the pelias-openaddresses package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node utils/download_data.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs pelias-openaddresses
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls pelias-openaddresses
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /code/pelias/openaddresses/npm-debug.log

from dockerfiles.

orangejulius avatar orangejulius commented on July 19, 2024

This looks like an out disk space error. The OpenAddresses full planet zip is 10GB, Who's on First requires 15GB, and it looks like you're trying to download a full OSM planet file (39GB).

Even assuming you have the disk space for all that, this import is not likely to succeed with the Dockerfiles setup.

If you haven't tried running Pelias with a smaller import yet, I'd suggest you start there. After that, take a look at our full install guide if you haven't yet. It's a little out of date, but pay special attention to the "Considerations for full-planet builds" section if you really do want to import the whole planet. Let me know if you have questions with the full planet build :)

from dockerfiles.

Binternet avatar Binternet commented on July 19, 2024

Well I am trying to run a full-planet on an AWS i3.4xlarge instance, i've managed to run it with the Dockerfiles setup for the default Portland data, but i'll try again with the full install guide. will keep you posted!

from dockerfiles.

orangejulius avatar orangejulius commented on July 19, 2024

Well then, an i3.4xlarge is a little different than say a personal laptop :)

Keep in mind it will take several days and you'll probably want a dedicated Elasticsearch cluster outside of the one provided by the Dockerfiles. The importers should work fine for a full planet import, but Elasticsearch will need to be treated with more care than this repo can likely provide. It's meant just for getting up and running quickly.

from dockerfiles.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.