headwaymaps / headway Goto Github PK
View Code? Open in Web Editor NEWSelf-hostable maps stack, powered by OpenStreetMap.
Home Page: https://about.maps.earth
License: Apache License 2.0
Self-hostable maps stack, powered by OpenStreetMap.
Home Page: https://about.maps.earth
License: Apache License 2.0
I had this idea while talking with some folks in the OpenStreetMap slack. I think it would be really cool to kick people back to OSM to map their own local places, improving OSM, headway and maps.earth for everybody.
I like the way it's explained here. Specific to pelias but still. https://geocode.earth/blog/2020/virtuous-cycle
Currently Headway uses bright
style but it would be nice to allow the style to be changed from the docker-compose.yml
file, as an environment entry.
For a long time I didn't think I wanted to support the full planet use-case but the fact of the matter is that's what everyone wants, and I think it would be really exciting for people and maybe hopefully convince some folks to help me work on the frontend. :)
Also consider switching to Pelias? I don't want to cause a bunch of churn but it will be extremely expensive to run a full planet installation of nominatim, and imports are very slow. Plus we currently store about 2x as much geocoding data as we really need to since we have two geocoders. Pelias has search & autocomplete, and a places API that could replace nominatim.
Currently headway is set up with a single area/city which decides which data is pulled in from all sources.
One common scenario I think should be considered: Single country, with transit direction for 1-3 cities inside that country (for smaller countries this can still be less resources than some of the larger cities)
So we may have a larger map but only transit integration for a subset of it.
Note that this is not the same as specifying arbitrary areas or combinations thereof (implementing that would solve this though),which would great but may not be as trivial to implement
It would be nice to have the option of clicking/tapping a spot on the map to enter that as the start (or end) point of the route.
Just wanted to put it out there, because, as other tickets and HN comments have mentioned, the address autocomplete isn't super sophisticated yet, so adding this functionality would make a it just a bit easier to use the routing capabilities in the meantime.
For example, my home address doesn't appear in the autocomplete at all, so I typed in a nearby restaurant which did pop up. With that location set as the destination, having the option to click near my house would be super convenient. For a variety of reasons, I find that I use that functionality on Google Maps quite a lot, actually.
Thanks for this ambitious project! I'm looking forward to following it :)
Do a bbox intersection with the feeds from mobilitydata.org, then download the ones that intersect, I guess? That makes sense for now, but if people do larger installations they may want to pick and choose which ones they download.
Hi Ellen,
docker-compose.yml
file so that each stack fetches a common image from the Registry instead of fetching it from the local folder -- which is the way docker stacks are supposed to work.It did not work because Headway saved dependencies within the containers instead of externalizing them as data.
So here is what happens:
NewYork
as an image to our Registrydocker-compose.yml
file of Philadelphia
so that the image underlying each container is fetched from our Registrydocker-compose up -d
as usual/data/Philadelphia.photon.tar.bz2
file, and for a good reason: the image had been created for NewYork
Here is the trace from the console:
Extracting photon index
+ echo 'Extracting photon index'
+ cd /
+ pbzip2 -d
+ cat /data/Philadelphia.photon.tar.bz2
+ tar x
cat: /data/Philadelphia.photon.tar.bz2: No such file or directory
pbzip2: producer_decompress: *ERROR: when reading bzip2 input stream
Terminator thread: premature exit requested - quitting...
Here is a simple fix: load the file from a mounted volume instead, eg. from the /data
folder mounted local volume.
OTP right now is the biggest memory hog by far, so it would be nice to let people disable it if they don't have memory to spare.
I just got an API key for the OneBusAway real-time API in my area, so now I have access to some real-time transit feeds. I think it would be very cool to plot transit vehicle locations on the map, and I believe OpenTripPlanner can incorporate real-time delay information into its routing which would be very cool to implement.
Getting the following errors in browser console after running make Sucre.up
.
GET /static/[email protected] 404 (Not Found)
GET /static/[email protected] 404 (Not Found)
Error: Source layer "park" does not exist on source "openmaptiles" as specified by style layer "park".
Error: Source layer "park" does not exist on source "openmaptiles" as specified by style layer "park_outline".
Probably OSM Bright, because if memory serves it's easier to self-host than some of the others.
This was tracked in my head but it fell off my radar so it makes sense to track it explicitly I think. The trip planner frontend is the only mode that is completely unusable right now. Whereas the others are just missing ETAs and a decent nav experience, the trip planner is completely lacking because it doesn't display steps at all, only polylines representing each sub-trip.
I tried HEADWAY_FORCE_BBOX="-74.051971 40.622813 -73.893013 40.751418"
and it still does not work
Hi Ellen,
Thank you for the great work on NYC! I have tried installing the stack and it passed the error point I had mentioned.
However, I get yet another blocking error further down, this time at ITAG=headway_build_fonts
, when installing fontnik.
Here is the trace in the console:
Step 2/23 : RUN npm install fontnik
---> Running in a8d8a75e27b6
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm ERR! network request to https://registry.npmjs.org/minimist failed, reason: getaddrinfo ENOTFOUND registry.npmjs.org
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2022-08-11T18_21_08_481Z-debug.log
The command '/bin/sh -c npm install fontnik' returned a non-zero code: 1
make: *** [Makefile:151: /mnt/cru/kuq/hwk/data/fonts.tar] Error
As I understood it is an error while connecting to the NPM registry, I tried to connect manually using curl:
curl https://registry.npmjs.org/rc
And I got a correct response.
So I assume there is still an error there.
S.
Originally posted by @asitemade4u in #66 (comment)
So, I was able to build Philadelphia and to launch the docker stack successfully.
But when I access the server on port 8080:
I tried (on Debian) with Firefox and Brave browsers and the same.
So:
NewYork
stack.env
file: HEADWAY_FORCE_BBOX="-74.051971,40.622813,-73.893013,40.751418"
I would be thrilled to help finance the project but would prefer to donate a one time donation.
I know that is possible with GitHub: please allow it.
Right now we only hit the pelias autocomplete endpoint which doesn't perform interpolation, among other issues. Letting users press enter to see an actual search result page would give us the opportunity to hit the search endpoint and get additional address coverage that way.
It would be really nice to have continuous integration that clones the repo, builds it for a small city and at least does health checks for all the services we bring up. There's a lot of churn happening right now but when the dust settles that seems like a good goal.
I also want to do more code review. I've sworn off committing directly to main because it's clear that other people actually care about this project now, but just now with #23 I realized there isn't really anyone around to review anything except one person, and I don't want to shove all the review work onto one person. So if you want to help out with that leave a comment here!
See: #82 (comment)
We should push images in the CI config to the GitHub container registry, then headway users can pull those images instead of having to build their own. Care will need to be taken to ensure that images can be overridden for local development.
This init container thing we do right now doesn't scale well to cover the planet. It's not reasonable to have one copy of each artifact per pod when the artifacts are tens or hundreds of gigabytes. My current plan is to create a longhorn RWX volume claim and share that between all of the pods. Each pod will mount the volume claim as read-only and serve data out of it. I'll have to write a new service to manage the data that's kept in the pvc and help do the following:
That service will need to be able to access the cluster's api (kubelet? still getting up to speed on terminology) for items 2 and 3.
Right now nominatim is just used as a source of truth for OSM data. Only the lookup endpoint is exposed. There's no reason we couldn't replace it with something more lightweight built on top of postgis. That could drastically reduce the memory requirements for the full Headway stack, potentially even to the point that the whole planet could be hosted with 64GB of RAM which is about the point where it becomes feasible to do.
Headway is GREAT and works fast.
I was wondering if it was possible to add one or several personal layers of POIs on top of the OSM tiles.
It would be useful, for example, to add GPS tracks, geolocate photos taken, etc.
I don't want people getting the wrong impression about Headway's transit capabilities now that we don't build with OTP by default.
In the same vein as my previous #82 post, I think you should "make visible" the PostgreSQL database installed by the Nominatim stack.
That would allow users to rely on their own database (in our case, one that is installed on another server) instead of deploying a database local to Headway.
That would also allow a better use of already existing resources, such as database backups, replication, etc.
Now that custom extracts are supported, we really need viewport biasing for autocomplete and (once it's implemented) the search results page.
Offline geocoding would probably be best accomplished by getting Carmen to run in the browser, per #50. This seems like it will require removing rayon and rocksdb from carmen-core, rewriting or repackaging vtquery and probably also other things. I'm guessing Carmen itself will need to be ported over to use local storage instead of the node filesystem APIs too. After the dust settles performance will also need to be evaluated in terms of response time, latency from a cold cache, and subjective quality of the results.
OpenTripPlanner would be a much better choice for transit than valhalla. It's unclear if valhalla's transit support is even maintained.
I want to make a website for the project that captures the convenience of all those sites out there that tell you to pipe curl into bash to install their software. I own the domain maps.earth and want to use it for this site.
What I'm imagining is a world map rendered with pmtiles, and a UI on top of it that lets you search for an administrative area that may not be in the BBBike extract list, e.g. Atlanta, Georgia. The UI will let people draw a bounding box over an area they care about and prepare an OSM extract for them. (tbd how to make this scale, but I have some ideas). It would also collect GTFS feeds for the bounding box, and maybe even let people disable some of them. Once everything is selected, a make command could be generated with a token for that user's selections.
Running the command would kick off the OSM extract and spin wait on the user's machine while it's running, then download the extract and GTFS feeds and kick off the rest of the build per normal.
This is a long-term suggestion but I've looked at the logs for the demo server and almost all of the geocoder requests are for cities outside of the Seattle metro area. It turns out people really just want to see their own cities on the map before they get really interested.
This should solve the issues that show up when you click on a result that is an OSM way (or maybe also a relation) and the rich display name is replaced with the name of the neighborhood or suburb.
I went to ToorCamp 2022 recently and had a conversation with someone who suggested I use Gradle for the build system of Headway instead of Make. They mentioned a few benefits, including the fact that if a build step fails then the artifacts from that build step aren't cached or written to disk. It would also cache build artifacts in a smarter way, meaning things don't get evicted from the cache needlessly.
In other words if I ran:
make Adelaide
make Seattle
make Adelaide
In step 3, Gradle would use the cached Adelaide artifacts from step 1.
Both of these seem huge to me, because they would completely negate the benefits of using docker build
instead of docker run
, reduce docker image clutter, and reduce wasted effort spent on updating Makefile style to work on various platforms/versions.
If nobody objects to the use of Gradle in the near future, I think I'll begin work on this. I'm entirely open to using just about anything that provides the cache and reproducibility-on-error benefits that Gradle seems to, just want to evaluate everything on its merits. :)
Just splitting this out into its own issue to prevent clutter in #50.
Valhalla is currently building with emscripten in https://github.com/headwaymaps/valhalla but a lot of the porting process remains. Tile fetches and tile caching are the most obvious examples, requiring the use of emscripten's networking capabilities and filesystem capabilities respectively.
Looks like bash variables aren't being assigned properly in the makefile.
Steps to reproduce:
git clone $HEADWAY_GIT_URL
cd headway
cp .env.example .env
make Amsterdam
Expected: Build succeeds.
Actual:
ITAG=headway_build_gtfs_$(echo Amsterdam | tr '[:upper:]' '[:lower:]')
HEADWAY_BBOX=$(grep "Amsterdam:" web/bboxes.csv | cut -d':' -f2)
docker build ./gtfs_build --build-arg HEADWAY_BBOX="${HEADWAY_BBOX}" --tag ${ITAG}
flag needs an argument: --tag
See 'docker build --help'.
make: *** [/Users/evinsellin/code/headway/data/Amsterdam.gtfs.tar] Error 125
I'll have a fix for this shortly.
When hosting behind a reverse proxy using https, tiles cannot be loaded.
I was able to track it down to the magic_string_magic
dummy host that is defined in https://github.com/headwaymaps/headway/blob/main/web/nginx.conf.template#L53.
It seems to only redirect accesses to http://magic_string_magic
, but when the whole page is https
, also this call will be https://magic_string_magic
.
I was able to fix it by manually changing both occurrences of http://magic_string_magic
to https://magic_string_magic
(but of course this only works temporary until the container gets re-created).
I think a proper fix would be to have an ENV var defining whether the whole stack is hosted behind https and changing the entries in nginx.conf.template
accordingly (could maybe even be dynamically parsed from HEADWAY_PUBLIC_URL).
Use of OpenStreetMap data requires attribution to be provided when data from the project is utilised.
Pelias doesn't offer a url param to control how strongly to bias the results towards the focus point, so in many cases I've found that it doesn't bias strongly enough. Look into getting this implemented upstream?
It would be really nice to have a good kubernetes config or config template for Headway. See #39 for additional information.
#21 nominally added this feature but when I ran it I found the following error message:
rm -rf /home/ellen/dev/headway/data/*.mbtiles
rm -rf /home/ellen/dev/headway/data/*.nominatim.sql
rm -rf /home/ellen/dev/headway/data/*.osm.pbf
rm -rf /home/ellen/dev/headway/data/*.photon
rm -rf /home/ellen/dev/headway/data/bbox.txt
rm -rf /home/ellen/dev/headway/data/sources
rm -rf /home/ellen/dev/headway/data/nominatim_flatnode
rm -rf /home/ellen/dev/headway/data/nominatim_pg
docker images | grep ^headway_build_ | tr -s ' ' | cut -d' ' -f3 | xargs docker rmi
Error response from daemon: conflict: unable to delete f7354fa160e3 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 32a0db2e6aa4 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 7687b7da0d06 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 7f5243e4dfa0 (must be forced) - image is referenced in multiple repositories
make: *** [Makefile:150: clean_all] Error 123
Hi and thank you for this great project!
Unfortunately for me, I can't go through with the building of NewYork base data.
The make NewYork
command begins failing at 16:53:00.408
with this message:
ERROR (OTPMain.java:55) The provided transit date have no trips within the configured transit service period. See build config 'transitServiceStart' and 'transitServiceEnd': [2012-07-28, 2025-07-28]
16:53:00.415 INFO (OtpStartupInfo.java:47) OTP SHUTTING DOWN (version: 2.1.0, ser.ver.id: 2.1.0, commit: 0d24cc0a3f7ddddf1db87d805e09fd0ca70eb5b5, branch: master)
The command '/bin/sh -c java ${JAVA_MEM_ARGS} -jar /otp/otp-shaded.jar --build --save .' returned a non-zero code: 100
make: *** [Makefile:101: /mnt/cru/kuq/headway-main/data/NewYork.graph.obj] Error 100
As a result, the transportation data is not buit and the whole stack fails after Valhalla fails when launched.
Here is the (expected) error message in Valhalla's docker console:
cp: cannot stat '/data_mount/NewYork.valhalla.tar': No such file or directory
Please help.
Thank you. Stephen
Lots of users have asked for this, and it could alleviate the need for a really slick bootstrapping UI (see #49). There are a number of blockers, mostly privacy related:
edit: These are just the hard blockers, there's a lot of architectural stuff that would need to happen before this becomes feasible.
It would be nice if there was a way to update them too but that sounds like a can of worms so it probably just makes sense to track that separately.
I just found this: graphhopper/graphhopper#1743 and seems like this might not be a priority for the graphhopper team. I know OTP supports transit shapes. I was also not as impressed with Graphhopper for bicycle routing as I have been with Valhalla.
What I'm envisioning is that the user's location could be projected onto the route polyline, and the scroll position along the route timeline set appropriately. Bonus points to adjust the ETA based on current location and the splits of the steps remaining. Extra bonus points to adjust the split of the current step based on how far through it the user is.
Super bonus points to adjust the polyline style for the portion of the polyline that's already been traveled.
TBD whether Valhalla's GPS tracking service makes sense to use here. The privacy implications of that are pretty bad if the server is untrusted, but implementing that wouldn't necessarily block implementation of offline routing because Valhalla does compile to WASM.
I think the reason that Headway got a lot of traction on HN is because it currently, right now, solves a problem that people have. It provides maps for everyday use in a chosen metro area, without compromising privacy. If the project moves towards decentralized or federated hosting of an entire world map, I want to make absolutely certain that I'm personally happy with the level of privacy it provides (see: #50). But I also want to make sure that I'm not forgetting that a lot of people don't want to make requests to servers they don't control, at all, period, end of story. Headway must continue to be self-hostable. This issue exists to track that. The default should be for headway to bring up a self-hosted maps stack. We may choose to add a make planet
target or some environment variables that control federation, but I don't want to lose track of what made people so excited about Headway in the first place.
Bottom line: Headway instances should never call out to external servers at runtime unless specifically configured to do so. Nor should they ever serve a page that solicits clients to call out to any external servers unless specifically configured to do so. At least, that's my opinion.
I'm noticing that since nominatim is not installed on the (container local) postgres server on the photon import step, the following error is yielded:
ERROR: could not access file "/usr/local/lib/nominatim/module//nominatim.so": No such file or directory
ERROR: function public.transliteration(text) does not exist
Could be resolved by either installing the module inside the photon container, or use a shared postgres server (either independent or the one bundled in the nominatim container)
Or maybe this is a non-issue?
This may improve search result quality on https://maps.earth/
Right now I know of these cartography issues:
But there may be others.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.