Giter VIP home page Giter VIP logo

openhistoricaltiles's Introduction

OpenHistoricalTiles

This repository (OpenHistoricalTiles) has a gh-pages branch, which is used to create a more customized demo. https://open-historical-map-labs.github.io/openhistoricaltiles/

The tilejson URL is https://vtiles.openhistoricalmap.org/index.json and this URL will provide configuration info for vectile consumers. This is served by Tessera and an Apache proxy.

The root of that service website https://vtiles.openhistoricalmap.org/ will bring up a demo using raster tiles, but will also point to vector tile sources.

Launching Tessera and PostgreSQL

To start and stop the services:

/home/ubuntu/OPENMAPTILES/openhistoricaltiles/tessera_service.sh stop
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/tessera_service.sh start
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/postgresql_service.sh stop
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/postgresql_service.sh start
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/ohmdiffs_service.sh stop
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/ohmdiffs_service.sh start

The Tessera service runs on port 8080 on all interfaces, Internet and localhost. The firewall blocks the general public from accessing port :8080 and there is an Apache proxy redirecting all website requests to localhost:8080 so Tessera answers them. This Apache proxy provides HTTPS/SSL service.

PostgreSQL runs in a Docker container, but redirects port 5432 so it may be used like a typical, non-Dockerized PostgreSQL.

OSM diffs are consumed every 5 minutes by their service, which is a Docker wrapper over imposm run The output is sent to /home/ubuntu/OPENMAPTILES/openmaptiles/data/ohmdiffs_service.log and the logfile is cycled periodically (daily, 7 days kept). See /etc/logrotate.d/openhistoricalmap_diffs for the log-cycling configuration.

PostgreSQL

You may use PostgreSQL as usual:

psql -U openmaptiles openmaptiles

Tessera Config File

The file openmaptiles.tm2source/data.yml is the configuration for what data services exist. It also configures global stuff like the minzoom and maxzoom If this file is changed, you will need to restart tessera: sudo service tessera restart

This file is based on the build/openmaptiles.tm2source/data.yml file which was generated during openhistoricaltiles setup. It was then split off and modified for cleanup and additional attributes and layers.

Terminology note: "Datasource" is what we normally call a "layer" or "feature group" such as buildings, lakes, or coastlines. A "Layer" is the whole data service.

This file is recompiled whenever the OpenMapTiles step make build/openmaptiles.tm2source/data.yml is run (which is part of make). So you should stow your edits someplace else.

Debugging Tip

Tessera performs a query against each datasource (layer) by running the defined SQL query. This means that if any of your SQL definitions don't work properly, the service won't start properly. Fortunately, it will crash with a verbose SQL error and you'll know what's going on.

In fact, while developing the datasourcelayers, you may find it useful to manually start the service without systemctl so you can see the error messages:

/home/ubuntu/OPENMAPTILES/openhistoricaltiles/tessera_service.sh stop
/home/ubuntu/OPENMAPTILES/openhistoricaltiles/tessera_service.sh start

openhistoricaltiles's People

Contributors

danrademacher avatar greeninfo avatar gregallensworth avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openhistoricaltiles's Issues

Datefilter map: Make URL params more forgiving of missing date

URL param is as designed: Black Rock City, 2008-2010

https://openhistoricalmap.github.io/openhistoricaltiles/datefilter/#15.00/40.77470/-119.21600/2008-01-01,2010-12-31/

The URL param parser expects all params, or else will not process any of them. For example, this URL would not even zoom to the location:

https://openhistoricalmap.github.io/openhistoricaltiles/datefilter/#15.00/40.77470/-119.21600/

Make the URL param parser more forgiving, so the date filter is effectively optional and it will still pan & zoom even if not setting a date filter.

Replace non-OHM borders with OHM borders at zoom < 6

Right now, we're currently seeing modern boundaries at zoom < 6.

As a user, I would like to eventually see the borders only for whatever time we select, and I'd like to be able to select a time for any zoom.

So, at any zoom, the only borders that should be rendered should be:

  • in OHM
  • relevant for whatever time is selected (later work)

Building footprints lack name/date info

Building footprints do not have a start/end date listed, nor do the osm_building_polygon and osm_building_multipolygon tables even have a name field. We would like for building footprints to have all three: a name, and start + end dates.

May need to backtrack and check out OMT's techniques for loading buildings... or even purge-and-reload if we need to go back to an initial import.

Diffs consumption re boundaries

Expanding on the work done in #13 in regards to boundaries...

We've been following the openmaptiles method, in which boundaries are loaded. This is less than optimally, as it takes a half-hour to tun and is "just a second thing" Ideally, we would modify the mapping.yaml so that the boundaries were imported via imposm3 like everything else.

See what we can do about this. Ideal targets would include:

  • remove the ohmborders tasks and documentation, as they would not be used anymore

  • adjust the mapping.yaml to import boundaries. refer to the ohmborder source code to find the filtering and features

  • purge the DB and reload, confirm presence of boundaries

  • make some boundary-related change to OHM data, watch it propagate into vector tiles in our usual timeframe

Crib a Tangram style as basemap

We know we want to make a general purpose Tangram style for two reasons:

  1. Better underlayment for the Date Issues map (https://openhistoricalmap.github.io/openhistoricaltiles/dateerrors/#14/40.7721/-119.2137)
  2. Better future use for a date filter slider

We know lots of people who worked on these. And the endpoints on this page provide the style files:
https://www.nextzen.org/

Live previews:
https://tangrams.github.io/cinnabar-style
https://tangrams.github.io/refill-style
https://tangrams.github.io/tron-style
https://tangrams.github.io/walkabout-style

Based on that, I would say Cinnabar is best starting point:
image

Typical "map" colors, no relief, nothing too fancy.

style files:
https://www.nextzen.org/carto/cinnabar-style/9/cinnabar-style.zip

Add name information to more layers

Expanding on #1 so as to make the best of #4

Labels are displayed as the XXX_name layer, e.g. waterway_name and layer_transportation_name These are point layers generated by the OSM import tools, into wholly-new Db tables such as layer_transportation_name

However, the geometry-only layers such as water and waterway and transportation have minimal SQL queries for just the geometry This means that mouse-and-tooltip and click-info behaviors do not work properly -- no attributes are included.

Add at least the name field to these layers:

  • water
  • waterway
  • streets
  • building
  • poi
  • place
  • park

There may be more in the future, but for the immediate dataset this would be an improvement.

Build map style to highlight date issues

http://editor.openmaptiles.org/ has a "sourcers" menu where we can load our own endpoint:

image

Example by Seth using local tiles:
image

Seth's comments:

This uses natural earth + other datasets for additional context; i think we'd want to drop them for "pure" historical data. OpenMapTiles has a mechanism for generalizing layers, so i think we should be able to use OHM data for low zooms

Minor change in Plus/minus buttons, lose home, underline dates

Now that we have the timeline live and working, Tim and I agreed that Plus/Minus/Home don't make a lot of sense where we have them, and Home not at all.

Let's change this around like this instead:

image

Seems pretty clear, then once done I'll send it out to the OHM community list for wider feedback

TimeSlider demo, create and implement URL hash control I/O

Expanding on the MBGL TimeSlider control demonstration, built in #33

Implement on this demo, a new map control which will write URL params of the map's state, and on startup will read URL params to set the map's state.

  • X/Y/Z center and zoom
  • Selected date
  • Currently-selected date range
  • MBGL version for this demo

    • Serializer: on timer or event, update URL hash
    • Deserializer on startup: read URL hash, and apply to initial Map and TimeSlider
  • Leaflet version for its demo

    • Serializer: on timer or event, update URL hash
    • Deserializer on startup: read URL hash, and apply to initial Map and TimeSlider

New subdomain for vectortiles endpoint

Pulling out from #1:

Please set up a new domain hostname record vtiles.openhistoricalmap.org

Set this as a CNAME record pointing to ec2-18-209-171-18.compute-1.amazonaws.com

Add osm_id to all datasources (layers)

Relevant to #3 in which we would be highlighting features which have bad date data.

In order to *do * anything about these features, we would need the osm_id

  • add to the layer back-end query wrappers, and to the Datasource items, the osm_id
    • accept that not all data are from OSM/OHM, so "" will be normal
  • front-end, add as proof/test.

Time slider in a world of divergent date formats

As we have known from our first engagement with this work, dates come in many different formats in OHM.

Our code those far has assumed YYYY-MM-DD, and ideally the community would settle on a single standard and go with that. Though uncertainty is part of the picture, and what to do about that?

Here's an example of an area with lots of year-only dates in 1830:
http://www.openhistoricalmap.org/edit#map=18/49.54794/17.73503

Timeslider demos here are blank because dates don't conform to our code's expectations. Do we start down the road of trying to guess dates from different formats? Or do we potentially look to combine Date Errors highlight work with timeslider?

Data / Carto issue: Great Lakes vanish at z6+

Starting view: http://localhost:9000/#5.50/43.13520/-78.17927/

Zoom in a notch more, and see that it changes from z5 to z6 and cartography makes a transition.

Unfortunately part of this transition is that the Great Lakes vanish!

Same here, with some Canadian lakes slightly northeast in Quebec / Ontario: http://localhost:9000/#5.97/49.53909/-74.88084/

Something in the SQL queries is filtering these out? The change happens at exactly 6.00 so I'm guessing a query, as opposed to a vector tile rendering glitch.

New Borders data process: planning issue

Here's what we said in latest scope:

The current approach uses the same method as OSM, consuming borders from Natural Earth as a totally separate, nondiffed process. It would be much better to draw borders directly form OHM, since these are likely to be a main element of interest to OHM mappers. Here's the approach:

  1. Create IMPOSM 3 mapping for borders including generalization
  2. Modify OpenHIstoricalTiles vectiles schema to ingest borders
  3. Use those in carto display, dealing with double overlap borders, so hard to use dashed lines, so skip those
  4. Remove the OSM-borders tool

Seth do you think these are specific enough that we can just treat 1-4 as distinct issues, or should we break down further?

URL params on dateissue map don't load map to expected location

During Slack discussion Gregor said:

https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#11.89/50.36570/2.81261/
That’s that area near Givenchy. I see that everything has bad dates and nothing had properly formatted dates, so expect the map to be blank.

I clicked that link expecting it to load a location in France, zoomed in at z11.89.

Instead, I get default center on mainland US:
image

This is the same view as if I load https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/ , which leads me to suspect that URL has is not parsed on pageload

Blank dates: Missing data or eternal features?

Pulling this out into an issue:

e.g. for the blank start or end dates, I'd like to treat those as data entry omissions. For dawn of time / present day, how badly would things break if we used a wildcard, like '*'?

Seems like two concerns here:

  • What can we accommodate in the timeline slider/date filter? What's the lest hack way here?
  • What do OHM editors/contributors think about a wildcard?

If we're going to use an arbitrary string, I suspect we'd want something other that an asterisk, which seems likely to get mistaken for an actual wildcard.

This will also be relevant to Open-Historical-Map-Labs/openhistoricalmap-docker#2 -- if we implement a date picker there to help with data formatting, how do we also allow someone to enter whatever thing means "eternal"?

New combined map of date issues + date filtering

Our larger goal here, is a combined OHM map that could ultimately act as the front-page map of www.openhistoricalmap.org At its base this could be a merge/mashup of the dateissues and datefilter maps, with a few additional behaviors.

Reference:

http://www.openhistoricalmap.org/

https://openhistoricalmap.github.io/openhistoricaltiles/datefilter/#14.00/40.76861/-119.23122/2009-08-15,2009-09-15/

https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#14.00/40.77159/-119.21816/

  • new MBGL map, with date filter for features in which dates are good

  • behavior: click feature, get info

  • control: layer switcher, toggles a date issues behavior for those with invalid/missing dates (panel listing features + orange/red features on map)

  • behavior: click feature, get info for bad-date items

  • layer switcher should also allow for picking a real basemap for reference. Maybe OSM?

  • control: geocoder address search and zoom

  • control: geolocation zoom

  • control: welcome modal

  • layer control: Map Notes This may require some CORS-related work first as it's on a different URL origin, so we can access the endpoints over at www.openhistoricalmap.org

  • map control: Add Note, adds a temporary marker to the center of the map and opens a tool for posting to OHM endpoint where a note is placed onto the map (then refresh the layer?)

  • improve styling & cosmetics somewhat, working with Tim

Prototype Date Filter

Black Rock City is our test case:
https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#15.00/40.77470/-119.21600/

This should work for filters that can separate 2008 and 2009.

Allow user to select start_date and end_date.

  • Someday a slider, but for now could be a basic date-picker.
  • Or if single-day granularity is an issue, it could also be a Year Picker that doesn't allow closer resolution than a whole year *and appends -01-01 to start and -12-31 to end

I'm still a little foggy on how exactly these filters work in MBGL. So there might be discussion needed and expansion of this into multiple issues.

New building not imported into osm_poi_polygon

A new building was submitted last night. Its geometry came across, but not a lot else.

The submitted feature's XML was:

<?xml version='1.0' encoding='UTF-8'?>
<osmChange version="0.6" generator="Osmosis 0.46">
  <create>
    <node id="2083847608" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3975147" lon="-104.7099774"/>
    <node id="2083847609" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.397533" lon="-104.7098836"/>
    <node id="2083847610" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3974953" lon="-104.7098742"/>
    <node id="2083847611" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3975259" lon="-104.7097401"/>
    <node id="2083847612" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3974289" lon="-104.709681"/>
    <node id="2083847613" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3973921" lon="-104.7098755"/>
    <node id="2083847614" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.397433" lon="-104.7098876"/>
    <node id="2083847615" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825" lat="40.3974197" lon="-104.7099546"/>
    <way id="198201500" version="1" timestamp="2018-08-08T04:20:14Z" uid="583" user="danrademacher" changeset="3825">
      <nd ref="2083847608"/>
      <nd ref="2083847609"/>
      <nd ref="2083847610"/>
      <nd ref="2083847611"/>
      <nd ref="2083847612"/>
      <nd ref="2083847613"/>
      <nd ref="2083847614"/>
      <nd ref="2083847615"/>
      <nd ref="2083847608"/>
      <tag k="addr:city" v="Greeley"/>
      <tag k="addr:housenumber" v="1890"/>
      <tag k="addr:postcode" v="80931"/>
      <tag k="addr:street" v="24th Street"/>
      <tag k="building" v="house"/>
      <tag k="building:levels" v="2"/>
      <tag k="start_date" v="1983"/>
    </way>
  </create>
</osmChange>

This did create an entry in osm_building_polygon that looks about right:

  • Polygon geometry is present,
  • its building column is set to house
  • the buildinglevels and start_date are present and populated as well
  • address number, street name, and ZIP are not part of this structure, so are skipped

That's all good. But...

I was expecting that the building would have a presence in osm_poi_polygon as well, where additional attributes and tags could be had, e.g. name and tags For example, OSM ID 198099771 has a presence in osm_poi_polygon with a name and a set of tags.

I see 4106 records in osm_building_polygon and 2129 records in osm_poi_polygon So about half of building polygons will not get this additional name/tag entry?

What factors determine this?

Remove NaturalEarth urbanareas

Per discussion today, remove the NaturalEarth "liver spots" indicating urban areas.

See OMT SQL file layers/landuse/landuse.sql

Note that up through z8 these NE urban areas are the only content at all, and osm_landuse_polygon derivatives don't come in until z9 As such, come up ith something to go into the higher-level tables:

  • landuse_z4
  • landuse_z5
  • landuse_z6

Make url refreshes smoother (remove flash of South Bronx)

Currently, when the demo is refreshed for a URL-specified time & place, eg. https://openhistoricalmap.github.io/openhistoricaltiles/mbgl-control-timeslider/demo/#13.401/50.05426/19.95121/600,600-1950

Immediately on hitting refresh, you briefly see a flash of the South Bronx, which is weird, as that's not where the map was before the refresh & not where it ends up after the refresh is done.

Desired behavior is not to see the South Bronx, and only to see the area you were originally looking at.

Crosswalk from MB Streets to openMaptiles schemas

for one of these styles:

We need to get a reference schema from OpenMapTiles.

We must have OMT style files in this repo somewhere.

@gregallensworth can you point a link in this issue to where the Bright map style is? Maybe that would give us enough to go on. At least a starting place.

┆Created By: Dan Rademacher

TimeSlider demo, Change OHM map style

Expanding on the MBGL TimeSlider control demonstration, built in #33

Jeff said via email: "the style for the prototype map doesn't look the same as the existing style on OHM. We'll be working on making some changes to it"

I need to get specifics from Jeff as to where they have implemented a new vector tile style, so I can merge/copy that new style into the two TimeSlider demos (MBGL and Leaflet-MBGL)

  • confirm with Jeff which map style the demos should be using; if the comparison is to some raster tiles, then we may need to create a new Issue with Tim as to cartography and implementing the target look in vector tile styles.

  • Adapt style into the MBGL Control demo

  • Adapt style into the Leaflet-over-MBGL demo

┆Created By: Dan Rademacher

Boundaries lack date information

Created in response to a comment #3 (comment) that this shape is not showing its start_date in the dateissues map.

http://www.openhistoricalmap.org/way/198180496
https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#7.38/44.91845/-69.47637/

Confirmed here, that boundaries lack any date info. I have comments in here, that boundaries seem to be imported via import-osmborder which was a separate process (and one which may be separate from the later diff-consuming process???), and that these boundaries lacked date information in the osm_border_linestring table and its derivatives.

Remove the OSM-borders tool

Broken out from #35. After #47 is done (which implies wholly new borders in maps and upstream data) then we strip out the OSM borders tool

Time Slider: There is no year 0

In the Gregorian calendar there is no year 0.

  • Dates skip directly from -0001-12-31 to 0001-01-01
  • The delta between June 30 +1 and June 30 -1 is not 2 years, but 1 year.

For UI purposes, this is an Interesting quirk, since input[type-"range"] form controls do not allow for individual values being skipped. It may be necessary to write something new completely from scratch.

┆Created By: Dan Rademacher

Datefilter map: Add panel of examples

Add to the datefilter map a list of examples, hashes to places where the date filter is interesting.

  • Black Rock City 2008 and 2009 come to mind

Others?

Odd minute replication requests

Is this you guys?

18.209.171.18 - - [28/Aug/2018:11:44:15 -0400] "GET /replication/minute/000/009/785.state.txt HTTP/1.1" 404 491 "-" "Imposm3 0.4.0dev-20180802-9be1e5a"

Map load event still a bit twitchy

Reference:

https://openhistoricalmap.github.io/openhistoricaltiles/datefilter/#14.00/40.76861/-119.23122/2009-08-15,2009-09-15/

https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#14.00/40.77159/-119.21816/

In both cases, starting from a cleared cache etc. the page loads but the initial state of the map doesn't always catch up. The dateissues one won't always load the sidebar content, and the datefilter one won't always load the vector data e.g. Burning Man 2008

It almost always works as I shift-reload during development and after the first time loading... but sometimes it doesn't.

Maybe keep banging on it, see if we can narrow it down and improve it?

Use new borders in vectiles for carto display

Broken out from #35:

Use borders ingested in #46 in carto display, dealing with double overlap borders, so hard to use dashed lines, so skip those

Since we're using OpenMapTiles OSM-Bright style now and not yet scoped for implementing new style, we assume this is where that change will need to be made.

Rendering bug building=house

Some building=house are not rendered in vector tiles.

http://www.openhistoricalmap.org/way/198280708 has building=house and start_date=1830 but is not rendered (https://openhistoricalmap.github.io/openhistoricaltiles/mbgl-control-timeslider/demo/#18.637/49.54870/17.73439/1859,1700-2018), although it has very same notation as other houses.

http://www.openhistoricalmap.org/way/198283485 has building=house and start_date=1944. Again, it is not rendered after 1944, even though it has same notation as other houses: https://openhistoricalmap.github.io/openhistoricaltiles/mbgl-control-timeslider/demo/#18.637/49.54897/17.73465/1944,1700-2018 (Before 1944, there is an other house with different shape.)

Datefilter map: Date filter enhancements

Per chat today, the datefilter map's timeslider could use some improvements:

  • Apply filter only when Apply button is clicked; current change handler updates too aggressively, and "feels clunky" when one is updating the dates, e.g. in the middle of entering new dates one enters a range of "2008-01-01 to 2000-01-01" momentarily.

  • Existing calendar widget allows one to select a year, but that's not obvious: one must mouse over the year to discover this. Can this behavior always be in place so folks can find it?

  • Add UI below the date boxes, to go forward or backward 1 year and forward or backward 10 years. This would adjust both dates, "shifting the window" with one click.

Add date fields to output

Relevant to #3 and conceptually similar to #6 ... add the start_date and end_date fields, so they are exposed via the web services. They are present in all/most of the osm_XXX DB tables, just need to rewrite the various views and wrapper functions.

This is conceptually similar to work done in #6 but will be on a much larger scale as it would be relevant to every (or almost every?) view and wrapper function.

Documentation improvement: get OHM planet file in PBF format

See the Import OHM PBF section of https://github.com/OpenHistoricalMap/openmaptiles/blob/openhistoricalmap/README.md#import-ohm-pbf

The instructions here should be to acquire and load a PBF version of the OHM planet file.

Touch base with @jeffreyameyer and document a procedure here, whether there is an endpoint or URL where the OHM PBF may be had, or that it requires asking for help from other OHM staff. (and if the latter, documentation could be helpful there as well?)

┆Created By: Dan Rademacher

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.