@nickpeihl I think this repo should be the base of the new file service (vectors.maps.elastic.co), just like the landing-page repo is used for maps.elastic.co.

I propose the following usage:

• on master merge, Jenkins runs some shell script in this repo, e.g. build-site.sh
• The build-site.sh simply builds and runs a docker container, passing it several env variables with credentials (e.g. VAULT_TOKEN / VAULT_URL to be able to read needed secrets like google cloud password)
• The docker container generates the desired state of the vector directory - including the manifest files - one per version, and all data files. Note that if needed, some files may be present multiple times under different names (e.g. a numeric ID and a proper name). Afterwards, the docker rsyncs the files to the cloud.
• There will be an env variable for RUN_PROFILE, e.g. "prod" or "staging". Docker will use it to upload data to either production or staging buckets, and will also generate different URLs inside the manifest.
• When RUN_PROFILE is "prod", docker should also do an additional step - backup the content of the production buckets to the archival bucket - just like we do with the landing page.

We will then create a job template to run this, with two instances - staging and prod. Staging will be triggered on master merge. Production will be manual - you will have to login into kibana-ci and run it yourself when ready.

CC: @thomasneirynck

We want to discourage users from joining on label and name fields. Instead we plan to assign each field as type id or property. Property fields should not show up as Join Fields in Kibana by default. Enabling the use of property fields as a join field could be configured as an option in Kibana.yml.

We would want to remove the catalogue.

In order for Kibana to support reverse-proxies (elastic/kibana#29825 (comment)), we need to make the EMS-endpoints in Kibana configurable. Catalogue introduces unnecessary cruft.

Older versions would still need to support the catalogue, but this can be a transparant change in an upcoming version.

Noticed that the USA States map has an enormous square as one of the pieces.

Try this in Vega:

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "config": {"kibana": {"type": "map", "latitude": 40, "longitude": -97, "zoom": 4}},
  "data": [
    {
      "name": "usastates",
      "url": {"%type%": "emsfile", "name": "USA States"},
      "format": {"type": "json", "property": "features"}
    }
  ],
  "marks": [
    {
      "type": "shape",
      "from": {"data": "usastates"},
      "transform": [{"type": "geoshape", "projection": "projection"}]
    }
  ]
}

Just scroll waaay out and you'll find the giant square.

To better distinguish the type of vector file, each file in the data directory and the build (dist/files) directory should have a .geo.json or .topo.json file extension for GeoJSON and TopoJSON formats, respectively. This should get carried over to the vector manifest url value as well.

This does not apply to v1 and v2 manifests, so nothing will break in production if we fix this before releasing a new EMS version.

Example
china_provinces_v1.geo.json
usa_zip_codes_v2.topo.json

cc @nyurik

JSON files are not human readable. We should support HJSON as a way to write source files. This will make it easier to read and write WikiData sources that include SPARQL. HJSON can be compiled automatically to JSON as machine readable sources.

Since formats[] is an array of objects, we should consider each Object in the array a format. So rather than formats[].format.format the schema should be formats[].format.type.

For example

--- manifest_cur.json   2018-12-03 13:26:07.000000000 -0800
+++ manifest_fix.json   2018-12-03 13:26:17.000000000 -0800
@@ -15,7 +15,7 @@
       ],
       "formats": [
         {
-          "format": "geojson",
+          "type": "geojson",
           "url": "https://vector-staging.maps.elastic.co/files/world_countries_v1.geo.json?elastic_tile_service_tos=agree",
           "legacy_default": true
         }

See https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961

In order to enforce backwards compatibility and avoid breaking changes, we need to be able to specify which version of Kibana data sources can work with. For example, older Kibana versions don't support topojson.

Selecting a data field containing the official 2-alpha canton codes does not seem to match the ISO 3166-2 code join field for the Switzerland Cantons vector map

In 2017, the four provinces of the Friuli-Venezia Giulia region of Italy were disbanded and their administrative functions were diverted to the municipal and regional governments. As a result the EMS file service currently does not show these four provinces.

However, statistical data still exists and data may still be produced for the former provinces of Friuli-Venezia Giulia. For example, ISO 3166-2 codes and European NUTS codes still exist for the former provinces. So it stands to reason that we should provide the geographical boundaries of these provinces.

The four former provinces we should add boundaries for are Gorizia, Pordenone, Udine, and Trieste. Since we are only adding provinces and not removing any functionality, I believe it is safe to make this change for all versions of EMS.

See also: elastic/kibana#33230

The GeoLite2 IP databases used by the GeoIP Ingest plugin include country subdivision codes (ISO 3166-2) so they can be used with our county subdivision vector layers.

The ISO codes in the GeoIP database are split into two fields: the country code (ex. DE for Germany) and the one, two, or three digit subdivision code (ex. BE for Berlin).

An example response:

...
hits: [{
  "geo": {
     "timezone": "Europe/Berlin",
     "subdivision-name": "Land Berlin",
     "city-name": "Berlin",
     "country-code2": "DE",
     "postal-code": "10715",
     "subdivision-code": "BE",
     "country-name": "Germany",
     "continent-code": "EU",
     "location": {
       "lat": 52.486,
       "lon": 13.3271
     }
  }
  ...
}, ...]
...

Our GeoJSON files for the subdivisions include the ISO code as a single field (ex. DE-BE) as specified in the ISO 3166-2 standard.

{
  "type": "FeatureCollection",
  "features": [{ 
    "type": "Feature",
    "id": "Q64",
    "properties": {
      "iso_3166_2": "DE-BE",
      "label_en": "Berlin",
      "label_de": "Berlin"
    },
   "geometry": {...
    }, 
    ...
  ]
}

Thus the only current way to join IP addresses ingested by the GeoIP plugin to a country subdivision is to create a scripted field in Kibana that joins the country code and subdivision code separated by a dash (-).

Alternatively, we could add fields to the GeoJSON layers that split the full ISO 3166-2 code into country and subdivision codes. However, this is problematic because the subdivision codes listed in the GeoLite2 database are not unique to a single country. For example, the subdivision code 'C' exists in both Spain (ES-C as A Coruña) and in Ireland (IE-C as Connaught). So a user creating a subdivisions region map visualization for website visits from a single country would need to be sure to add a filter for only that country in order to avoid seeing erroneous information.

If we were able to join region maps on multiple fields then this might not be an issue. But that may be out of scope for Kibana.

My preference is to stick to keeping the full ISO code in the GeoJSON files and direct users to create scripted fields to join. Perhaps this would be a good blog post once more country subdivision vector layers are in production.

@elastic/kibana-gis

Link to earlier discussion here: https://github.com/elastic/infra/issues/4577

With the EMS landing page, we are providing a way for users to send feedback on layers.

There is somewhat of an open-ended problem. Over time, we may want to look into a technological solution.

This is odd, because this issue should have been resolved by #18. But it appears that it was not fixed during the refactor. The ISO 3166 code for Germany states has been unusable at least since deployment to GCP buckets because the field name in the data file is incorrect.

Kibana version: 7.3

Elasticsearch version: 7.3

Server OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): download page

Description of the problem including expected versus actual behavior:
In the France Departments map of EMS, city Lyon is not part of any vector shape, it should be part of the Rhône department. Lyon was previously labelled as "Metropolis of Lyon" which was deleted in 7.3

Steps to reproduce:

1. Go to https://maps.elastic.co/#file/france_departments
2. Zoom in on France

There are a couple of examples of these manifest tests that use /files/<filename> URLs. These should be changed to /blob/<id> because v1 and v2 manifests never use the files/<filename> URL.

I've seen that some geojson files contains not minified geojson, like world-countries, australia-states ,finland regions.
Some of them are only partially minified, some others, like the world-countries, are not minified at all.
If you are going to move to vector tiles or topojson in the future it you will have a better compression, but since we are using geojson currently, what about minifiy every geojson we produce before sending it to production?

We only have a single style for the road-data.

Kibana uses CSS transforms on these images to:

• mute the style to create unsaturated/desatured styles
• invert the colors for a dark theme

This way, we generate 3 different basemaps, from that single raster layer.

1. road_map; no desaturatization

2. road_map; desaturation

3. road_map; dark theme

We should deprecate this approach as soon as we can. elastic/kibana#18657.

With the upcoming EMS v3 we will have the ability to publish multiple baselayers, including multiple styles https://github.com/elastic/infra/issues/2907.

Ideally, we have multiple style-sheets (saturated/unsaturated * day/night). These could then be used for eventual client-side vector rendering too. But we also want this generated as actual images, because we will need continued support for raster based rendering as there are currently no plans to rewrite the Visualizations.

cc @alexfrancoeur @nickpeihl @mentat @nyurik

Some of the geoJson data is not compatible with other technologies.

This is likely due to some of the features being malformed; e,g. faulty inner ring orientations, or self-crossing polygons, are possible causes.

There's two known cases where data from EMS is not compatible.

1. The world layer is not compatible with Elasticsearch when ignore_malformed is turned off. This settings is under consideration to be deprecated. I didn't double check, but I think a couple of countries were throwing errors (e.g. Canada, Russia).
2. The French department layer is not compatible with the Mapbox GL JS API (tested on v0.45.0)

There is probably more.

Is there a way we can correct/fix the data, perhaps as part of the automated data pipeline?

We are a mapping startup in Pakistan and our team is working on Elastic Search tools to search and visualize data, currently we are working on Kibana Visualizations for our country, but the lower level (provinces, districts, tehsil) vector layers are not available due to which we are unable to join our polygon data and make a region map visualization.

Since we are a mapping company, we can share these data sets with you, so that it can be made available for all users.

https://en.wikipedia.org/wiki/First-level_NUTS_of_the_European_Union

We'll need to investigate whether we can simply add NUTS codes to existing subdivision layers or if a new layer needs to be created.

Vector layers that use Sophox for data generation should include the Wikidata ID as a property field. This could possibly be used with custom integrations where quantitative data about the regions (ex. population, demographics, etc) could be visualized with data in Elasticsearch.

We should have unit tests that verify that all join fields in the manifest files actually exist in the vector data. This would allow us to catch data errors like in #51 prior to publishing.

Zip code boundaries

Our current US zip code boundaries comes from an older and probably outdated GeoCommons dataset. The US Census provides annual approximations of zip code boundaries in their Zip Code Tabulation Areas (ZCTAs).

The ZCTA shapefile downloadable from the Census is 850MB uncompressed which makes it very hard to simplify for web display.

Simplifying using GDAL

We can use the ogr2ogr command from the GDAL library to convert the shapefile to geojson but even with only one field and simple coordinates the output file is over 1GB.

ogr2ogr -f GeoJSON -select ZCTA5CE10 -lco COORDINATE_PRECISION=6 zcta.geojson /vsizip/tl_2017_us_zcta510.zip

I tried to simplify this to topojson, but the topojson library chokes on this even on a very powerful 2017 MacBook Pro.

npx topojson -q 1e4 -o zcta_topo.json zcta.geojson >> JavaScript head out of memory

Another method I tried was using the -simplify option in ogr2ogr. The simplify argument is a unit of measure based on the spatial reference system of the shapefile. Since the srs for the ZCTAs is WGS84 the unit is a lat/lon measure.

ogr2ogr -f "GeoJSON" -lco COORDINATE_PRECISION=6 -select ZCTA5CE10 -simplify 0.006 zcta.geojson /vsizip/tl_2017_us_zcta510.zip

This creates a much smaller GeoJSON file (30MB) which the TopoJSON can easily handle and we end up with a more managable (but still too large) 13MB topojson file. Additionally, the topology of the dataset is very poor at medium to large scales.

npx topojson -q 1e5 -o zcta_topo.json zcta.geojson

ZCTAs processed by GDAL centered on New York City at zoom level 10

Simplifying using Postgis

1. Create a docker volume to use for persistence
   docker volume create postgresql

2. Run the postgis docker
   docker run --name postgis -p 25432:5432 -it --mount source=postgresql,target=/var/lib/postgresql kartoza/postgis

3. Load the zcta shapefile into postgis
   ogr2ogr -f "PostgreSQL" -progress -select "ZCTA5CE10" -overwrite -lco OVERWRITE=yes -nln zcta -nlt PROMOTE_TO_MULTI -t_srs "EPSG:4326" PG:"dbname='gis' host='localhost' port='25432' user='docker' password='docker'" ~/Downloads/tl_2017_us_zcta510/tl_2017_us_zcta510.shp

4. Sample query with st_simplifypreservetopology (New England). This takes a long time to run for the entire country and we still lose a lot of the topology.
   select st_simplifypreservetopology(wkb_geometry, 0.025) as thegeom, zcta5ce10 from zcta where zcta5ce10 like '0%' OR zcta5ce10 like '1%'

ZCTAs processed by PostGIS centered on New York City at zoom level 10

Simplifying using Mapshaper (Best solution)

The Mapshaper library can output TopoJSON directly from the shapefile without JavaScript memory heap errors. This command creates a ~6MB topojson file that we can use. It also manages to keep topology very well by assuming that very close verticies and edges should be coincident.

npx -p mapshaper mapshaper-xl tl_2017_us_zcta510.shp snap -simplify 0.1% -filter-fields ZCTA5CE10 -rename-fields zip=ZCTA5CE10 -o format=topojson zcta_mapshaper.json

Here are styled comparisons between the Mapshaper TopoJSON and the full resolution shapefile in QGIS at different scales.

ZCTA GIF comparing Census shapefile and Mapshaper TopoJSON centered on New York City at zoom level 5

ZCTA GIF comparing Census shapefile and Mapshaper TopoJSON centered on New York City at zoom level 7

ZCTA GIF comparing Census shapefile and Mapshaper TopoJSON centered on New York City at zoom level 10

Other possibilities

• We can play with different mapshaper simplify parameters and compare file size and how the output looks at different scales
• Other ideas???

We should test the versions parameters for all sources to ensure they are valid semver ranges. Invalid ranges can cause layers to be excluded from the manifest.

From @magnuslarsen (elastic/kibana#34255)

Describe the feature:
A map with danish municipalities.

I've found an already existing (and open source) geoJSON map, however it probably needs some adjustment with the properties:
https://github.com/Neogeografen/dagi/blob/master/geojson/kommuner.geojson

It has two existing properties, though in danish, so let me translate:

• KOMKODE = municipality code (LAU-1 code (prefixed with a 0))
• KOMNAVN = municipality name (in danish)

"kommune" is the danish word for municipality
See:
https://en.wikipedia.org/wiki/List_of_municipalities_of_Denmark
https://en.wikipedia.org/wiki/NUTS_statistical_regions_of_Denmark#Local_administrative_units

Let me know if i can help in any way!

Kibana version: 6.6.2 and above

Elasticsearch version: 6.6.2

Server OS version: macOS Mojave

Browser version: chrome, any

Browser OS version: -

Original install method (e.g. download page, yum, from source, etc.): docker image from docker.elastic.co/kibana/kibana-oss:6.6.2

Steps to reproduce:

1. Create a new index in ElasticSearch
2. Use/ Map the index in Kibana
3. Go to "Visualize", "Create a Visualization" and use "Region Maps"

Describe the feature: Am trying to use the Vector Maps for "New Visualization", there are a number of European countries, but Bulgaria is missing. Would be great if you can add it as provinces and regions. I can provide the geojson files.

It would be nice to have additional metadata about the vector layers we provide. Currently, we only expose the data attribution in the manifest for vector layers.

Raster styles have a formatted markdown string for attribution. This can be removed, since the localized attribution is already listed at the top-level.

e.g.:

• https://tiles.maps.elastic.co/v7.2/manifest : has localized attribution
• https://tiles.maps.elastic.co/styles/osm-bright.json : no need for attribution

I am working on a project where I need to get geo-information in and around switzerland, namely get iso_3166_2 codes of certain areas. I have imported vector layer shapes from https://maps.elastic.co/#file/world_countries for Switzerland, Germany, Austria, Italy and France and it works very well.
Unfortunately there is no map available yet for Liechtenstein (https://en.wikipedia.org/wiki/Liechtenstein). This leads to a blind spot between Switzerland and Austria. Please add a Liechtenstein Layer as soon as possible to close that gap.
Thank you very much in advance.

Elasticsearch version : 7.4.0

Sincerly
Bernhard

With the introduction of feature-state in the Kibana Maps app, we can consider publishing our vector layers as file-based vector tiles. This would allow us to produce much higher resolution datasets than is currently possible with GeoJSON and TopoJSON.

For example, the USA Zip Codes layer currently served as TopoJSON is dramatically simplified to reduce the file size for web map consumption. Compare the following screenshots of zip codes near New York City.

Simplified TopoJSON at zoom 10

Full resolution shapefile at zoom 10

We could theoretically publish zip codes as a directory tree of vector tiles and maintain nearly the same resolution of the shapefile.

Vector Tiles at zoom 10

Full resolution shapefile at zoom 10

GIS files can be converted into vector tiles using the Tippecanoe tool.

1. Download and extract the US Census ZCTA shapefile from ftp://ftp2.census.gov/geo/tiger/TIGER2018/ZCTA5/tl_2018_us_zcta510.zip
2. Convert the shapefile into line delimited GeoJSON using gdal.
   ogr2ogr -f GeoJSONSeq -lco RS=YES -lco ID_FIELD=ZCTA5CE10 -sql "SELECT ZCTA5CE10 as zip FROM tl_2018_us_zcta510" zcta.geojsons ~/Downloads/tl_2018_us_zcta510/tl_2018_us_zcta510.shp
3. Use tippecanoe to generate a directory of vector tiles from the GeoJSON file. This may take a while.
   tippecanoe --read-parallel --detect-shared-borders -S10 -zg -e usa_zip_codes --coalesce-densest-as-needed --no-tile-compression --generate-ids zcta.geojsons

The result vector tiles directory contains nested subdirectories for the {z}, {x}, and {y} tiles. So they can be stored and served using the same cloud storage we use for the other vector files (e.g. https://vector.maps.elastic.co/data/<layer_name>/{z}/{x}/{y}.pbf). And the vector tiles can be styled the same way we style GeoJSON or TopoJSON datasets.

In addition to higher resolution datasets, the vector tiles passed to the client can be much smaller than the GeoJSON and TopoJSON formats. Also, the Maps app in Kibana has native support for vector tiles which means the data is rendered much more quickly.

The Region Maps and Vega Maps in Kibana do not support vector tiles, so we would still maintain support for those visualizations by publishing simplified GeoJSON or TopoJSON datasets.

This is the proposed tiles manifest schema we developed a while back to support additional styles and formats.

The new tiles manifest replaces the top level services property with more appropriate styles.

name is a new property which is an object of language codes to select the appropriate label for the select box in kibana (kibana/#17267).

formats is an array of objects defining the tile service formats (format:raster and format:vector) and the respective url for the style.json (#8). The style.json document contains the url for the service (e.g. /{z}/{x}/{y}.[png,pbf]).

minZoom, maxZoom, and attribution have been removed from the manifest because they are defined in the style.json.

{
  "styles": [
    {
      "id": "road_map",
      "name": {
        "en": "Road Map - Bright"
      },
      "formats": [
        {
          "format": "raster",
          "url": "http://tiles.maps.elstc.co/styles/positron/style.json"
        },
        {
          "format": "vector",
          "url": "http://tiles.maps.elstc.co/styles/positron.json"
        }
      ]
    },
    {
      "id": "dark-matter",
      "name": {
        "en": "Road Map - Dark"
      },
      "formats": [
        {
          "format": "raster",
          "url": "http://tiles.maps.elstc.co/styles/dark-matter/style.json"
        },
        {
          "format": "vector",
          "url": "http://tiles.maps.elstc.co/styles/dark-matter.json"
        }
      ]
    },
    ...
  ]
}

The USA States have the two-digit postal code (ex. AK), but should also have the ISO 3166-2 region code (ex. US-AK) similar to how we do with other countries.

Saved objects in Kibana reference the URL by human readable name, not the layer id. This needs to be fixed in all existing layers.

For example.

--- manifest_cur.json   2018-12-03 13:31:55.000000000 -0800
+++ manifest_fix.json   2018-12-03 13:32:48.000000000 -0800
@@ -250,7 +250,7 @@
           }
         }
       ],
-      "legacy_ids": ["world_countries"],
+      "legacy_ids": ["World Countries"],
       "layer_name": {
         "aeb-arab": "البلاد",
         "af": "land",

Kibana 7.1.1 struggled to render data for canton name Zurich on a region map displaying Swiss Cantons, using the Switzerland Cantons vector map and name (en) as join field.

South Sardinia got the ISO code so I added it, but the old Sicily provinces are still listed under the Region so the I had to tweak the previous query to return all the 107 boundaries

     #defaultView:MapRegions
    SELECT ?id ?iso_3166_2 ?istat ?label_en ?label_it
    WHERE {
      SERVICE <https://query.wikidata.org/sparql> {
        {
          SELECT 
            ?id
            (SAMPLE(?id2) AS ?id2) 
            (SAMPLE(?iso_3166_2) AS ?iso_3166_2) 
            (SAMPLE(?istat) AS ?istat) 
            (SAMPLE(REPLACE(?label_en, "Province of ", "")) as ?label_en)
            (SAMPLE(REPLACE(?label_it, "provincia di ", "")) as ?label_it)
            WHERE { {
            VALUES (?entity) {
              (wd:Q38)
            }
            ?entity (wdt:P150/wdt:P150) ?id.
            ?id wdt:P300 ?iso_3166_2.
            MINUS {
              ?id (wdt:P576|wdt:P582) ?end.
              FILTER(?id NOT IN(
              wd:Q16112,
              wd:Q16150,
              wd:Q16154,
              wd:Q16166,
              wd:Q16208,
              wd:Q16227,
              wd:Q16251,
              wd:Q16277,
              wd:Q16288,
              wd:Q16184,
              wd:Q16247,
              wd:Q16293,
              wd:Q16294
              ))
            }
            OPTIONAL { ?id wdt:P635 ?istat. }
            OPTIONAL {
              ?id rdfs:label ?label_en.
              FILTER((LANG(?label_en)) = "en")
            }
            OPTIONAL {
              ?id rdfs:label ?label_it.
              FILTER((LANG(?label_it)) = "it")
            }
          } UNION {
            # ISO Code IT-AO refers to a province that no longer exists. The GeoLite2 IP database lists cities from the Aosta Valley
            # regiohn so we'll use the region boundaries for the IT-AO code.
            VALUES ?entity { wd:Q38 }
            ?entity wdt:P150 ?id .
            FILTER ( ?id = wd:Q1222 )
            VALUES ?iso_3166_2 { "IT-AO" }
            OPTIONAL { ?id wdt:P635 ?istat. }
            OPTIONAL {
              ?id rdfs:label ?label_en.
              FILTER((LANG(?label_en)) = "en")
            }
            OPTIONAL {
              ?id rdfs:label ?label_it.
              FILTER((LANG(?label_it)) = "it")
            }
          } UNION {
            # Sicily comunes
            ?id wdt:P31/wdt:P279*  wd:Q21190155.
            ?id wdt:P300 ?iso_3166_2.
            
            OPTIONAL { ?id wdt:P635 ?istat. }
            OPTIONAL {
              ?id rdfs:label ?label_en.
              FILTER((LANG(?label_en)) = "en")
            }
            OPTIONAL {
              ?id rdfs:label ?label_it.
              FILTER((LANG(?label_it)) = "it")
            }
          } UNION {
            # Sicily metropolitan cities
            VALUES (?id) {
              (wd:Q20795016)
              (wd:Q20991250)
              (wd:Q20991246)
            }
            ?id wdt:P300 ?iso_3166_2.
            
            OPTIONAL { ?id wdt:P635 ?istat. }
            OPTIONAL {
              ?id rdfs:label ?label_en.
              FILTER((LANG(?label_en)) = "en")
            }
            OPTIONAL {
              ?id rdfs:label ?label_it.
              FILTER((LANG(?label_it)) = "it")
            }
          }
          }
          GROUP BY ?id
        }
      }
    }

Context

EMS published vector data. This data comes from various sources, including Wikidata, Naturalearth.

See https://vector.maps.elastic.co/v2/manifest for full list of current offering.

The current files were created ad-hoc. There is not much in the way of code/process/docs to regenerate these files

• Germany/Canada: https://github.com/elastic/infra/issues/3475
• World countries: https://github.com/elastic/infra/issues/2737
• USA zip codes: https://github.com/elastic/infra/issues/3476
• United states: https://github.com/elastic/infra/issues/2017#issuecomment-297153050

There is a big risk that these files will grow stale. We need a way to keep these fresh, and improve continuously when issues pop up.

Goal

• We need to ability to generate these files, from source, and have their schema comply with our current files.

• Files need to be geojson, except the zip-file which is topojosn. Future files can be one or the other. Geojson preferred whenever reasonable. (is more human-readable/hackable). Topojson for larger files.

• Each file needs to be accompanied with a separate meta-data file (includes info about attribution, fields (human readable label/discription), ...). This information is important because it is required by Kibana to boostrap the UI. We will likely also use this info in the EMS-landing page.

• Some files may require corrections/additions. E.g. misspellings in the world file/missing countries. Ideally this can be rolled up in this "generation" process.

• We need backwards compatibility. Addition of new properties and features is fine, but we cannot take away properties or features compared to previous versions of the tile.

Integration with EMS (??)

How to integrate with EMS is not entirely clear yet. e.g. How will these new files be "picked up" by EMS. Right now, we have an upload form. I'd suggest we work with that for now, until we have a clearer grasp of where we want to go, technology wise.

Example

Suppose we have the countries file, what we need is a program that spits out:

1. geojson featurecollection

"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"name": "Antigua and Barbuda",
"iso2": "AG",
"iso3": "ATG"
},
"geometry": {
"type": "MultiPolygon",
"coordinates": [
[
[
[
-61.686668,
17.024441000000152
],
[
-61.887222,
17.105274
],
[
-61.794449,
17.1633300000001
],
[
-61.686668,
17.024441000000152
]
]
],
[
[
[
-61.72917199999989,
17.608608
],
[
-61.853058,
17.583054000000104
],
[
-61.873062,
17.703888
],
[
-61.72917199999989,
17.608608
]
]
]
]
}
},
{
"type": "Feature",
"properties": {
"name": "Algeria",
"iso2": "DZ",
"iso3": "DZA"
},
"geometry": {
"type": "MultiPolygon",
"coordinates": [
[
[
[
2.96361,
36.802216
],
[
4.785832,
36.894722
],
[
5.328055,
36.640274
],
[
6.398333,
37.086388
],
[
......

2. Metadata json object (separate file)

{
"attribution": "[Made with NaturalEarth](http://www.naturalearthdata.com/about/terms-of-use) | [Elastic Maps Service](https://www.elastic.co/elastic-maps-service) | WHATEVER_SOURCES",
"name": "World Countries",
"format": "geojson",
"fields": [
{
"name": "iso2",
"description": "Two letter abbreviation"
},
{
"name": "name",
"description": "Country name"
},
{
"name": "iso3",
"description": "Three letter abbreviation"
}
],
"created_at": "WHATEVER_DATE_OF_GENERATION",
"tags": []
}

We should have a layer of the regions of Italy along with the existing provinces layer. According to several sources [1] [2], regions are more commonly used than provinces in Italy. One region (Friuli-Venezia Giulia) has even disbanded its provinces.

The Minsk region of Belarus has a variety of small enclaves and exclaves. These shapes can look jagged and odd due to simplification process we use to generalize feature geometries for data visualization in Elastic Maps Service. However, this does not appear to be an easy fix and would require manual editing.

Kibana visualizations created prior to v6.6 use name to uniquely reference the layer in the v1 and v2 manifests. The name property was dropped in v6.6 manifest and replaced with layer_id. So Kibana needs a way to map the name property in saved objects to the correct layer_id in the v6.6 manifest. To resolve this, we need to add a new property called legacy_ids to the v6.6 manifest.

The legacy_ids will contain an array of strings. It is expected that all v1 and v2 layers will have exactly one item in the array. However, we may need to add items to this array in future manifest versions.

If you try to open the page on IE11, it shows just a blank screen.
The following error is in the console:

HTML1300: Navigation occurred.
maps.elastic.co
DOM7011: The code on this page disabled back and forward caching. For more information, see: http://go.microsoft.com/fwlink/?LinkID=291337
maps.elastic.co
Unhandled promise rejection ReferenceError: 'fetch' is undefined
   "Unhandled promise rejection"
   {
      [functions]: ,
      __proto__: { },
      description: "'fetch' is undefined",
      message: "'fetch' is undefined",
      name: "ReferenceError",
      number: -2146823279,
      stack: "ReferenceError: 'fetch' is undefined
   at Anonymous function (https://maps.elastic.co/main.bundle.js:114:530583)
   at w (https://maps.elastic.co/main.bundle.js:26:1220479)
   at Anonymous function (https://maps.elastic.co/main.bundle.js:26:1220288)
   at e[t] (https://maps.elastic.co/main.bundle.js:26:1220673)
   at r (https://maps.elastic.co/main.bundle.js:114:530085)
   at Anonymous function (https://maps.elastic.co/main.bundle.js:114:530058)
   at M (https://maps.elastic.co/main.bundle.js:26:1199119)
   at Anonymous function (https://maps.elastic.co/main.bundle.js:114:530025)
   at Anonymous function (https://maps.elastic.co/main.bundle.js:114:530801)
   at Anonymous function (https://maps.elastic.co/main.bundle.js:114:531019)",
      Symbol()_w.yjoxcvwozfd: undefined,
      Symbol()_x.yjoxcvwozfd: undefined,
      Symbol(react.async_mode)_u.yjoxcvwozsd: undefined,
      Symbol(react.call)_n.yjoxcvwozsd: undefined,
      Symbol(react.context)_t.yjoxcvwozsd: undefined,
      Symbol(react.element)_m.yjoxcvwozsd: undefined,
      Symbol(react.forward_ref)_v.yjoxcvwozsd: undefined,
      Symbol(react.fragment)_q.yjoxcvwozsd: undefined,
      Symbol(react.portal)_p.yjoxcvwozsd: undefined,
      Symbol(react.provider)_s.yjoxcvwozsd: undefined,
      Symbol(react.return)_o.yjoxcvwozsd: undefined,
      Symbol(react.strict_mode)_r.yjoxcvwozsd: undefined
   }

All good on Edge and the other browsers.

cc @thomasneirynck

Ref: https://twitter.com/huangshuming/status/982059129865879552

Chile has three Political - Administrative layers of territory subdivision
16 regions that hold 56 Provinces and those provinces hold 346 communes.

Here is a geojson with the polygons

https://drive.google.com/file/d/1C4unrWIxhMwJ1HofFZJfVNUXgzMMXwOS/view?usp=sharing

I hope this helps.

the original source is available online in the website of real state ministry

http://www.ide.cl/descarga/capas/item/division-politica-administrativa-2018.html

For the regions, here are the iso codes:

https://es.wikipedia.org/wiki/ISO_3166-2:CL

We should own the vector tile generation from OSM data ourselves iso. rely on Klokan's tiles.

This will give us more flexibility with licensing for on-prem deployments too, as well as the ability to do more sophisticated processing (e.g. of coinciding boundaries).

cc @nickpeihl @mentat @nyurik

Related to elastic/kibana#23407

Localization in EMS will require changes to the EMS manifests and Kibana.

Region maps in Kibana will need to access the locale setting in kibana.yml and use that to determine which vector manifest to choose from the catalogue manifest (modifying this function).

Catalogue manifest schema

Here's a first draft of how I think the catalogue manifest would appear for the vector layers. We would generate different vector manifests for each locale in the i18n dictionary and fall back to the top level manifest if necessary.. The list of locales should match the locales in the kbn-i18n package . Note, I haven't changed anything in the tiles_v2 element, but that could change, too, depending on how we handle basemap internationalization.

{
  "services": [
    {
      "id": "tiles_v2",
      "name": "Elastic Maps Tile Service",
      "manifest": "https://tiles.maps.elastic.co/v2/manifest",
      "type": "tms"
    },
    {
      "id": "geo_layers",
      "name": "Elastic Maps Vector Service",
      "manifest": "https://vector.maps.elastic.co/v6.6/en-US/manifest",
      "type": "file",
      "i18n": { 
        "af": {
          "manifest": "https://vector.maps.elastic.co/v6.6/af/manifest"
        },
        ...,
        "es": {
          "manifest": "https://vector.maps.elastic.co/v6.6/es/manifest"
        },
        ...,
      }
    }
  ]
}

Vector manifests schema

Here is an example of how I assume the Spanish locale vector manifest would appear (https://vector.maps.elastic.co/v6.6/es/manifest). The top level name property in each layer must remain the same to avoid breaking visualizations but a description property is added which could be used as the label in the select box instead of the name property. This would be similar to how the description property in the fields is used.

{
  "layers": [
    {
      "attribution": "[Made with NaturalEarth](http://www.naturalearthdata.com/about/terms-of-use)|[Elastic Maps Service](https://www.elastic.co/elastic-maps-service)",
      "weight": 1,
      "name": "World Countries",
      "description": "País",
      "url": "https://vector.maps.elastic.co/files/world_countries_v1.geo.json?elastic_tile_service_tos=agree",
      "format": "geojson",
      "fields": [
        {
          "type": "id",
          "name": "iso2",
          "description": "ISO 3166-1 alpha-2"
        },
        {
          "type": "id",
          "name": "iso3",
          "description": "ISO 3166-1 alpha-3"
        },
        {
          "type": "property",
          "name": "name",
          "description": "Nombre"
        },
      ],
      "created_at": "2017-04-26T17:12:15.978370",
      "tags": [],
      "id": 5659313586569216
    },
   ...,
    {
      "attribution": "© [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors|[Elastic Maps Service](https://www.elastic.co/elastic-maps-service)",
      "weight": 0,
      "name": "Provincia de Canadá",
      "url": "https://vector.maps.elastic.co/files/canada_provinces_v1.geo.json?elastic_tile_service_tos=agree",
      "format": "geojson",
      "fields": [
        {
          "type": "id",
          "name": "iso_3166_2",
          "description": "ISO-3166-2"
        },
        {
          "type": "property",
          "name": "label_en",
          "description": "Nombre (inglés)"
        },
        {
          "type": "property",
          "name": "label_fr",
          "description": "Nombre (francés)"
        }
      ],
      "created_at": "2018-01-23T13:07:00.346990",
      "tags": [],
      "id": 5715999101812736
    },
    ...
  ]
}

I'm sure there are unanswered questions and probably other things I haven't considered. So please let me know your thoughts @elastic/kibana-gis.

Currently, attribution is pipe delimited.

Future versions should just model this as an array.

{
"layers": [
{
"attribution": "[Made with NaturalEarth](http://www.naturalearthdata.com/about/terms-of-use) | [Elastic Maps Service](https://www.elastic.co/elastic-maps-service)",
"name": "World Countries",

Will require corresponding change in Kibana

Kibana 7.1.1 struggled to render data for code 69 on a region map displaying France, using the France Departments vector map and INSEE department code as join field.

Code 69 represents: Département du Rhône (69) − COG | Insee

It seems we are limiting 8.0.0 users to 10 zoom levels in master

https://tiles.maps.elastic.co/v2/default/13/2414/3082.png?elastic_tile_service_tos=agree&my_app_name=kibana&my_app_version=8.0.0&license=894371dc-9t49-4997-93cb-8o2e3r7fa6a8

you are only allowed up to zoom level 10

cc: @nyurik @nickpeihl @thomasneirynck

We need to be able to map source data fields to fields used in Elastic Map Service.