USA States buggy about ems-file-service HOT 5 CLOSED

this is likely related to #7

from ems-file-service.

Thanks @mimitsu. It took a lot of research to figure out why this is happening.

The GeoJSON polgyons we create for the Elastic Maps Service conform to the right-hand rule (rhr) for winding order as outlined in the RFC7946 GeoJSON standard. However, d3 and thusly Vega assume that polygons are drawn in the opposite order and are incompatible with RFC7946 standard. (More info at these links 1, 2, 3).

As you might expect, there is an existing library (rfc7946-to-d3) to reverse the GeoJSON winding order to work with d3.

The Elasticsearch geo_shape datatype is agnostic about winding order. I've tested this by ingesting the same polygon in right-hand rule and reversed winding order and running a successful GeoShape query on both.

Option 1

Since the EMS vector files are primarily used with Kibana, I'm inclined to drop the conformity to the right-hand rule standard on the GeoJSON files in EMS and switch the winding order to match what d3 expects.

Here's a table showing compatibility for Kibana and Elasticsearch.

Option 2

Another option we could look at is modifying the Vega logic in Kibana to detect rfc7946 winding polygons and using the aforementioned rfc7946-to-d3 library to mutate the polygon winding to work with Vega/d3. One advantage to this is that users would not need to worry about winding order in their own polygons in Elasticsearch to work with Vega.

@nyurik @thomasneirynck please share your thoughts.

from ems-file-service.

If we do not have any geojson usages beyond d3/vega, it might be ok to forgo the RFC. It does make me cringe to not abide by the standards, but in this case it might make sense because noone else uses it anyway. For the future Kibana versions, I think we should simply use topojson internally, and convert it to geojson on the fly when requested by Vega.

from ems-file-service.

This is a bug in d3.

@nyurik We do have geojson usage beyond d3/vega. In the regions-maps and in the upcoming GIS app.

Serving all EMS files as topojson is a long term improvement. But EMS should continue to server this file as geojson, as people use the landing page to download these files. Geojson is a more convenient and adopted format for interoperability than Topojson is right now.

I don't think it is a good idea to purposefully break this file according to the spec, just so it can work with Vega in the short term. Vega-visualizations in Kibana are also still an experimental feature and situations like this is exactly the reason why it was flagged for this at the get-go.

Much of the concern is based on principle (we're suggesting to introduce a bug in our data so we an accommodate a bug in d3). It could relieve some short-term pain for Vega users. But it will also introduce long term issues.

• Could we identify this floating square, where does it come from? What happens if we just remove it?
• Does the file with the faulty-order work in the GIS-app? We should most definitely not make this change if the file does not render correctly in MapboxGL (ie. the GIS-app).

from ems-file-service.

• Could we identify this floating square, where does it come from? What happens if we just remove it?

I'm not sure how we could automatically identify and remove the floating square in the Vega plugin. It seems that it could take the form of any of the features.

This Vega code loads a d3 compatible (GeoJSON spec invalid) USA States file. I created this file by running the rfc7946-to-d3 command against the states file in this PR. In this case you won't see the floating square.

Note, you'll have to set vega.enableExternalUrls: true in your Kibana config to test it.

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "config": {"kibana": {"type": "map", "latitude": 40, "longitude": -97, "zoom": 4}},
  "data": [
    {
      "name": "usastates",
      "url": "https://gist.githubusercontent.com/nickpeihl/5e02cd0b2e717a99ae7f025e9f41f3cc/raw/29dc03ad20a83d6928a26102e75b0b2ccec5b126/usa_states_v1.d3.json",
      "format": {"type": "json", "property": "features"}
    }
  ],
  "marks": [
    {
      "type": "shape",
      "from": {"data": "usastates"},
      "transform": [{"type": "geoshape", "projection": "projection"}]
    }
  ]
}

• Does the file with the faulty-order work in the GIS-app? We should most definitely not make this change if the file does not render correctly in MapboxGL (ie. the GIS-app).

I've also tested this d3 compliant file in the GIS Plugin and found that it displayed correctly. So mapbox-gl doesn't appear to care about the winding order.

1. Add the mapping to ES

curl -X PUT "localhost:9200/usa_states_d3" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "doc": {
      "properties": {
        "geometry": {
          "type": "geo_shape",
          "tree": "quadtree",
          "precision": "1.0m",
          "distance_error_pct": 0.025
        }
      }
    }
  }
}
'

2. Parse and upload the features into ES

curl https://gist.githubusercontent.com/nickpeihl/5e02cd0b2e717a99ae7f025e9f41f3cc/raw/29dc03ad20a83d6928a26102e75b0b2ccec5b126/usa_states_v1.d3.json | \
jq -c '.features[] | { index: { _index: "usa_states_d3", _type: "doc" } }, .' | \
curl -H "Content-Type: application/x-ndjson" -XPOST --data-binary @- http://localhost:9200/_bulk | \
 jq '.items[].index.result'

from ems-file-service.