Comments (5)
this is likely related to #7
from ems-file-service.
Thanks @mimitsu. It took a lot of research to figure out why this is happening.
The GeoJSON polgyons we create for the Elastic Maps Service conform to the right-hand rule (rhr) for winding order as outlined in the RFC7946 GeoJSON standard. However, d3 and thusly Vega assume that polygons are drawn in the opposite order and are incompatible with RFC7946 standard. (More info at these links 1, 2, 3).
As you might expect, there is an existing library (rfc7946-to-d3
) to reverse the GeoJSON winding order to work with d3.
The Elasticsearch geo_shape
datatype is agnostic about winding order. I've tested this by ingesting the same polygon in right-hand rule and reversed winding order and running a successful GeoShape query on both.
Option 1
Since the EMS vector files are primarily used with Kibana, I'm inclined to drop the conformity to the right-hand rule standard on the GeoJSON files in EMS and switch the winding order to match what d3 expects.
Here's a table showing compatibility for Kibana and Elasticsearch.
Usage | rfc7946 winding | d3 winding |
---|---|---|
Kibana Vega | No | Yes |
Kibana region map | Yes | Yes |
Kibana GIS plugin | Yes | Yes |
Elasticsearch geo_shape datatype | Yes | Yes |
Elasticsearch GeoShape query | Yes | Yes |
Option 2
Another option we could look at is modifying the Vega logic in Kibana to detect rfc7946 winding polygons and using the aforementioned rfc7946-to-d3
library to mutate the polygon winding to work with Vega/d3. One advantage to this is that users would not need to worry about winding order in their own polygons in Elasticsearch to work with Vega.
@nyurik @thomasneirynck please share your thoughts.
from ems-file-service.
If we do not have any geojson usages beyond d3/vega, it might be ok to forgo the RFC. It does make me cringe to not abide by the standards, but in this case it might make sense because noone else uses it anyway. For the future Kibana versions, I think we should simply use topojson internally, and convert it to geojson on the fly when requested by Vega.
from ems-file-service.
This is a bug in d3.
@nyurik We do have geojson usage beyond d3/vega. In the regions-maps and in the upcoming GIS app.
Serving all EMS files as topojson is a long term improvement. But EMS should continue to server this file as geojson, as people use the landing page to download these files. Geojson is a more convenient and adopted format for interoperability than Topojson is right now.
I don't think it is a good idea to purposefully break this file according to the spec, just so it can work with Vega in the short term. Vega-visualizations in Kibana are also still an experimental feature and situations like this is exactly the reason why it was flagged for this at the get-go.
Much of the concern is based on principle (we're suggesting to introduce a bug in our data so we an accommodate a bug in d3). It could relieve some short-term pain for Vega users. But it will also introduce long term issues.
- Could we identify this floating square, where does it come from? What happens if we just remove it?
- Does the file with the faulty-order work in the GIS-app? We should most definitely not make this change if the file does not render correctly in MapboxGL (ie. the GIS-app).
from ems-file-service.
- Could we identify this floating square, where does it come from? What happens if we just remove it?
I'm not sure how we could automatically identify and remove the floating square in the Vega plugin. It seems that it could take the form of any of the features.
This Vega code loads a d3 compatible (GeoJSON spec invalid) USA States file. I created this file by running the rfc7946-to-d3
command against the states file in this PR. In this case you won't see the floating square.
Note, you'll have to set vega.enableExternalUrls: true
in your Kibana config to test it.
{
"$schema": "https://vega.github.io/schema/vega/v3.json",
"config": {"kibana": {"type": "map", "latitude": 40, "longitude": -97, "zoom": 4}},
"data": [
{
"name": "usastates",
"url": "https://gist.githubusercontent.com/nickpeihl/5e02cd0b2e717a99ae7f025e9f41f3cc/raw/29dc03ad20a83d6928a26102e75b0b2ccec5b126/usa_states_v1.d3.json",
"format": {"type": "json", "property": "features"}
}
],
"marks": [
{
"type": "shape",
"from": {"data": "usastates"},
"transform": [{"type": "geoshape", "projection": "projection"}]
}
]
}
- Does the file with the faulty-order work in the GIS-app? We should most definitely not make this change if the file does not render correctly in MapboxGL (ie. the GIS-app).
I've also tested this d3 compliant file in the GIS Plugin and found that it displayed correctly. So mapbox-gl doesn't appear to care about the winding order.
- Add the mapping to ES
curl -X PUT "localhost:9200/usa_states_d3" -H 'Content-Type: application/json' -d'
{
"mappings": {
"doc": {
"properties": {
"geometry": {
"type": "geo_shape",
"tree": "quadtree",
"precision": "1.0m",
"distance_error_pct": 0.025
}
}
}
}
}
'
- Parse and upload the features into ES
curl https://gist.githubusercontent.com/nickpeihl/5e02cd0b2e717a99ae7f025e9f41f3cc/raw/29dc03ad20a83d6928a26102e75b0b2ccec5b126/usa_states_v1.d3.json | \
jq -c '.features[] | { index: { _index: "usa_states_d3", _type: "doc" } }, .' | \
curl -H "Content-Type: application/x-ndjson" -XPOST --data-binary @- http://localhost:9200/_bulk | \
jq '.items[].index.result'
from ems-file-service.
Related Issues (20)
- [Meta] EMS 8.0 HOT 1
- France Departments winding order incompatible with Vega HOT 5
- Meta: rename master to main
- Upgrade to Node 16
- Upgrade all GeoJSON layers to follow RFC7946
- Remove region identifiers with values that are too common HOT 1
- Telangana and Andhra Pradesh regions names in India are incorrect
- Fix region name for GB-LND in administrative regions layer
- New Zealand EMS administrative map seems to be missing. Can this please be added? HOT 1
- "<svg onload=alert(1337)>%00"><img src=x onerror=alert`1`//"
- Fix Morocco regions label
- Upgrade to NodeJS 18
- [EMS] Include commit identifier in release artifacts
- Generate EMS File Service manifests for serverless
- [EMS] Create a minimal document for root directory
- Error Loading US Zip Codes GeoJSON file to index HOT 4
- Update ZCTAs to 2020 data
- Fix the content-type for the index.html file in the vector catalog
- Washington DC borders as County vs State are different HOT 1
- Dependency Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ems-file-service.