Giter VIP home page Giter VIP logo

h3ron's Introduction

h3ron

ci dependency status

Rust libraries for the H3 geospatial indexing system.

Maintenance status

In january 2023 the h3o library - a port of H3 to rust - has been released. This brings many benefits including type safety, compilation to WASM and performance improvements (example: issue comparing raster to h3 conversion).

As a result these libraries will most certainly only receive dependency upgrades in the future. Some parts will maybe be ported to h3o, like the raster to H3 conversion already was.

Crates

This repository consists of multiple crates:

High-level rust API for H3.

This crate builds on-top the official C library. The recently released h3o project provides an implementation of H3 in pure rust.

Documentation | Changelog

bindgen-generated bindings for the statically linked libh3 C library.

Documentation | Changelog

Integration with the ndarray crate to generate H3 cells from raster data (using gdal, ...)

Documentation | Changelog

Graph algorithms on edges of the H3 spatial indexing system.

Documentation | Changelog

Integration of the h3 geospatial indexing system with polars dataframes by providing extension traits to UInt64Chunked and DataFrame.

Python bindings

Python bindings for parts of the functionalities are available in the h3ronpy extension now located in an own repository. For an overview of some features complementary to libh3 please see the README of the python bindings.

Why this name?

Well, coming up with a good name for a project while avoiding naming conflicts is hard. On the other hand are animal-based names always pretty easy to remember.

How to pronounce it? I got no idea - probably like the heron bird family.

License

MIT

Some data in the data directory is derived from OpenStreetMap and as such is copyright by the OpenStreetMap contributors. For the OSM license see OSMs Copyright and License page.

h3ron's People

Contributors

fwfichtner avatar jaykickliter avatar manevillef avatar nmandery avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

h3ron's Issues

Add documentation for experimental `local_to_ij` and `ij_to_local`

I've been trying to undestand the results of the methods:

  • h3_to_local_ij
  • local_ij_to_h3

and I don't get it.

In the test example you take an Index i (0x89283080ddbffff) and get a neighbor N through k_ring(1) and the result is CoordIJ { i: 1111, j: 620 }.
I think it has something to do with the resolution of i (9) but I don't understand how to use this result.

I would expect CoordIJ between two neighbours to be between (-1, -1) and (1, 1).

I can't find documentation on this on Uber H3 lib so if you understand this result can you explain it or add documentation on the methods?

A faster h3IsValid

First and foremost: thanks for this great crate ๐Ÿ™
Been using for a few days, and the high level API is really enjoyable ๐Ÿ‘

I've been working with H3 these last few months, and one thing leading to another I've found that the current implementation of h3IsValid seems suboptimal (contains for-loops that could be avoided).
Given that I have to validate set of cells that can be quite large in some case, I gave it a shot at optimizing it.

After a few attempts, I arrrived to a solution that seems to be isofunctional while being x3 to x4 faster than the original implementation:

Code is not necessarily more complex (I would even say it's simpler but I'm not objective here ๐Ÿ˜€), though there are one or two tricks that do deserve detailed explanations.

The two biggests improvements are:

  • replacing the loop checking for invalid digits using an adaptation of an old nul-byte detection algorithm (technology from the past come to save the future ๐Ÿ˜›)
  • replacing the loop checking for invalid pentagons

You can find the code here (with some unit tests).
The benchmarks I used is here.

I've also run a check again a few billions of value (~30% valid/70% invalid) with my implementation against the official ones to ensure that the results were identical.
Interestingly enough, I can observe the same x3 speedup in this case (on my machine ~8s vs ~24s)

Dunno if it's of interest to you, but I'm sharing it just in case ๐Ÿ™‚

Would probably be better to implement this upstream, but I don't have the time nor the motivation to start a pull request in C ๐Ÿ˜… (that being said, they do have a PR in progress about this, and I did left a comment over there as well ๐Ÿ™‚).

fix conversion to GeoDataFrame for empty dataframes

....
  File "/home/nico/.cache/pypoetry/virtualenvs/h3get-fb-population-mj_UiQ-H-py3.8/lib/python3.8/site-packages/h3ronpy/util.py", line 34, in h3index_column_to_geodataframe
    geometry=[Polygon(h3.h3_to_geo_boundary(h, geo_json=True)) for h in np.nditer(df[column_name].to_numpy())],
ValueError: Iteration of zero-sized operands is not enabled

Build problem due to geo dependency

The latest crate version of h3ron is v0.17.0, which uses v^0.23 of the geo crate (https://crates.io/crates/h3ron/0.17.0/dependencies). However, this version of geo seems to be somehow defective (see rust-lang/rust#110776) and therefore, the build process on my machine fails with:

error[E0275]: overflow evaluating the requirement
error: could not compile `geo` (lib) due to previous error

When installing h3ron pointing to the github source in Cargo.toml, the problem vanishes. So it looks like the crate dependencies need to be updated to a higher geo version.

Add a `CHANGELOG` for each releases

It would be nice to have a list of changes (especially breaking ones) available.

Also Github provides the release system that can use this file.

If you also add a CONTRIBUTING, contributors could write their PR changes under a Unreleased section

Open to inline comments?

I have been going trough the h3ron code and added some comments to get a better understanding.

What are your thoughts on inline code comments? Should I submit a pull request with my additions or do you prefer to keep it simple?

Are there particular style preferences regarding comments?

Vanya :)

h3-sys WASM support

It would be neat to add WASM support which I imagine can be done adding some build task in h3-sys that uses emscripten to generate the lib object that is later statically linked.

error: building the downstream libh3 C library using the Solana SDK via cmake

Hi,
solana has this helloworld program that you can build (and deploy on a validator): https://github.com/solana-labs/example-helloworld

It works well before adding the h3ron library.

adding this dependency:

h3ron = "0.13.0"

and running:

 npm run build:program-rust

would generate this error

Error: Function _ZN15crossbeam_epoch8internal6Global7collect17hdc7ca3f69bd0b12bE Stack offset of -4144 exceeded max offset of -4096 by 48 bytes, please minimize large stack variables
*** stack smashing detected ***: terminated
error: could not compile `crossbeam-epoch`

Caused by:
  process didn't exit successfully: `rustc --crate-name crossbeam_epoch --edition=2018 /home/pc/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.9.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=1bb560ac1e04b0d7 -C extra-filename=-1bb560ac1e04b0d7 --out-dir /home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps --target bpfel-unknown-unknown -L dependency=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps -L dependency=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/release/deps --extern cfg_if=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps/libcfg_if-1643e4a2c6efb510.rmeta --extern crossbeam_utils=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps/libcrossbeam_utils-c2f18398aedce3c4.rmeta --extern lazy_static=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps/liblazy_static-327a97971696967c.rmeta --extern memoffset=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps/libmemoffset-1762d3769b1ca8cc.rmeta --extern scopeguard=/home/pc/Documents/funspace/crypto-world/lab/solanaworkspace/example-helloworld/src/program-rust/target/bpfel-unknown-unknown/release/deps/libscopeguard-83203493741a9def.rmeta --cap-lints allow -C lto=no` (signal: 6, SIGABRT: process abort signal)
warning: build failed, waiting for other jobs to finish...
error: build failed

Could you please help?

Implement `is_neighbor_to` for indexes

Hello again,

h3ron-h3-sys binding provides the h3IndexesAreNeighbors which is not wrapped by h3ron.

It could be nice to have Index::is_neighbor_to(&Index) , and also a check on Vec<Index> to ensure all indexes are both:

  • the same resolution
  • consecutive neighbors

Rewrite C dependency in Rust?

Less of an issue, more questions:

Are there any plans to get rid of the C dependency? How much work do you think it would be? Cmake dep is a huge bummer It would also be nice to use const rather than all the generated tables.

Additionally, as you have the most relevant experience to answer this... the h3 api is very C-shaped with a lot of freestanding functions...if you were to write an idiomatic rust lib from the start, would the API be similar to h3ron or would you do things differently?

Add wrappers for missing `h3ron-sys` bindings

A few methods are not available in the h3ron crate but are available in the C bindings h3ron-sys.

  • h3Line
  • h3LineSize
  • getBaseCell
    etc

I can try to make a PR with the implementation if this is too much work

Rust 1.53+ warns about undefined behavior in bindgen-generated code

This is tracked in this upstream bindgen issue: rust-lang/rust-bindgen#1651

Example:

warning: dereferencing a null pointer
  --> /home/.../h3ron/target/debug/build/h3ron-h3-sys-b89be661850ddf33/out/bindings.rs:31:19
   |
31 |         unsafe { &(*(::std::ptr::null::<GeoCoord>())).lat as *const _ as usize },
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this code causes undefined behavior when executed
   |
   = note: `#[warn(deref_nullptr)]` on by default

docs.rs build is currently broken due to cmake version

The cmake version available in the docs.rs build-env is too old, installing a newer version would require a distro upgrade to ubuntu 22.04. From the build logs:

[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr]   CMake Error at CMakeLists.txt:15 (cmake_minimum_required):
[INFO] [stderr]     CMake 3.20 or higher is required.  You are running version 3.16.3
[INFO] [stderr] 

Related upstream issue: rust-lang/crates-build-env#84

Bumping h3ron-sys

Hi,

H3 recently merged the following PR: uber/h3#749

I've updated h3o with the new values, but since I've some test running against H3 through h3ron-sys, they are now failing because I'm comparing against an old version.

Would it be possible to bump the commit targeted by h3ron-sys and release a new version?

Handling error codes

Hello,

According to H3 RFC 4 any H3 Methods can return some error code which would be good to implement to improve the error system.
I wrote a draft like this:

/// Errors as defines in [RFC] 4.0.0
///
/// [RFC]: https://github.com/uber/h3/blob/master/dev-docs/RFCs/v4.0.0/error-handling-rfc.md
#[derive(Debug, Copy, Clone, PartialOrd, PartialEq)]
pub(crate) enum InternalErrorCode {
    /// The operation failed but a more specific error is not available
    Failed, // 1
    /// Argument was outside of acceptable range (when a more specific error code is not available)
    Domain, // 2
    /// Latitude or longitude arguments were outside of acceptable range
    LatLonDomain, // 3
    /// Resolution argument was outside of acceptable range
    ResDomain, // 4
    /// H3Index cell argument was not valid
    CellInvalid, // 5
    /// H3Index directed edge argument was not valid
    DirectionalEdgeInvalid, // 6
    /// H3Index undirected edge argument was not valid
    UniDirectionalEdgeInvalid, // 7
    /// H3Index vertex argument was not valid
    VertexInvalid, // 8
    /// Pentagon distortion was encountered which the algorithm could not handle it
    Pentagon, // 9
    /// Duplicate input was encountered in the arguments and the algorithm could not handle it
    DuplicateInput, // 10
    /// H3Index cell arguments were not neighbors
    NotNeighbors, // 11
    /// H3Index cell arguments had incompatible resolutions
    ResMismatch, // 12
    /// Necessary memory allocation failed
    Memory, // 13
    /// Unknown error code
    UnknownError(i32),
}

impl InternalErrorCode {
    /// Checks if the H3 return value is an error and returns the associated error code
    pub fn get_error(value: i32) -> Option<Self> {
        match value {
            0 => None,
            1 => Some(InternalErrorCode::Failed),
            2 => Some(InternalErrorCode::Domain),
            3 => Some(InternalErrorCode::LatLonDomain),
            4 => Some(InternalErrorCode::ResDomain),
            5 => Some(InternalErrorCode::CellInvalid),
            6 => Some(InternalErrorCode::DirectionalEdgeInvalid),
            7 => Some(InternalErrorCode::UniDirectionalEdgeInvalid),
            8 => Some(InternalErrorCode::VertexInvalid),
            9 => Some(InternalErrorCode::Pentagon),
            10 => Some(InternalErrorCode::DuplicateInput),
            11 => Some(InternalErrorCode::NotNeighbors),
            12 => Some(InternalErrorCode::ResMismatch),
            13 => Some(InternalErrorCode::Memory),
            v => Some(InternalErrorCode::UnknownError(v)),
        }
    }

    /// Checks if the H3 return value is an error
    pub fn is_error(value: i32) -> bool {
        value != 0
    }
}

But in many cases the error codes don't work as the RFC intended.
For example: the is_neighbor_to method according to documentation should return E_SUCCESS = 0 in case of success, but instead it returns 1 as h3ron correctly checks.

In different documentation I found out that it should indeed return 1 on success and 0 on failure, as opposed to the RFC v4.

So I don't know if we should try to implement this partially or completely forget about it.

Use heap allocations for longedges to reduce memory usage of prepared graphs

https://github.com/nnethercote/perf-book

use std::collections::HashSet;

struct LongEdge {
    in_edge: u64,
    out_edge: u64,
    edge_path: Vec<u64>,
    cell_lookup: HashSet<u64>,
}

struct OwnedEdgeValueBoxed {
    weight: f32,
    longedge: Option<Box<(LongEdge, f32)>>,
}

struct OwnedEdgeValue {
    weight: f32,
    longedge: Option<(LongEdge, f32)>,
}

fn main() {
    dbg!(std::mem::size_of::<OwnedEdgeValue>());
    dbg!(std::mem::size_of::<OwnedEdgeValueBoxed>());
}

Output:

    Finished dev [unoptimized + debuginfo] target(s) in 0.58s
     Running `target/debug/playground`
[src/main.rs:22] std::mem::size_of::<OwnedEdgeValue>() = 104
[src/main.rs:23] std::mem::size_of::<OwnedEdgeValueBoxed>() = 16

FYI: HexSet

This is not an issue. I just want to share a crate that uses h3ron and may be of interest to you and your users: HexSet.

TLDR: we wanted a way to check if points lay in large H3 polyfills which is faster than our current linear search. Hit testing went from the order of 50 uS down to one or two digit nS on my sludgy 2013 Mac Pro.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.