How to access link of Data in repo

cuspatial is referenced as a trajectory throughout these docs. Update to reflect the latest naming scheme.

distance_and_speed has a weird help doc that is based on an example from an earlier function. It should stand alone.

Is your feature request related to a problem? Please describe.
Installing and maintaining cuspatial depends on a well documented build system.

Describe the solution you'd like
Let's add cuspatial to the CI system so that we know exactly how to build it and can always verify correctness of the latest version.

Describe the bug
There is currently a lot of redundancy in the conda meta.yamls for libcuspatial and cuspatial. The dependencies should be optimize/minimized to make our conda packages as small as possible.

What is your question?
I see you feed in numpy array, but since a shapely object or arcgis object is the most common geometry object, would it be better to feed in the object type into a column and convert it on the fly?

Describe the bug
The SOA form of timestamps isn't immediately compatible with cudf. SOA timestamps are a specific 64 bit binary format with regions for y,m,d,hh,mm,ss,wd,yd,ms, and pid. These regions all need to be computed into a single datetime value to be compatible with cudf.

Is your feature request related to a problem? Please describe.
When computing distance and speed, the speed is scaled on the assumption of datetime64[ms]. We need to validate that the calculations are still correct when using any of the many other resolutions of datetime64 like [ns] and [s].

Describe the solution you'd like
Write distance and speed tests using other timestamp speeds at the user level. Once that's done, if speed results are not uniform, add an argument to trajectory_distance_and_speed.cu to communicate the timestamp resolution from python.

If a value in the polygon_ids is greater than the number of polygons based on the vertex offsets, there's an illegal memory access. Perhaps we should either change the name from polygon_ids to something that makes this clear, or allow arbitrary IDs.

import cudf
import cuspatial
​
result = cuspatial.point_in_polygon_bitmap(
            cudf.Series([0, -8, 6.0]), # x coordinates of 3 query points
            cudf.Series([0, -8, 6.0]), # y coordinates of 3 query points
            cudf.Series([2,3]), # unique id of two polygons
            cudf.Series([5, 10]), # position of last vertex in each polygon
            # polygon coordinates, x and y
            cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0]),
            cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0]),
        )
cudf.Series(result)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-65afad57b1e2> in <module>
      9             # polygon coordinates, x and y
     10             cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0]),
---> 11             cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0]),
     12         )
     13 cudf.Series(result)

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuspatial/core/gis.py in point_in_polygon_bitmap(x_points, y_points, polygon_ids, polygon_end_indices, polygons_x, polygons_y)
    132         polygon_end_indices,
    133         polygons_x,
--> 134         polygons_y,
    135     )
    136 

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

RuntimeError: CUDA error encountered at: /jenkins/workspace/gpuCI/cuspatial/branches/cuspatial-cpu-branch-build/cpp/src/spatial/point_in_polygon.cu:94: 77 cudaErrorIllegalAddress an illegal memory access was encountered

Succeeds:

import cudf
import cuspatial
​
result = cuspatial.point_in_polygon_bitmap(
            cudf.Series([0, -8, 6.0]), # x coordinates of 3 query points
            cudf.Series([0, -8, 6.0]), # y coordinates of 3 query points
            cudf.Series([1,2,3,4]), # unique id of two polygons
            cudf.Series([5, 10,15,20]), # position of last vertex in each polygon
            # polygon coordinates, x and y
            cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0]),
            cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0]),
        )
cudf.Series(result)
0    3
1    1
2    2
dtype: int32

Fails:

import cudf
import cuspatial
​
result = cuspatial.point_in_polygon_bitmap(
            cudf.Series([0, -8, 6.0]), # x coordinates of 3 query points
            cudf.Series([0, -8, 6.0]), # y coordinates of 3 query points
            cudf.Series([1,2,3,5]), # unique id of two polygons
            cudf.Series([5, 10,15,20]), # position of last vertex in each polygon
            # polygon coordinates, x and y
            cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0]),
            cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0]),
        )
cudf.Series(result)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-97e451ef14a0> in <module>
      9             # polygon coordinates, x and y
     10             cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0]),
---> 11             cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0]),
     12         )
     13 cudf.Series(result)

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuspatial/core/gis.py in point_in_polygon_bitmap(x_points, y_points, polygon_ids, polygon_end_indices, polygons_x, polygons_y)
    132         polygon_end_indices,
    133         polygons_x,
--> 134         polygons_y,
    135     )
    136 

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

RuntimeError: CUDA error encountered at: /jenkins/workspace/gpuCI/cuspatial/branches/cuspatial-cpu-branch-build/cpp/src/spatial/point_in_polygon.cu:94: 77 cudaErrorIllegalAddress an illegal memory access was encountered

Describe the bug
A close enough review of the point-in-polygon example code could reveal this, but it is not at all obvious. Polygons specified as input to this function need to be closed - their starting vertex must be the same as the ending vertex.

Clarify this in the docs.

[Edit: this issue was moved from cuDF]

I wish cudf would provide support for GIS type functions. I am not sure if GIS type functions belong in cudf, or if they belong in their own separate library, but it makes sense that at the very least they should be very compatible with the rest of the functionality provided by cudf.

I would like to see functions like:

• point-in-polygon
• nearest points from a constant radius
• perimeter
• area
• convex hull
• bounding box
• and more

I think these sort of functions/features would address the needs of GIS data analytics and open up even more uses for cudf and rapidsai.

It returns two same-length Serieses. Lets put them in a DF.

The second ask from @MurrayData. Perhaps you could be more specific?

Describe the bug
Derive returns results that do not include any timestamp relevant information. Is it actually used in the computation, or can it be removed?

Steps/Code to reproduce bug

        import cudf
        import cuspatial
        result = cuspatial.derive(
            cudf.Series([0, 1, 2, 3]),
            cudf.Series([0, 0, 1, 1])
            cudf.Series([0, 0, 1, 1])
            cudf.Series([0, 10, 0, 10])
        )
        print(result)
        (2,    trajectory_id  length  position
         0              0       2         2
         1              1       2         4)

Describe the bug
Checking to see where certain points lie in 2 polygons, results in the following error:
ValueError: Length of new column names: 2 does not match length of previous column names: {l_old_cols}

Steps/Code to reproduce bug

import dask_cudf
df = dask_cudf.read_csv("cuspatial_error_sample_data.csv").set_index('Unnamed: 0')
df = df.compute()
df[['LATITUDE', 'LONGITUDE']].head()

	LATITUDE	LONGITUDE
Unnamed: 0		
0	70.9333333	-8.6666667
1	70.9333333	-8.6666667
2	70.9333333	-8.6666667
3	70.9333333	-8.6666667
4	70.9333333	-8.6666667

import cuspatial

poly_x  = cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0])
poly_y =  cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0])

query_ids = cudf.Series()
query_ids = query_ids.append(1)
poly_ids = query_ids.append(2)

query_offsets = cudf.Series()
query_offsets = query_offsets.append(5)
poly_offsets = query_offsets.append(10)

query_x = cudf.Series([0, -8.4, 6.0, 7.])
query_y = cudf.Series([0, -8, 6.0, 100.])
bitmaps = cuspatial.point_in_polygon_bitmap(
     df['LONGITUDE'], df['LATITUDE']
    poly_ids, poly_offsets,
    poly_x, poly_y
)

Error StackTrace:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-25-bc1d531045e2> in <module>
     17     df['LATITUDE'], df['LONGITUDE'],
     18     poly_ids, poly_offsets,
---> 19     poly_x, poly_y
     20 )

/conda/envs/rapids/lib/python3.6/site-packages/cuspatial-0.10.0a0+142.gcbe0513-py3.6-linux-x86_64.egg/cuspatial/core/gis.py in point_in_polygon_bitmap(x_points, y_points, polygon_ids, polygon_end_indices, polygons_x, polygons_y)
    141     )._apply_support_method("astype", dtype="bool")
    142     result_bools.columns = [
--> 143         f"in_polygon_{x}" for x in list(reversed(polygon_ids))
    144     ]
    145     result_bools = result_bools[list(reversed(result_bools.columns))]

/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in columns(self, columns)
   1002             if hasattr(self, "multi_cols"):
   1003                 delattr(self, "multi_cols")
-> 1004             self._rename_columns(columns)
   1005 
   1006     @property

/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in _rename_columns(self, new_names)
   1023                 "match length of previous column names: {l_old_cols}"
   1024             )
-> 1025             raise ValueError(msg)
   1026 
   1027         mapper = dict(zip(old_cols, new_names))

ValueError: Length of new column names: 2 does not match length of previous column names: {l_old_cols}

Expected behavior
Callingpoint_in_polygon_bitmap works when the test latitude and longitude are created with cudf.Series() but fails when read from the file and called in the above manner.
When the following is passed to point_in_polygon_bitmap , it works without generating an error:

query_x = cudf.Series([0, -8.4, 6.0, 7.])
query_y = cudf.Series([0, -8, 6.0, 100.])

Environment details:

• Environment location: Docker rapidsai-nightly
• Method of cuSpatial install: from source

Additional context
cudf version: '0.10.0a+1081.g485a511'

Describe the bug

unit tests should always assert that the results of the API they test are correct. This is not the case of the following:

• coordinate_transform_toy.cu
• trajectory_derive_toy.cu
• trajectory_distance_and_speed_toy.cu
• trajectory_spatial_bounds_toy.cu
• trajectory_subset_toy.cu

Expected behavior
Compute expected results and compare using GTEST / GMOCK assertions / expectations.

Note we should also remove the "_toy" suffix on these tests and rename them to test_<api name>.cpp (or .cu as necessary).

There's also no reason to only have one test (TEST_F or TYPED_TEST) per API. There should be plenty of tests, exercising different situations, such as mal-formed inputs, inputs with/without nulls, empty inputs, empty results, etc.

See RAPIDS cuDF cpp/tests for lots of good examples.

Surfaced this with @thomcom . There may be something going on with the array offsetting. Example tweaked from the pytest.

EDIT: This appears to be non-deterministic or perhaps undefined indexing behavior?

import numpy as np
import pytest
​
import cudf
from cudf.utils import cudautils
from cudf.tests.utils import assert_eq
​
import cuspatial
​
result = cuspatial.point_in_polygon_bitmap(
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([1,2,3]), # three polygons
        cudf.Series([5, 10, 13]),
        cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0,   4, 8, 2]),
        cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0,   4, 8, 6]),
    )
​
print(cudf.Series(result))
​
result = cuspatial.point_in_polygon_bitmap(
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([1,2,3]), # three polygons
        cudf.Series([5, 10, 14]),
        cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0,   4, 8, 2]),
        cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0,   4, 8, 6]),
    )
​
print(cudf.Series(result))
​
​
result = cuspatial.point_in_polygon_bitmap(
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([0, -8, 6.0, 3]),
        cudf.Series([1,2,3]), # three polygons
        cudf.Series([5, 10, 15]),
        cudf.Series([-10.0, 5, 5, -10, -10, 0, 10, 10, 0, 0,   4, 8, 2]),
        cudf.Series([-10.0, -10, 5, 5, -10, 0, 0, 10, 10, 0,   4, 8, 6]),
    )
​
print(cudf.Series(result))
​
0    3
1    1
2    2
3    3
dtype: int32
0    7
1    1
2    2
3    7
dtype: int32
0    3
1    5
2    2
3    3
dtype: int32

Attempting to build cuSpatial with cuDF branch-0.11 fails with the following:

-- Build files have been written to: /rapids/cuspatial/cpp/build
Scanning dependencies of target cuspatial
[  6%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/shp/polygon_shapefile_reader.cu.o
[ 12%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/soa/polygon_soa_reader.cu.o
[ 18%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/soa/point_soa_reader.cu.o
[ 25%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/soa/uint32_soa_reader.cu.o
[ 31%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/soa/timestamp_soa_reader.cu.o
[ 37%] Building CUDA object CMakeFiles/cuspatial.dir/src/query/spatial_window_points.cu.o
/rapids/cuspatial/cpp/src/query/spatial_window_points.cu:18:10: fatal error: utilities/cuda_utils.hpp: No such file or directory
 #include <utilities/cuda_utils.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/cuspatial.dir/build.make:127: recipe for target 'CMakeFiles/cuspatial.dir/src/query/spatial_window_points.cu.o' failed
make[2]: *** [CMakeFiles/cuspatial.dir/src/query/spatial_window_points.cu.o] Error 1
CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/cuspatial.dir/all' failed
make[1]: *** [CMakeFiles/cuspatial.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

It looks like the recent move of utilities to the legacy subfolder is causing it.

Feature description
While calling point_in_polygon_bitmap with more than 32 polygons, it results in the following error

RuntimeError: cuDF failure at: /rapids/cuspatial/cpp/src/spatial/point_in_polygon.cu:134: #polygon of polygons can not exceed bitmap capacity (32 for unsigned int)

Describe the solution you'd like
Automatic batching for the polygons would be useful to avoid multiple calls to the function and user-defined batching.

Describe alternatives you've considered
Manually batching the polygons and making multiple calls and append the dataframes.

Additional context
StackTrace of the error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-15-b2c41b336501> in <module>
     17     df['LATITUDE'], df['LONGITUDE'],
     18     ids, offsets,
---> 19     lat, lng
     20 )
     21 # bitmaps = cuspatial.point_in_polygon_bitmap(

/conda/envs/rapids/lib/python3.6/site-packages/cuspatial-0.10.0a0+144.g5e188f1-py3.6-linux-x86_64.egg/cuspatial/core/gis.py in point_in_polygon_bitmap(x_points, y_points, polygon_ids, polygon_end_indices, polygons_x, polygons_y)
    133         polygon_end_indices,
    134         polygons_x,
--> 135         polygons_y,
    136     )
    137 

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

cuspatial/_lib/spatial.pyx in cuspatial._lib.spatial.cpp_point_in_polygon_bitmap()

RuntimeError: cuDF failure at: /rapids/cuspatial/cpp/src/spatial/point_in_polygon.cu:134: #polygon of polygons can not exceed bitmap capacity (32 for unsigned int)

Temporary gdf_columns* allocated are never freed in the named files.

Following the build instructions in README.md I get the following error snippet:

(rapids) root@1bdf09244fab:/rapids/cuspatial/python/cuspatial# python setup.py build_ext --inplace           
Compiling cuspatial/bindings/query.pyx because it changed.                                                       
Compiling cuspatial/bindings/soa_readers.pyx because it changed.                                                 
Compiling cuspatial/bindings/spatial.pyx because it changed.                                                     
Compiling cuspatial/bindings/trajectory.pyx because it changed.                             
[1/4] Cythonizing cuspatial/bindings/query.pyx                                                        
                                                                                                  
Error compiling Cython file:
------------------------------------------------------------
...
# cython: profile=False
# distutils: language = c++
# cython: embedsignature = True
# cython: language_level = 3

from cudf.bindings.cudf_cpp cimport *
^
------------------------------------------------------------

cuspatial/bindings/query.pxd:8:0: 'cudf/bindings/cudf_cpp.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
from cudf.bindings.cudf_cpp cimport *
from libcpp.pair cimport pair                                                                                                                  
cdef extern from "query.hpp" namespace "cuspatial" nogil:                                                                                     

    cdef pair[gdf_column, gdf_column] spatial_window_points(
                       ^
------------------------------------------------------------

cuspatial/bindings/query.pxd:13:24: unknown type in template argument

Omitted:
Many repeated failured about cuspatial/bindings/query.pxd:14:14: 'gdf_scalar' is not a type identifier and cuspatial/bindings/query.pxd:18:14: 'gdf_column' is not a type identifier from different .pxd files.

Describe the bug

cuspatial::subset_trajectory_id can be rewritten to use a cudf::gather(). This will enable it to automatically support null input data elements.

Likewise, cuspatial::spatial_window_points can probably use cudf::copy_if with a custom functor. Currently it uses thrust::copy_if, which can't support nulls.

The values returned by derive in the position column actually denote the end_position of the derived trajectories. Update the nomenclature.

• Is there any requirement for “OGC validity” of polygons? Issues such as self-intersections, “back tracking”, repeating points, etc. can be problematic at times. https://www.opengeospatial.org/standards/sfa
• Are there any min/max latitude ranges? Sometimes things go bad if approaching either north or south pole.
• Are there any special considerations for polygons or lines that cross the +/- 180 degree longitude line? For example, hurricane tracks/windfields near Fiji and New Zealand or the Aleutian Islands of Alaska (Attu), or the coastline of Russia.
• Winding order of exterior/interior rings?
Thanks

For ease of use, it would be useful if point in polygon queries could return an N x K DataFrame of Boolean values instead of a binary bitmap, where each point gets a row (N) and each polygon a column (K). This would make it easier to do things like filtering to only rows that fall within a specific polygon.

What is your question?

Is it possible for someone to create a simple notebook example on how to run a point in polygon test with cuSpatial?

In looking at the demos folder for several hours, I'm not seeing how to get from this:

import shapely,shapefile
from shapely.geometry import Polygon
plyreader = shapefile.Reader('/content/drive/My Drive/osm_stuff/osm_us_landuse.shp')
polygon = plyreader.shapes()
plys = []
for shape in polygon:
    plys.append(Polygon(shape.points))

To the code where I pass something into the cuspatial.point_in_polygon_bitmap function.

For background, I'm coming from geopandas and a heavy pandas background. The convenstions and code I'm seeing in cuSpatial are different and I'd like to get some help on understanding how to use the library. It seems powerful from the description, but I'm looking for an example that is straightforward. Thanks in advance for the help.

If possible, can someone's example use a real shapefile like the demo examples? However, the demo scripts in cuSpatial read in the file but I never saw a use of testing for point in polygon.

So, to be clear, I'm asking if someone can make a simple example of loading a shapefile, loading points, and testing if points are inside the polygons that come from the shapefile.

The last binding that needs to be updated is in query.pyx.

lonlat_to_xy_km_coordinates doesn't define its transformation exactly. It is just the haversine equation? We should document this.

Similarly, we may want to enable the definition of different coordinate systems when using this function. @zhangjianting what do you think?

Vincenty Distance is an iterative formula to calculate the shortest distance between two points on a spheroid. It is more accurate than simple great circle calculations, such as Haversine, which uses an average radius for the Earth.

Vincenty takes into the account the flattening of the Earth as an oblate spheroid to calcualte to an accuracy of 0.5 mm on the WGS84 ellipsoid.

While differences between Haversine and Vincenty are small over short distances, noticeable errors arise over longer distances, particularly if calculating aircraft trajectories.

I suggest adding Vincenty Distance to cuSpatial.

This is my implementation in CUDA JIT

@cuda.jit(device=True)
def vincenty_distance(lon_1, lat_1, lon_2, lat_2,a,f):
    lon_1 = lon_1 * math.pi / 180.0 
    lon_2 = lon_2 * math.pi / 180.0 
    lat_1 = lat_1 * math.pi / 180.0 
    lat_2 = lat_2 * math.pi / 180.0
    b = (1 - f) * a                  # semi-minor axis
    u_1 = math.atan((1 - f) * math.tan(lat_1))
    u_2 = math.atan((1 - f) * math.tan(lat_2))
    L = lon_2 - lon_1
    Lambda = L                       # set initial value of lambda to L
    Lambda_2 = L + 1.0
    sin_u1 = math.sin(u_1)
    cos_u1 = math.cos(u_1)
    sin_u2 = math.sin(u_2)
    cos_u2 = math.cos(u_2)
    iters = 0
    while iters == 0 or (iters < 200 and abs(Lambda_2 - Lambda) > 1E-13):
        iters += 1
        cos_lambda = math.cos(Lambda)
        sin_lambda = math.sin(Lambda)
        sin_sigma = math.sqrt((cos_u2 * sin_lambda)**2 + (cos_u1 * sin_u2 - sin_u1 * cos_u2 * cos_lambda)**2)
        cos_sigma = sin_u1 * sin_u2 + cos_u1 * cos_u2 * cos_lambda
        sigma = math.atan2(sin_sigma, cos_sigma)
        sin_alpha = (cos_u1 * cos_u2 * sin_lambda) / sin_sigma
        cos_sq_alpha = 1 - sin_alpha**2
        cos2_sigma_m = cos_sigma - ((2 * sin_u1* sin_u2) / cos_sq_alpha)
        C = (f / 16) * cos_sq_alpha * (4 + f * (4 - 3 * cos_sq_alpha))
        Lambda_2 = Lambda
        Lambda = L + (1 - C) * f * sin_alpha * (sigma + C * sin_sigma * (cos2_sigma_m + C * cos_sigma * (-1 + 2 * cos2_sigma_m**2)))
    u_sq = cos_sq_alpha * ((a**2 - b**2) / b**2)
    A = 1 + (u_sq / 16384.0)*(4096.0 + u_sq * (-768.0 + u_sq * (320.0 - 175.0 * u_sq)))
    B = (u_sq / 1024.0) * (256.0 + u_sq * (-128.0 + u_sq * (74.0 - 47.0 * u_sq)))
    delta_sig = B * sin_sigma * (cos2_sigma_m + 0.25 * B * (cos_sigma * (-1 + 2 * cos2_sigma_m**2) - (1.0 / 6.0) * B * cos2_sigma_m * (-3.0 + 4.0 * sin_sigma**2) * (-3.0 + 4.0 * cos2_sigma_m**2)))
    return b * A * (sigma - delta_sig)                 # output distance in meters

where a = semi-major axis in metres and f = flattening

For the WGS84 ellipsoid, these values are:

a = 6378137.0
f = 0.0033528106647474805

I have tested the above against online converters provided by mapping agencies.

The sample below shows the difference between Vincenty (v_dist) against Haversine (h_dist) in km on the NY Taxi dataset:

Describe the bug
This function returns a Series, but should return an nxn Dataframe, where n is the len(sizes) of the third argument. The orientation and method for building the DataFrame from Series should be carefully built, as it can be inefficient.

Is your feature request related to a problem? Please describe.
It should be possible to use cuspatial as a standalone library. One of the easiest ways of doing this is to create a conda environment file that makes installing all of the dependencies easy.

Describe the solution you'd like
Let's build a conda environment for cuspatial.

Describe the bug

import cuspatial

results in

ModuleNotFoundError: No module named 'cuspatial._lib.spatial'

Environment details

• We are running on AWS on a g3s.xlarge.

• We used this environment.yml to build the container for JupyterHub.

SOA has bindings for the trajectory demo test, but may need one or two more that were not used by that test. Double check them.

@MurrayData's top ask from us in terms of feature requests. This might be a whale, or it might be a pretty straightforward file read and coordinate transform. @MurrayData, where should I start?

Is your feature request related to a problem? Please describe.
cuspatial doesn't have any named bindings - it is used by knowing the exact package structure and importing as appropriate. Future users will want to be able to reflect on the structure of each package from their interpreters.

Describe the solution you'd like
Define __init__.py with important public members of the project exposed for users.

I am trying to run the hausdorff_clustering_test_toy.py demo code on my G3 AWS instance.
The code hasn't been modified in any way. I am running in within the cudf_dev environment that I built using this command:

  conda env create --name cudf_dev --file conda/environments/cudf_dev_cuda10.0.yml

I installed cuspatial from source following your instructions.

The error is as follows:

Traceback (most recent call last):
File "hausdorff_clustering_test_toy.py", line 33, in
distance = cuspatial.directed_hausdorff_distance(pnt_x, pnt_y, cnt)
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/cuspatial-0.11.0a0+37.g51b2557-py3.7-linux-x86_64.egg/cuspatial/core/gis.py", line 65, in directed_hausdorff_distance
result = cpp_directed_hausdorff_distance(x, y, count)
File "cuspatial/_lib/spatial.pyx", line 120, in cuspatial._lib.spatial.cpp_directed_hausdorff_distance
File "cuspatial/_lib/spatial.pyx", line 121, in cuspatial._lib.spatial.cpp_directed_hausdorff_distance
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/cudf-0.11.0a0+1810.g4f60dae.dirty-py3.7-linux-x86_64.egg/cudf/core/series.py", line 1448, in astype
raise e
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/cudf-0.11.0a0+1810.g4f60dae.dirty-py3.7-linux-x86_64.egg/cudf/core/series.py", line 1444, in astype
data=self._column.astype(dtype, **kwargs)
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/cudf-0.11.0a0+1810.g4f60dae.dirty-py3.7-linux-x86_64.egg/cudf/core/column/column.py", line 954, in astype
return self.as_numerical_column(dtype, **kwargs)
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/cudf-0.11.0a0+1810.g4f60dae.dirty-py3.7-linux-x86_64.egg/cudf/core/column/numerical.py", line 172, in as_numerical_column
data=libcudf.typecast.cast(self, dtype).data, dtype=np.dtype(dtype)
File "cudf/_lib/typecast.pyx", line 39, in cudf._lib.typecast.cast
RuntimeError: CUDA error encountered at: /home/ubuntu/GeoSpatial/cudf/cpp/src/unary/legacy/unary_ops.cuh:66: 8 cudaErrorInvalidDeviceFunction invalid device function
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/numba/utils.py", line 754, in _exitfunc
f()
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/numba/utils.py", line 678, in call
return info.func(*info.args, **(info.kwargs or {}))
File "/home/ubuntu/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/rmm/rmm.py", line 219, in finalizer
librmm.rmm_free(handle, stream)
File "rmm/_lib/lib.pyx", line 239, in rmm._lib.lib.rmm_free
File "rmm/_lib/lib.pyx", line 228, in rmm._lib.lib.c_free
File "rmm/_lib/lib.pyx", line 48, in rmm._lib.lib.check_error
rmm.rmm.RMMError: RMM_ERROR_CUDA_ERROR

In build, let's add conda install -c rapidsai-nightly cuspatial so that interested parties don't only see that they have to build form source.

Is your feature request related to a problem? Please describe.
The easiest way to use cuspatial will be to add it to conda.

Describe the solution you'd like
Create conda recipes so that anyone can easily install and use cuspatial.

derive_trajectories, trajectory_distance_and_speed, and trajectory_spatial_bounds need to have updated python bindings written ASAP.

Is your feature request related to a problem? Please describe.
cuspatial tests currently depend on a variety of external data to run.

Describe the solution you'd like
In order to improve code stability and increase the usefulness of CI, let's write unit tests.

What formats should we start with @MurrayData?

Following the build instructions in README.md, I get the following error snippet:

-- CUDF: CUDF_HOME set to
-- CUDF: CONDA_PREFIX set to /conda/envs/rapids
-- CUDF: CUDF_LIBRARY set to /conda/envs/rapids/lib/libcudf.so
-- CUDF: CUDF_INCLUDE set to /conda/envs/rapids/include
-- CUDF SRC: CUDF_SRC_INCLUDE set to /cpp/src
-- CUDF: CUB_INCLUDE set to /cpp/thirdparty/cub
-- CUDF: JITIFY_INCLUDE set to /cpp/thirdparty/jitify
-- CUDF: CUDF_TEST_INCLUDE set to /cpp/
-- NVSTRINGS: NVSTRINGS_INCLUDE set to /conda/envs/rapids/include
-- NVSTRINGS: NVSTRINGS_LIBRARY set to /conda/envs/rapids/lib/libNVStrings.so
-- NVSTRINGS: NVCATEGORY_LIBRARY set to /conda/envs/rapids/lib/libNVCategory.so
-- NVSTRINGS: NVTEXT_LIBRARY set to /conda/envs/rapids/lib/libNVText.so
-- Using Nvidia Tools Extension
-- Configuring done
-- Generating done
-- Build files have been written to: /rapids/cuspatial/cpp/build
Scanning dependencies of target cuspatial
[  6%] Building CUDA object CMakeFiles/cuspatial.dir/src/io/soa/polygon_soa_reader.cu.o
/rapids/cuspatial/cpp/src/io/soa/polygon_soa_reader.cu:25:10: fatal error: utilities/error_utils.hpp: No such file or directory
 #include <utilities/error_utils.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/cuspatial.dir/build.make:62: recipe for target 'CMakeFiles/cuspatial.dir/src/io/soa/polygon_soa_reader.cu.o' failed
make[2]: *** [CMakeFiles/cuspatial.dir/src/io/soa/polygon_soa_reader.cu.o] Error 1
CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/cuspatial.dir/all' failed
make[1]: *** [CMakeFiles/cuspatial.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Is your feature request related to a problem? Please describe.
Currently cuSpatial supports reading polygon data from disk files in SoA (Structure of Arrays) format. A standalone tool under "data" was developed to convert ESRI shapefiles to SoA files. Instead of following the two-step approach, an integrated approach that reads ESRI shapefiles from disks to gdf columns is desirable and has been requested by several customers.

Describe the solution you'd like
Refine data/poly2soa.cpp code to make it a c++ API and subsequently develop a Python wrapper API for it.

Describe the bug
import cuSpatial does not successfully import and gives librmm_cffi error.

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuspatial/utils/gis_utils.py in <module>
      4 from numba import cuda
      5 
----> 6 from librmm_cffi import librmm as rmm
      7 
      8 

ModuleNotFoundError: No module named 'librmm_cffi'

Steps/Code to reproduce bug
import cuspatial

Expected behavior
cuspatail imports successfuly imports with no errros, and I can run cuspatial examples

Environment details (please complete the following information):

• Environment location: Docker - rapids-dev-nightly: latest
• Method of cuSpatial install: Docker

Additional context
This was working with a docker pull from 8 days ago

I'm trying to follow this example.

import cuspatial._lib.soa_readers as readers
import cuspatial._lib.spatial as gis

fpos, rpos, plyx, plyy = readers.cpp_read_ply_soa(fn)

Result:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-8c68f35633c5> in <module>
      3 
      4 
----> 5 fpos, rpos, plyx, plyy = readers.cpp_read_ply_soa(fn)

AttributeError: module 'cuspatial._lib.soa_readers' has no attribute 'cpp_read_ply_soa'

Is your feature request related to a problem? Please describe.
cuspatial appears to only support gdf_column at this time.

Describe the solution you'd like
Enable cudf Dataframes and Series in cuspatial methods.

Is your feature request related to a problem? Please describe.
Use cudf::table (and DataFrame in Python) type for inputs and outputs where it makes sense.

Describe the solution you'd like

Multiple cuSpatial APIs take and/or return multiple associated columns of the same length. For API simplicity and consistency these should be converted to use cudf::table and DataFrame. This is convenient because cudf::table provides trivial initializer list constructors that make it easy to convert from columns to tables, and accessors and iterators that make it trivial to get columns from a table.

I think it will make the API more compact and more consistent with cuDF.

For example this API:

std::pair<gdf_column,gdf_column> lonlat_to_coord(const gdf_scalar& cam_lon,
                                                 const gdf_scalar& cam_lat,
                                                 const gdf_column& in_lon,
                                                 const gdf_column& in_lat);

would become:

cudf::table lonlat_to_coord(const std::pair<gdf_scalar, gdf_scalar>& camera_longitude_latitude,
                                               const cudf::table& in_longitude_latitude);

Describe the bug
When reading in shapefiles I am consistently getting an error when trying to read using cuspatial.read_polygon_shapefile

Steps/Code to reproduce bug
I have attached a very basic shapefile with only a few geometries included, load this using cuspatial.read_polygon_shapefile

Example shapefile that produces error:

demo.zip

Expected behavior
Shapefile is read to GPU memory

Environment details (please complete the following information):

• Docker on QNAP
• Method of cuSpatial install: conda

Additional context
Error Text:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-b23fe52750b5> in <module>
      1 #read the shapefile into the GPU memory
----> 2 gpu_polygons = cuspatial.read_polygon_shapefile(poi_shp)
      3 

/opt/conda/envs/rapids/lib/python3.6/site-packages/cuspatial/io/shapefile.py in read_polygon_shapefile(filename)
     10     result = cpp_read_polygon_shapefile(filename)
     11     return (
---> 12         DataFrame({"f_pos": result[0], "r_pos": result[1]}),
     13         DataFrame({"x": result[2], "y": result[3]}),
     14     )

/opt/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in __init__(self, data, index, columns, dtype)
    183 
    184             for i, (col_name, series) in enumerate(data):
--> 185                 self.insert(i, col_name, series, forceindex=index is not None)
    186 
    187         self._add_empty_columns(columns, index)

/opt/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in insert(self, loc, column, value, forceindex)
   1557             )
   1558         self._cols[column] = self._prepare_series_for_add(
-> 1559             value, forceindex=forceindex, name=column
   1560         )
   1561         keys = list(self._cols.keys())

/opt/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in _prepare_series_for_add(self, col, forceindex, name)
   1518         series = Series(col, name=name) if not SCALAR else col
   1519         self._sanitize_columns(series)
-> 1520         series = self._sanitize_values(series, SCALAR)
   1521 
   1522         empty_index = len(self._index) == 0

/opt/conda/envs/rapids/lib/python3.6/site-packages/cudf/core/dataframe.py in _sanitize_values(self, series, SCALAR)
   1496                 return Series(arr)
   1497         elif len(self) > 0 and len(sind) != len(index):
-> 1498             raise ValueError("Length of values does not match index length")
   1499         return series
   1500 

ValueError: Length of values does not match index length

The existing bindings return Column objects, which tests then convert to Series to introspect. All bindings should here and henceforth only return cudf Series and Dataframe objects.

All of our data is stored inside a data warehouse as text format WKT strings. This is very efficient for us as it means that we can directly process spatial data on both clusters and items like BigQuery, likewise it gives us agility in moving between different file formats (ORC, Parquet, CSV etc)

It would be great if we could directly read WKT strings into cuSpatial without having to convert to shapefiles as our data changes so often and the workflow to convert is cumbersome