Giter VIP home page Giter VIP logo

Comments (5)

jlhumber avatar jlhumber commented on July 21, 2024 1

@CF-FHB-X version 0.2.0 has been pushed to pypi, the DATETIME type has been added to the enumerations and a converter is registered with SQLite to allow for queries run from python to return a datetime object (instead of text).

import pathlib
from datetime import timezone
from faker import Faker

from fudgeo.geopkg import GeoPackage, SpatialReferenceSystem, Field
from fudgeo.enumeration import GeometryType, SQLFieldType
from fudgeo.geometry import Point
from fudgeo.sql import ESRI_4326

from fudgeo import __version__

assert __version__ == '0.2.0'

fake = Faker()

# Create or connect to GeoPackage
path_gpkg = pathlib.Path.cwd().joinpath("fudgeo_tutorial.gpkg")
if path_gpkg.exists():
    gpkg = GeoPackage(path_gpkg)
else:
    gpkg = GeoPackage.create(path_gpkg)
srs = SpatialReferenceSystem("GCS_WGS_1984", "EPSG", 4326, ESRI_4326)

# Fake up some data
rows = []
for _ in range(100):
    lat, lon, city, cc, tz = fake.location_on_land()
    lon = float(lon)
    lat = float(lat)
    rows.append((fake.name(), city, cc, lon, lat, fake.date_object(),
                 fake.date_time(), fake.date_time(tzinfo=timezone.utc),
                 Point(x=lon, y=lat, srs_id=4326)))

# Define fields
fields = (
    Field("full_name", SQLFieldType.text),
    Field("city", SQLFieldType.text),
    Field("country_code", SQLFieldType.text, 2),
    Field("longitude", SQLFieldType.float),
    Field("latitude", SQLFieldType.float),
    Field("my_date", SQLFieldType.date),
    # These next 2 are the ones that have issues:
    Field("my_datetime", SQLFieldType.datetime),
    Field("my_datetime_tz", SQLFieldType.datetime),
)

# Create feature class
table_name = "Datetime_Test"
fc = gpkg.create_feature_class(table_name, srs, fields=fields,
                               shape_type=GeometryType.point, overwrite=True)

# Write out the data
with gpkg.connection as conn:
    conn.executemany(
        f"""INSERT INTO {table_name} (
                full_name, city, country_code,
                longitude, latitude, my_date, 
                my_datetime, my_datetime_tz, SHAPE) 
           VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", rows)

# Selecting the date and datetime columns gives date and datetime objects
with gpkg.connection as conn:
    cursor = conn.execute(f"""SELECT my_date, my_datetime, my_datetime_tz FROM {table_name}""")
    features = cursor.fetchall()

# table will now show in ArcGIS Desktop and ArcGIS Pro with 
# Date type fo the date and datetime columns

from fudgeo.

jlhumber avatar jlhumber commented on July 21, 2024

@CF-FHB-X the ValueError you are seeing is coming from internals of Python's SQLite implementation for the default converters / adapters for date and datetime. This is not an issue specific to fudgeo.

The exception you see happens on retrieval of the datetime_tz (i.e. with UTC offset / time zone aware) since the format of the internally stored string value (in SQLite) has trailing +## is not expected by the default adapters. In general the approach is to resolve datetimes to UTC before inserting.

As for ArcGIS Desktop (10.8.1) and ArcGIS Pro (3.0.0), I see that they interpret the datetime and datetime_tz columns as Double, which is fascinating guess of the data type since the most probably type is Text (likely a bug / enhancement request to be reported to them).

Overcoming this issue in ArcGIS ecosystem you're onto something with the use of DATETIME as the field type, that seems to make the data type for the column map to Date in ArcGIS Desktop and ArcGIS Pro. Then to overcome the trickle down issue of no longer getting datetime objects from python queries you can do this:

# Define fields
fields = (
    Field("full_name", SQLFieldType.text),
    Field("city", SQLFieldType.text),
    Field("country_code", SQLFieldType.text, 2),
    Field("longitude", SQLFieldType.float),
    Field("latitude", SQLFieldType.float),
    Field("date", SQLFieldType.date),
    # These next 2 are the ones that have issues:
    Field("datetime", "DATETIME"),
    Field("datetime_tz", SQLFieldType.timestamp),
)

...

with gpkg.connection as conn:
    cursor = conn.execute("""SELECT datetime "[timestamp]" FROM Datetime_Test""")
    features = cursor.fetchall()

from fudgeo.

CF-FHB-X avatar CF-FHB-X commented on July 21, 2024

Is TIMESTAMP actually even a supported data type for GeoPackages? I'm reading this spec: https://www.geopackage.org/spec/. It seems they recommend DATETIME for datetimes.

from fudgeo.

jlhumber avatar jlhumber commented on July 21, 2024

So it does, thanks for the link.

from fudgeo.

CF-FHB-X avatar CF-FHB-X commented on July 21, 2024

Just awesome. Thanks a million! Will test it all on Monday.

Thanks again!

from fudgeo.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.