data-catalog's People
Contributors
Watchers
data-catalog's Issues
Migrate sync process
According to @dnfeldman
there is supposed to be a background job that periodically parses dataset's metadata json and populates the relevant fields
which is not currently there...
Can you use a postgres trigger? It makes from me
The trigger can be specified to fire before the operation is attempted on a row (before constraints are checked and the INSERT, UPDATE, or DELETE is attempted); or after the operation has completed (after constraints are checked and the INSERT, UPDATE, or DELETE has completed); or instead of the operation (in the case of inserts, updates or deletes on a view). If the trigger fires before or instead of the event, the trigger can skip the operation for the current row, or change the row being inserted (for INSERT and UPDATE operations only). If the trigger fires after the event, all changes, including the effects of other triggers, are "visible" to the trigger.
API Support DELETE
We cannot delete resource or databases
Update polygon point coverage fails
The dataset has one resource:
postgres=# select * from resources where dataset_id='5ca18d80-dd88-4f95-9e1c-665289756748';
-[ RECORD 1 ]-+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
data_url | https://data.mint.isi.edu/files/cycles-input-data/africa/soil_weather/cycles-6.87500-37.87500.soil_weather.zip
resource_type | Zip
json_metadata | {"spatial_coverage": {"type": "Point", "value": {"x": 37.0, "y": 6.0}}, "temporal_coverage": {"end_time": "2021-03-30T00:00:00", "start_time": "2000-01-01T00:00:00"}}
layout | {}
created_at | 2023-06-09 15:33:12.448088
updated_at | 2023-06-09 15:33:12.448088
name | cycles-6.87500-37.87500.soil_weather.zip
id | 202c581c-7e81-4933-afad-28ef1d320118
dataset_id | 5ca18d80-dd88-4f95-9e1c-665289756748
provenance_id | 9ef60317-5da5-4050-8bbc-7d6826fee49f
is_queryable | t
The sync_dataset_metadata triggers the function _update_polygon_point_coverage_ds. However, the query doesn't work
with spatial_coverage as (
select
dataset_id,
ST_union(ST_Simplify(st_buffer(spatial_coverage_index.spatial_coverage, 0.1), 0.1)) as dataset_spatial_coverage
from resources
inner join spatial_coverage_index on resources.id = spatial_coverage_index.indexed_id
--and resources.is_queryable is TRUE
where st_geometrytype(spatial_coverage_index.spatial_coverage) like '%Point'
and dataset_id='{dsid}'
group by dataset_id
)
update datasets
SET spatial_coverage = sc.dataset_spatial_coverage
from spatial_coverage sc
where sc.dataset_id = datasets.id"""
Inspecting the issue, the SELECT is returning an empty dataset_spatial_coverage value
select
dataset_id,
ST_union(ST_Simplify(st_buffer(spatial_coverage_index.spatial_coverage, 0.1), 0.1)) as dataset_spatial_coverage
from resources
inner join spatial_coverage_index on resources.id = spatial_coverage_index.indexed_id
--and resources.is_queryable is TRUE
where st_geometrytype(spatial_coverage_index.spatial_coverage) like '%Point'
and dataset_id='5ca18d80-dd88-4f95-9e1c-665289756748'
group by dataset_id
postgres-# group by dataset_id;
-[ RECORD 1 ]------------+-------------------------------------
dataset_id | 5ca18d80-dd88-4f95-9e1c-665289756748
dataset_spatial_coverage |
api is not using the env
db = Settings.get_instance().database
# db_host = os.environ.get("DB_HOST"),
# db_port = os.environ.get("DB_PORT")
# db_username = os.environ.get("DB_USERNAME")
# db_password = os.environ.get("DB_PASSWORD")
# db_name = os.environ.get("DB_NAME")
if db.host == 'localhost':
db.host = "host.docker.internal"Dockerfile is not building the frontend
This is important for the process.env.MAPGL_ACCESS_TOKEN variable.
Critical security vulnerability - axios
warning serverless > @serverless/enterprise-plugin > @serverless/platform-client > [email protected]:
Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.