Microservice for on-demand synchronization of geographical vector datasources to a PostgreSQL/PostGIS database. The service provides an HTTP API for easy integration into other applications.
License: BSD 3-Clause "New" or "Revised" License
Option to discard features whose attributes might not be cast to the datatype of the column in the database.
This behavior should be optional configurable per layer and should be disabled by default-
This would be useful to handle
This option should allow a comma-seperated list of column names to handle composite primary keys.
The option should override existing primary keys
Allow layers to be pulled in an configurable interval.
In the most minimal case this would me mean adding a pull.job to the queue every n-seconds. In the best case batyr should estimate the total duration which it will take to finish all jobs which are queued and add the new pull event to the queue so that the interval is kept with as little delay as possible.
The second approach would mean collecting statistics on the duration of all job types and variants
I have tried to compile the project on debian stretch. Because of the updated libary versions in the distribution, i also updated the install steps.
I'm using the Gitlab CI Service for building, you can see the complete build progress here:
https://gitlab.com/lb1c/batyr/-/jobs/53577003
from /builds/lb1c/batyr/src/server/db/transaction.h:8,
from /builds/lb1c/batyr/src/server/db/connection.h:13,
from /builds/lb1c/batyr/src/server/worker.h:12,
from /builds/lb1c/batyr/src/server/worker.cpp:11:
/usr/include/gdal/ogrsf_frmts.h:245:25: note: declared here
static void DestroyDataSource( OGRDataSource * ) OGR_DEPRECATED("Use GDALDataset class instead");
^~~~~~~~~~~~~~~~~
src/server/CMakeFiles/batyrd.dir/build.make:614: recipe for target 'src/server/CMakeFiles/batyrd.dir/worker.cpp.o' failed
make[2]: *** [src/server/CMakeFiles/batyrd.dir/worker.cpp.o] Error 1
CMakeFiles/Makefile2:269: recipe for target 'src/server/CMakeFiles/batyrd.dir/all' failed
make[1]: *** [src/server/CMakeFiles/batyrd.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
ERROR: Job failed: exit code 1```
The encoding conversion could be handled in the database.
See https://raw.githubusercontent.com/pocoproject/poco/poco-1.6.0-release/CHANGELOG
This might allow dropping rapidjson. On the other side seems debian to be stuck on v1.3 and v1.4 of the poco libraries currently: https://packages.debian.org/search?keywords=poco
So this is more of a change for sometime in the future.
The backtrace send to stdout:
*** longjmp causes uninitialized stack frame ***: /usr/bin/batyrd terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f3d78cbb2a7]
/lib/x86_64-linux-gnu/libc.so.6(+0xef239)[0x7f3d78cbb239]
/lib/x86_64-linux-gnu/libc.so.6(__longjmp_chk+0x33)[0x7f3d78cbb1a3]
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4(+0xbc35)[0x7f3d71c41c35]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7f3d79704030]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xed77)[0x7f3d79703d77]
/lib/x86_64-linux-gnu/libpthread.so.0(sigwait+0x37)[0x7f3d79703df7]
/usr/lib/libPocoUtil.so.9(_ZN4Poco4Util17ServerApplication25waitForTerminationRequestEv+0x5d)[0x7f3d7ab5b49d]
/usr/bin/batyrd(_ZN5Batyr6Server4mainERKSt6vectorISsSaISsEE+0x131)[0x416fb1]
/usr/lib/libPocoUtil.so.9(_ZN4Poco4Util11Application3runEv+0x1b)[0x7f3d7ab4476b]
/usr/lib/libPocoUtil.so.9(_ZN4Poco4Util17ServerApplication3runEiPPc+0x79)[0x7f3d7ab5b809]
/usr/bin/batyrd(main+0x27)[0x4171f7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f3d78beaead]
/usr/bin/batyrd[0x4174f1]
To avoid issues like
2013-11-18 13:35:02 [Information] [Worker] job c8d14bb2e0f140a29fa6edd50a9b51925192 pulling layer "investements"
2013-11-18 13:35:02 [Information] [Worker] job c8d14bb2e0f140a29fa6edd50a9b51925192 geometry column geom_station_1 uses a known, defined SRID (21781). Reprojecting the geometries if they got a SRS assigned, otherwise assigning the SRID of the table.
2013-11-18 13:35:02 [Error] [Db::Transaction] query failed: prepared statement "batyr_insertc8d14bb2e0f140a29fa6edd50a9b51925192" already exists [sqlstate: 42P05]
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.