informatievlaanderen / vsds-tech-docs Goto Github PK
View Code? Open in Web Editor NEWTechnical documentation, part of the VSDS Portal
Home Page: https://informatievlaanderen.github.io/VSDS-Tech-Docs/
Technical documentation, part of the VSDS Portal
Home Page: https://informatievlaanderen.github.io/VSDS-Tech-Docs/
At Consuming an LDES: when importing the flow file NiFi warns about the VersionMaterialiseProcessor:
This line and below: swagger2.html
seems not to be included or empty...
In this line there are two different URLs to the TREE specification. It's best to just use https://w3id.org/tree/specification .
Describe the BUG
There is an extra "-" in command
curl -X 'POST'
'http://localhost:8080/admin/api/v1/dcat'
-H 'accept: text/plain'
-H 'Content-Type: text/turtle'
---data-raw ' @Prefix dct: http://purl.org/dc/terms/ .
@Prefix dcat: http://www.w3.org/ns/dcat# .
[] a dcat:Catalog ;
dct:title "My LDES'''es"@en ;
dct:description "All LDES'''es from publiser X"@en .
'
To Reproduce
Site link or Repository
When running the NiFi flow as described in Consuming an LDES, step 4, the VersionMaterialiseProcessor appears to be doing nothing, and some exceptions like the following appear in the logging:
nifi-graph | 2023-05-12 06:42:39,624 INFO [Timer-Driven Process Thread-3] org.hibernate.tuple.PojoInstantiator HHH000182: No default (no-argument) constructor for class: be.vlaanderen.informatievlaanderen.ldes.client.member.sqlite.Member (class must be instantiated by Interceptor)
nifi-graph | 2023-05-12 06:42:39,846 WARN [Timer-Driven Process Thread-8] o.h.t.s.i.ExceptionHandlerLoggedImpl GenerationTarget encountered exception accepting command : Error executing DDL "create table Member (id varchar(255) not null, primary key (id))" via JDBC Statement
nifi-graph | org.hibernate.tool.schema.spi.CommandAcceptanceException: Error executing DDL "create table Member (id varchar(255) not null, primary key (id))" via JDBC Statement
nifi-graph | at org.hibernate.tool.schema.internal.exec.GenerationTargetToDatabase.accept(GenerationTargetToDatabase.java:67)
nifi-graph | at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.applySqlString(AbstractSchemaMigrator.java:581)
nifi-graph | at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.applySqlStrings(AbstractSchemaMigrator.java:526)
nifi-graph | at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.createTable(AbstractSchemaMigrator.java:293)
nifi-graph | at org.hibernate.tool.schema.internal.GroupedSchemaMigratorImpl.performTablesMigration(GroupedSchemaMigratorImpl.java:74)
nifi-graph | at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.performMigration(AbstractSchemaMigrator.java:220)
nifi-graph | at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.doMigration(AbstractSchemaMigrator.java:123)
nifi-graph | at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:196)
nifi-graph | at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:85)
nifi-graph | at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:335)
nifi-graph | at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:471)
nifi-graph | at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1498)
nifi-graph | at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:56)
nifi-graph | at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:79)
nifi-graph | at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:54)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.client.member.sqlite.SqliteMemberRepository.<init>(SqliteMemberRepository.java:15)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.client.state.SqlitePersistedLdesStateManager.<init>(SqlitePersistedLdesStateManager.java:57)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getSqlitePersistedStateManager(LdesClientImplFactory.java:42)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getStateManager(LdesClientImplFactory.java:23)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getLdesService(LdesClientImplFactory.java:54)
nifi-graph | at be.vlaanderen.informatievlaanderen.ldes.processors.LdesClient.onScheduled(LdesClient.java:76)
nifi-graph | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
nifi-graph | at java.base/java.lang.reflect.Method.invoke(Method.java:577)
nifi-graph | at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
nifi-graph | at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
nifi-graph | at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
nifi-graph | at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
nifi-graph | at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1736)
nifi-graph | at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
nifi-graph | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nifi-graph | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
nifi-graph | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
nifi-graph | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
nifi-graph | at java.base/java.lang.Thread.run(Thread.java:833)
nifi-graph | Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table Member already exists)
nifi-graph | at org.sqlite.core.DB.newSQLException(DB.java:1179)
nifi-graph | at org.sqlite.core.DB.newSQLException(DB.java:1190)
nifi-graph | at org.sqlite.core.DB.execute(DB.java:988)
nifi-graph | at org.sqlite.core.CoreStatement.exec(CoreStatement.java:79)
nifi-graph | at org.sqlite.jdbc3.JDBC3Statement.lambda$execute$0(JDBC3Statement.java:52)
nifi-graph | at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454)
nifi-graph | at org.sqlite.jdbc3.JDBC3Statement.execute(JDBC3Statement.java:40)
nifi-graph | at org.hibernate.tool.schema.internal.exec.GenerationTargetToDatabase.accept(GenerationTargetToDatabase.java:54)
The first sentence of Retention policy has a link that doesn't point to an example:
A retention policy determines how long data will be kept ...
Describe the BUG
First of all, I find that is not explained what you want to show with this examples.
2 ldes:EventStream are created: observations and mobility-hindrances
The creation of the 2 event streams is done differently:
a ldes:EventStream ;
server:observations a ldes:EventStream ;
Why is that ? Does it mean that the 2 event streams are handled differently by the server ?
I continue:
observations will process members of type “https://data.vlaanderen.be/ns/mobiliteit#ObservationCollection”
mobility-hindrances will process members of type “https://data.vlaanderen.be/ns/mobiliteit#Mobiliteitshinder”
So I understand they are completely different, even though they have the same root..
I POST 3 observations on localhost:8080/observations. If I go to localhost:8080/observations/by-page?pageNumber=1 I do find the 3 observations.. so OK..
Than the doc says
"Validate your LDES server is being populated by going to http://localhost:8080/mobility-hindrances/by-page?pageNumber=1 and http://localhost:8080/mobility-hindrances/time-based. These streams should fill up as the LDES Client send members to your server."
How can I expect to have any results on mobility-hindrances when I post observations ? How are the 2 event streams linked together ?
I was thinking that some config/istructions specified in ldio.yml would do the link between observations and mobility-hindrances, but I do not see it.
To Reproduce
Site link or Repository
The section Basic concepts of linked data contains, right below the figure, the text:
Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream, Linked data event stream Visualization of an RDF (RDF 1.1 Primer (w3.org))
which is probably not intended.
Describe the BUG
In the quick start, the LDI configuration uses the endpoint endpoint: http://host.docker.internal:8080/mobility-hindrances
. The host.docker.internal
is OS-specific and not always available depending if and how Docker desktop is installed. When it fails, it does so silently (unless one is looking at the docker container logs) and so it can be confusing for folks following the Quick Start for the first time.
Since the LDI orchestrator is part of the same docker service, this can be replaced with the name of the service e.g. endpoint: http://ldio-workbench:8080/mobility-hindrances
Site link or Repository
https://informatievlaanderen.github.io/VSDS-Tech-Docs/quickstart/Publish_LDES
The link on this line points to an unexisting URL.
When starting the docker containers following the instructions in the README, the quick-start_ldes-server
container exits with an error. Here's part of the logging:
quick-start_ldes-server | 2023-04-04T07:27:59.916Z WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'applicationRunner' defined in class path resource [io/mongock/runner/springboot/config/MongockContext.class]: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server | 2023-04-04T07:27:59.930Z WARN 1 --- [ main] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [BatchSpanProcessor_WorkerThread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
quick-start_ldes-server | java.base@18-ea/jdk.internal.misc.Unsafe.park(Native Method)
quick-start_ldes-server | java.base@18-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
quick-start_ldes-server | java.base@18-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
quick-start_ldes-server | java.base@18-ea/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435)
quick-start_ldes-server | io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:252)
quick-start_ldes-server | java.base@18-ea/java.lang.Thread.run(Thread.java:833)
quick-start_ldes-server | 2023-04-04T07:27:59.958Z INFO 1 --- [ main] .s.b.a.l.ConditionEvaluationReportLogger :
quick-start_ldes-server |
quick-start_ldes-server | Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
quick-start_ldes-server | 2023-04-04T07:27:59.977Z ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
quick-start_ldes-server |
quick-start_ldes-server | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'applicationRunner' defined in class path resource [io/mongock/runner/springboot/config/MongockContext.class]: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server | at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:645) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at org.springframework.boot.SpringApplication.run(SpringApplication.java:310) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server | at be.vlaanderen.informatievlaanderen.ldes.server.Application.main(Application.java:9) ~[classes!/:na]
quick-start_ldes-server | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
quick-start_ldes-server | at java.base/java.lang.reflect.Method.invoke(Method.java:577) ~[na:na]
quick-start_ldes-server | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[ldes-server-application.jar:na]
quick-start_ldes-server | at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[ldes-server-application.jar:na]
quick-start_ldes-server | at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[ldes-server-application.jar:na]
quick-start_ldes-server | at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:466) ~[ldes-server-application.jar:na]
quick-start_ldes-server | Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server | at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | ... 25 common frames omitted
quick-start_ldes-server | Caused by: io.mongock.api.exception.MongockException: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server | at io.mongock.runner.core.executor.ExecutorBuilderBase.validateScanPackage(ExecutorBuilderBase.java:131) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.executor.ExecutorBuilderDefault.getExecutorByOperation(ExecutorBuilderDefault.java:28) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.executor.ExecutorBuilderBase.buildOperationExecutor(ExecutorBuilderBase.java:87) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.builder.RunnerBuilderBase.buildOperationExecutor(RunnerBuilderBase.java:239) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:146) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:131) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:126) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.springboot.base.builder.SpringbootBuilderBase.buildApplicationRunner(SpringbootBuilderBase.java:103) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at io.mongock.runner.springboot.base.config.MongockContextBase.applicationRunner(MongockContextBase.java:27) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
quick-start_ldes-server | at java.base/java.lang.reflect.Method.invoke(Method.java:577) ~[na:na]
quick-start_ldes-server | at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server | ... 26 common frames omitted
I tested on a Linux host, a colleague of mine tested on a Windows host with the same result.
In Ingesting sources (HTTP in), the link in "More information on the HTTP ingestion can be found here." points to a fragment that's not there.
Due to a revision of docs/quickstart.md
in commit bea5b4422 some corrections made earlier are undone. See the inline comments in the commit.
At Publishing your first LDES:
In the chapter Geospatial fragmentation there is a typo in the sentence
The number of tiles if 2^2n^
Also, the second ^
seems to be unnecessary.
See this line
The docker compose file in Replicate an LDES with the LDES Client doesn't work out of the box. The issue is that it tries to connect to localhost
from outside the ldes
network. To fix that, the client can be told to connect to the host's network instead of the Docker network by adding network_mode: host
:
services:
ldes-cli:
image: ghcr.io/informatievlaanderen/ldes-cli:20230222T0959
container_name: quick-start_ldes-client-cli
command: "--url http://localhost:8080/sample/by-page --input-format text/turtle"
network_mode: host
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.