Giter VIP home page Giter VIP logo

vsds-tech-docs's People

Contributors

ddvlanck avatar ghsnd avatar kkostov avatar mpparsley avatar samuvack avatar tomvbe avatar wlefever-cegeka avatar xdxxxdx avatar yalz avatar

Watchers

 avatar  avatar

vsds-tech-docs's Issues

docs/quickstart.md: Consuming an LDES: Running NiFi flow gives exceptions

When running the NiFi flow as described in Consuming an LDES, step 4, the VersionMaterialiseProcessor appears to be doing nothing, and some exceptions like the following appear in the logging:

nifi-graph  | 2023-05-12 06:42:39,624 INFO [Timer-Driven Process Thread-3] org.hibernate.tuple.PojoInstantiator HHH000182: No default (no-argument) constructor for class: be.vlaanderen.informatievlaanderen.ldes.client.member.sqlite.Member (class must be instantiated by Interceptor)

nifi-graph  | 2023-05-12 06:42:39,846 WARN [Timer-Driven Process Thread-8] o.h.t.s.i.ExceptionHandlerLoggedImpl GenerationTarget encountered exception accepting command : Error executing DDL "create table Member (id varchar(255) not null, primary key (id))" via JDBC Statement
nifi-graph  | org.hibernate.tool.schema.spi.CommandAcceptanceException: Error executing DDL "create table Member (id varchar(255) not null, primary key (id))" via JDBC Statement
nifi-graph  | 	at org.hibernate.tool.schema.internal.exec.GenerationTargetToDatabase.accept(GenerationTargetToDatabase.java:67)
nifi-graph  | 	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.applySqlString(AbstractSchemaMigrator.java:581)
nifi-graph  | 	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.applySqlStrings(AbstractSchemaMigrator.java:526)
nifi-graph  | 	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.createTable(AbstractSchemaMigrator.java:293)
nifi-graph  | 	at org.hibernate.tool.schema.internal.GroupedSchemaMigratorImpl.performTablesMigration(GroupedSchemaMigratorImpl.java:74)
nifi-graph  | 	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.performMigration(AbstractSchemaMigrator.java:220)
nifi-graph  | 	at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.doMigration(AbstractSchemaMigrator.java:123)
nifi-graph  | 	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:196)
nifi-graph  | 	at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:85)
nifi-graph  | 	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:335)
nifi-graph  | 	at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:471)
nifi-graph  | 	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1498)
nifi-graph  | 	at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:56)
nifi-graph  | 	at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:79)
nifi-graph  | 	at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:54)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.client.member.sqlite.SqliteMemberRepository.<init>(SqliteMemberRepository.java:15)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.client.state.SqlitePersistedLdesStateManager.<init>(SqlitePersistedLdesStateManager.java:57)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getSqlitePersistedStateManager(LdesClientImplFactory.java:42)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getStateManager(LdesClientImplFactory.java:23)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.client.LdesClientImplFactory.getLdesService(LdesClientImplFactory.java:54)
nifi-graph  | 	at be.vlaanderen.informatievlaanderen.ldes.processors.LdesClient.onScheduled(LdesClient.java:76)
nifi-graph  | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
nifi-graph  | 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
nifi-graph  | 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
nifi-graph  | 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
nifi-graph  | 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
nifi-graph  | 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
nifi-graph  | 	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1736)
nifi-graph  | 	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
nifi-graph  | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nifi-graph  | 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
nifi-graph  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
nifi-graph  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
nifi-graph  | 	at java.base/java.lang.Thread.run(Thread.java:833)
nifi-graph  | Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (table Member already exists)
nifi-graph  | 	at org.sqlite.core.DB.newSQLException(DB.java:1179)
nifi-graph  | 	at org.sqlite.core.DB.newSQLException(DB.java:1190)
nifi-graph  | 	at org.sqlite.core.DB.execute(DB.java:988)
nifi-graph  | 	at org.sqlite.core.CoreStatement.exec(CoreStatement.java:79)
nifi-graph  | 	at org.sqlite.jdbc3.JDBC3Statement.lambda$execute$0(JDBC3Statement.java:52)
nifi-graph  | 	at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454)
nifi-graph  | 	at org.sqlite.jdbc3.JDBC3Statement.execute(JDBC3Statement.java:40)
nifi-graph  | 	at org.hibernate.tool.schema.internal.exec.GenerationTargetToDatabase.accept(GenerationTargetToDatabase.java:54)

docs/_quickstart/Publish_LDES.md is not clear and has errors

Describe the BUG
First of all, I find that is not explained what you want to show with this examples.
2 ldes:EventStream are created: observations and mobility-hindrances

The creation of the 2 event streams is done differently:
a ldes:EventStream ;
server:observations a ldes:EventStream ;
Why is that ? Does it mean that the 2 event streams are handled differently by the server ?

I continue:
observations will process members of type “https://data.vlaanderen.be/ns/mobiliteit#ObservationCollection”
mobility-hindrances will process members of type “https://data.vlaanderen.be/ns/mobiliteit#Mobiliteitshinder”
So I understand they are completely different, even though they have the same root..

I POST 3 observations on localhost:8080/observations. If I go to localhost:8080/observations/by-page?pageNumber=1 I do find the 3 observations.. so OK..
Than the doc says
"Validate your LDES server is being populated by going to http://localhost:8080/mobility-hindrances/by-page?pageNumber=1 and http://localhost:8080/mobility-hindrances/time-based. These streams should fill up as the LDES Client send members to your server."
How can I expect to have any results on mobility-hindrances when I post observations ? How are the 2 event streams linked together ?
I was thinking that some config/istructions specified in ldio.yml would do the link between observations and mobility-hindrances, but I do not see it.

To Reproduce

Site link or Repository

Incorrect LDI host configuration

Describe the BUG
In the quick start, the LDI configuration uses the endpoint endpoint: http://host.docker.internal:8080/mobility-hindrances. The host.docker.internal is OS-specific and not always available depending if and how Docker desktop is installed. When it fails, it does so silently (unless one is looking at the docker container logs) and so it can be confusing for folks following the Quick Start for the first time.

Since the LDI orchestrator is part of the same docker service, this can be replaced with the name of the service e.g. endpoint: http://ldio-workbench:8080/mobility-hindrances

Site link or Repository
https://informatievlaanderen.github.io/VSDS-Tech-Docs/quickstart/Publish_LDES

README: quick-start_ldes-server exits with error.

When starting the docker containers following the instructions in the README, the quick-start_ldes-server container exits with an error. Here's part of the logging:

quick-start_ldes-server   | 2023-04-04T07:27:59.916Z  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'applicationRunner' defined in class path resource [io/mongock/runner/springboot/config/MongockContext.class]: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server   | 2023-04-04T07:27:59.930Z  WARN 1 --- [           main] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [BatchSpanProcessor_WorkerThread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
quick-start_ldes-server   |  java.base@18-ea/jdk.internal.misc.Unsafe.park(Native Method)
quick-start_ldes-server   |  java.base@18-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
quick-start_ldes-server   |  java.base@18-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
quick-start_ldes-server   |  java.base@18-ea/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435)
quick-start_ldes-server   |  io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:252)
quick-start_ldes-server   |  java.base@18-ea/java.lang.Thread.run(Thread.java:833)
quick-start_ldes-server   | 2023-04-04T07:27:59.958Z  INFO 1 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger : 
quick-start_ldes-server   | 
quick-start_ldes-server   | Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
quick-start_ldes-server   | 2023-04-04T07:27:59.977Z ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed
quick-start_ldes-server   | 
quick-start_ldes-server   | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'applicationRunner' defined in class path resource [io/mongock/runner/springboot/config/MongockContext.class]: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:645) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:310) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:3.0.5]
quick-start_ldes-server   | 	at be.vlaanderen.informatievlaanderen.ldes.server.Application.main(Application.java:9) ~[classes!/:na]
quick-start_ldes-server   | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
quick-start_ldes-server   | 	at java.base/java.lang.reflect.Method.invoke(Method.java:577) ~[na:na]
quick-start_ldes-server   | 	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[ldes-server-application.jar:na]
quick-start_ldes-server   | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[ldes-server-application.jar:na]
quick-start_ldes-server   | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[ldes-server-application.jar:na]
quick-start_ldes-server   | 	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:466) ~[ldes-server-application.jar:na]
quick-start_ldes-server   | Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.mongock.runner.springboot.base.MongockApplicationRunner]: Factory method 'applicationRunner' threw exception with message: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	... 25 common frames omitted
quick-start_ldes-server   | Caused by: io.mongock.api.exception.MongockException: Scan package for changeLogs is not set: use appropriate setter
quick-start_ldes-server   | 	at io.mongock.runner.core.executor.ExecutorBuilderBase.validateScanPackage(ExecutorBuilderBase.java:131) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.executor.ExecutorBuilderDefault.getExecutorByOperation(ExecutorBuilderDefault.java:28) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.executor.ExecutorBuilderBase.buildOperationExecutor(ExecutorBuilderBase.java:87) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.builder.RunnerBuilderBase.buildOperationExecutor(RunnerBuilderBase.java:239) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:146) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:131) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.core.builder.RunnerBuilderBase.buildRunner(RunnerBuilderBase.java:126) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.springboot.base.builder.SpringbootBuilderBase.buildApplicationRunner(SpringbootBuilderBase.java:103) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at io.mongock.runner.springboot.base.config.MongockContextBase.applicationRunner(MongockContextBase.java:27) ~[mongock-changeset-1-jar-with-dependencies.jar:na]
quick-start_ldes-server   | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
quick-start_ldes-server   | 	at java.base/java.lang.reflect.Method.invoke(Method.java:577) ~[na:na]
quick-start_ldes-server   | 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139) ~[ldes-fragmentisers-geospatial-jar-with-dependencies.jar:6.0.7]
quick-start_ldes-server   | 	... 26 common frames omitted

I tested on a Linux host, a colleague of mine tested on a Windows host with the same result.

docs/quickstart.md: Replicate an LDES with the LDES Client: doesn't connect to LDES server

The docker compose file in Replicate an LDES with the LDES Client doesn't work out of the box. The issue is that it tries to connect to localhost from outside the ldes network. To fix that, the client can be told to connect to the host's network instead of the Docker network by adding network_mode: host:

services:
  ldes-cli:
    image: ghcr.io/informatievlaanderen/ldes-cli:20230222T0959
    container_name: quick-start_ldes-client-cli
    command: "--url http://localhost:8080/sample/by-page --input-format text/turtle"
    network_mode: host

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.