odpi / egeria-docs Goto Github PK
View Code? Open in Web Editor NEWDocumentation repository for the Egeria project.
Home Page: https://egeria-project.org
License: Other
Documentation repository for the Egeria project.
Home Page: https://egeria-project.org
License: Other
If Kafka is down, the default kafka connector will try for around 10 times, to ensure the cluster is up. It does this by attempting to get a non-null list of available brokers via an AdminClient.
There are docs on this behaviour at: https://github.com/odpi/egeria/tree/master/open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector
These docs need to be incorporated in the new user docs on egeria-docs
The default also appears to be 60s, not 5s as the doc above states (at least in the coco environment, and I cannot see an override) ie
on Sep 13 12:46:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 2
Mon Sep 13 12:47:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 3
Mon Sep 13 12:48:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 4
Some servers are also members of multiple cohorts. In this case the retry loop is repeated per cohort. For example cocoMDS2 tries cocoCohort then devCohort. After cocoCohort fails, we report an exception saying initialization failed. However the code continues serially with the next cohort (this is all whilst the /instance server start operation is being executed.. and which is in danger of timing out at the http request level)
Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Exception OMRS-AUDIT-0006 Configuration error detected while connecting to cohort devCohort, exception org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException was caught with error message: OCF-KAFKA-TOPIC-CONNECTOR-400-002 Egeria was unable to initialize a connection to a Kafka cluster. The message in the exception was: java.util.concurrent.ExecutionException
Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Exception OMRS-AUDIT-0006 Supplementary information: log record id cb3bac6d-6eff-4661-b750-c70394fa0d76 org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException returned message of OCF-KAFKA-TOPIC-CONNECTOR-400-002 Egeria was unable to initialize a connection to a Kafka cluster. The message in the exception was: java.util.concurrent.ExecutionException and stacktrace of
OCFCheckedExceptionBase{reportedHTTPCode=400, reportingClassName='org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector', reportingActionDescription='KafkaMonitor.waitForThisBroker', reportedErrorMessage='OCF-KAFKA-TOPIC-CONNECTOR-400-002 Egeria was unable to initialize a connection to a Kafka cluster. The message in the exception was: java.util.concurrent.ExecutionException', reportedErrorMessageId='OCF-KAFKA-TOPIC-CONNECTOR-400-002 ', reportedErrorMessageParameters=[java.util.concurrent.ExecutionException, egeria.omag.openmetadata.repositoryservices.cohort.devCohort.OMRSTopic.registration, org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s)], reportedSystemAction='The system is unable initialize.', reportedUserAction='Ensure that Kafka is available', reportedCaughtException=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s), reportedCaughtExceptionClassName='java.util.concurrent.ExecutionException', relatedProperties=null}
at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector.start(KafkaOpenMetadataTopicConnector.java:290)
at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.start(OMRSTopicConnector.java:258)
at org.odpi.openmetadata.repositoryservices.metadatahighway.OMRSCohortManager.setSecurityVerifier(OMRSCohortManager.java:347)
at org.odpi.openmetadata.repositoryservices.metadatahighway.OMRSMetadataHighwayManager.setSecurityVerifier(OMRSMetadataHighwayManager.java:128)
at org.odpi.openmetadata.repositoryservices.admin.OMRSOperationalServices.setSecurityVerifier(OMRSOperationalServices.java:850)
at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector$KafkaStatusChecker.waitForBrokers(KafkaOpenMetadataTopicConnector.java:502)
at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector.start(KafkaOpenMetadataTopicConnector.java:272)
... 61 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Cohort OMRS-AUDIT-0060 Registering with open metadata repository cohort devCohort using metadata collection id 9614d9b7-d223-4a8c-926d-23ce286c477c
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Cohort OMRS-AUDIT-0062 Requesting registration information from other members of the open metadata repository cohort devCohort
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0031 The iotCohort cohort inbound event manager is starting with 2 type definition event consumer(s) and 2 instance event consumer(s)
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0026 Initializing listener for cohort iotCohort
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0019 An OMRS Topic Connector has registered with an event bus connector for topic egeria.omag.openmetadata.repositoryservices.cohort.iotCohort.OMRSTopic.registration
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0001 Connecting to Apache Kafka Topic egeria.omag.openmetadata.repositoryservices.cohort.iotCohort.OMRSTopic.registration with a server identifier of 89af21fe-e3fa-4dfe-a307-bfe782616006
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 1
[I 2021-09-13 12:46:05.873 ServerApp] Saving file at /common/environment-check.ipynb
Mon Sep 13 12:46:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 2
The retry loop in logging an audit message only reports:
Mon Sep 13 12:34:56 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 10
It may be useful to enhance this with
As this will help someone monitoring the system and understanding what is going wrong.
cocoMDS2 continues to do the same with iotCohort
After cohort initialization is complete we continue with OMASs ie:
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Connected Asset Services
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Connected Asset Services has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-METADATA-MANAGEMENT-0001 The Open Connector Framework (OCF) Metadata Management Service is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-METADATA-MANAGEMENT-0003 The Open Connector Framework (OCF) Metadata Management Service has initialized a new instance for server cocoMDS2
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAG-ADMIN-0010 The Open Metadata Access Services (OMASs) are starting
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Asset Catalog OMAS
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Asset Catalog OMAS has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CATALOG-0002 The Asset Catalog Open Metadata Access Service (OMAS) is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CATALOG-0001 The Asset Catalog Open Metadata Access Service (OMAS) has initialized a new instance for server cocoMDS2
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Asset Consumer OMAS
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Asset Consumer OMAS has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CONSUMER-0001 The Asset Consumer Open Metadata Access Service (OMAS) is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0001 Connecting to Apache Kafka Topic egeria.omag.server.cocoMDS2.omas.assetconsumer.outTopic with a server identifier of 89af21fe-e3fa-4dfe-a307-bfe782616006
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 1
After the failure of asset catalog shutdown then starts:
Mon Sep 13 13:05:58 BST 2021 cocoMDS2 Shutdown OMAG-ADMIN-0005 The cocoMDS2 server has begun the shutdown process
So currently the behaviour is
The intent needs clarification and documentation.
It's worth noting that broker availability is one possible check, but there are other failures. For example a topic may not exist - this may be short lived as it will get created by producers, but is still something that could continue indefinately for some topics. Further, if auto topic creation is disabled it will never exist without admin action. This is already handled within the producer/consumer directly and events are already queued to be sent.
#5926 added 2 new properties to KafkaTopic
Document Link: https://odpi.github.io/egeria-docs/introduction/key-concepts/
In the Metadata types part,
Under "An example: GlossaryTerm, RelationalColumn and SemanticAssignment", the 3rd paragraph:
"... it is defined using a RelationshipDef (another subtype of EntityDef, this time specific to relationships). "
I think "another subtype of EntityDef" should change to "another subtype of TypeDef".
Getting error when running OMAG server
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java -Dserver.port=9443 -jar server-chassis-spring-3.4-SNAPSHOT.jar
Project Egeria - Open Metadata and Governance
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / / / / ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ / ____ _ _
/ / / // /|/ // /| | / / __ _ \ / _ \ / __/| | / // _ \ / __/ / // // // | / \ / / / | / // || |
/ // // / / // ___ |/ // / / // _// / | |/ // // / / __ // // / \ / / / // / // / / / / /
_/// //// ||_/ // ___/// |/ _/// // // _////// _/// // /_/
:: Powered by Spring Boot (v2.5.6) ::
2021-11-19 15:28:37.507 INFO 1380 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9443 (https)
2021-11-19 15:28:43.048 ERROR 1380 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.12.jar!/:5.3.12]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.odpi.openmetadata.serverchassis.springboot.OMAGServerPlatform.main(OMAGServerPlatform.java:95) ~[classes!/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:229) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:43) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.12.jar!/:5.3.12]
... 22 common frames omitted
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:238) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:282) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:213) ~[spring-boot-2.5.6.jar!/:2.5.6]
... 24 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1075) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:234) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 26 common frames omitted
Caused by: java.lang.IllegalArgumentException: /home/dimikhan/egeria-install/egeria-omag-3.4-SNAPSHOT/server/truststore.p12 (No such file or directory)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:231) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1208) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1294) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:614) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1072) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 28 common frames omitted
Caused by: java.io.FileNotFoundException: /home/dimikhan/egeria-install/egeria-omag-3.4-SNAPSHOT/server/truststore.p12 (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method) ~[na:na]
at java.base/java.io.FileInputStream.open(FileInputStream.java:219) ~[na:na]
at java.base/java.io.FileInputStream.(FileInputStream.java:157) ~[na:na]
at java.base/java.io.FileInputStream.(FileInputStream.java:112) ~[na:na]
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86) ~[na:na]
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184) ~[na:na]
at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:197) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLHostConfig.getTruststore(SSLHostConfig.java:730) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:422) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:245) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 34 common frames omitted
As per - https://odpi.github.io/egeria-docs/education/tutorials/omag-server-tutorial/task-starting-the-omag-server-platform/ the OMAG server will be started.
Following page - https://odpi.github.io/egeria-docs/education/tutorials/installing-egeria-tutorial/
Using Windows 10 with WSL2 - Linux version Ubuntu 20.04 LTS.
Could able execute git clone https://github.com/odpi/egeria.git and run mvn clean install. But when checking egeria-install/egeria-omag-3.4-SNAPSHOT/server path I dont see the same directory structure as mentioned in https://odpi.github.io/egeria-docs/education/tutorials/installing-egeria-tutorial/ page. Below is the directory structure.
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ ls lib server-chassis-spring-3.4-SNAPSHOT.jar
However, finally tried to start the server as below -
'dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java -Dserver.port=9443 -jar server-chassis-spring-3.4-SNAPSHOT.jar'
- Egeria: 3.4
- OS: Windows 10 WSL2 Ubuntu 10.04 TLS
- Java:
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
- Browser (for UI issues): N/A
- Additional connectors and integration: N/A
No response
I just went to https://odpi.github.io/egeria-docs/guides/operations/kubernetes/intro.md as navigated to from Using -> Operations -> Egeria in Kubernetes. This was working until recently but is now giving a 404
Should the road map and status (https://egeria.odpi.org/open-metadata-publication/website/roadmap/) be under "Overview" with links to releases ?
To capture an overall strategic outline of what repositories we should have, and what content should be in each. Should form a "roadmap" that we gradually drive towards with separating out the core repository content + provide initial documentation for what all the repositories are, why the exist, and where to go for what.
There's a reasonable chunk of work to do regarding migrating the Types documentation. The tasks would involve:
Naturally follow the documentation guide as part of the updates (remove the license footers from each page, retain the license headers, ensure appropriate use of capitalization vs italics and bold, etc).
The base model (area 0) has been migrated already according to these principles as a starting point -- only 6 more areas to go! ๐ฌ
(For reference, here's the "example of 'good'" in the repo itself: https://github.com/odpi/egeria-docs/tree/main/site/docs/types/0, with this being the output: https://odpi.github.io/egeria-docs/types/0/0010-base-model/)
The description of the servers is not accurate - probably comes from an out-of-date file. The platform implements the
services and the servers are collections of activated services that host connectors to the different technologies that egeria is exchanging metadata with. Then take people to the key concepts to explain more detail rather than sending them off to look at the deep implementation details in terms of connectors, services and frameworks.
Recently I migrated the docs for our helm charts, and improved some basic Kubernetes into information, including use of microk8s, over to https://odpi.github.io/egeria-charts
Opening to track migrating this into our new shiny doc repo
When debugging, it can be useful to log both requests/response made to the server (within the client, 3rd party app, or chassis) as well as the server chassis's inbound requests & response sent.
From a client perspective there are options such as using RestTemplate logging ( spring property logging.level.org.springframework.web.client.RestTemplate=DEBUG )
For the server, we run tomcat, and access logs can be enabled as per https://howtodoinjava.com/spring-boot2/logging/embedded-server-logging-config . There are also other options such as enabling the spring logging actuator, or a logging filter that may be clearer.
As a first step we should document suggested current logging mechanisms, and if insufficient look to enhance that capability in the code
(Originated from a user's query)
It would be useful to capture key information about the GitHub repositories we use for Egeria.
This is primarily for developers.
The information could contain
An alternative option is to keep the link in the docs light, with only a repo name/link, and then point to a well defined .md file within each repository. This should be of a mandated format, but is then owned by the linked project. we loose searchability, and it is dependent on each repo making the change. It also may make it tricker to identify inconsistencies
We could model this in egeria.....
MkDocs Material theme is now at v8 ๐ (https://github.com/squidfunk/mkdocs-material/releases/tag/8.0.0)
This adds support for i.e. code annotations, but requires some minor updates to the mkdocs.yml
structure.
The egeria developer guide can be found fairly easy by navigating from the home page.
Currently we have 5 types of interfaces listed. However two link to the docs on writing a new connector, whilst the other 3 all link to the list of OMASs.
It would be helpful to
See also odpi/egeria#3458
cc: @CDaRip2U
Having just setup a clean postman workspace today, the latest version (I was using 9.2.0) now allows import of collections direct from GitHub
This can be added as a suggested route for importing our collections.
Need to refine the docs to provide this as a preferred path, but also point out they can be done individually
Also review which collections should be imported.
Raising as a reminder to add this into Dojo docs
The URL of the page that the broken link appears on:
https://odpi.github.io/egeria-docs/getting-started/dojo/
Menu "Hands-on Labs Preview"
The link on that page (right-click, and copy it) that is broken:
https://odpi.github.io/egeria-docs/getting-started/hands-on-labs/index.md
Our Dojo series is being updated for early 2022.
This issue will track changes for Day 1
on the page https://odpi.github.io/egeria-docs/services/omvs/
following links do not work
Our current release notes are fairly brief, and focus on the main new features or significant changes in a release.
There is a variety of information that would be useful to include such as
In some cases we may wish to include this, in other cases point to a query
However to generate these automatically (at least as a draft to cut/paste) we need a more rigid process - documentation and enforcement through actions ie to
Opening as a future item to look at ....
We document how to configure TLS when running egeria at https://odpi.github.io/egeria-docs/guides/admin/configuring-the-omag-server-platform/?h=tls#brief-background-on-tls
However this only covers directly launching the chassis, and not when running in docker.
Our docker image for Egeria is based on Redhat UBI openjdk11 image.
Various environment variables can be set.
We currently set
ENV JAVA_OPTS_APPEND="-XX:MaxMetaspaceSize=1g"
to avoid some memory issues found with CTS, but more generally this environment variable allows other JVM options to be passed. This would include the ssl configuration referred to in the original link
The docs here should be extended to cover the docker image (there is currently none), and specifically how TLS can be configured when using the image
Further, this should be extended to our other docker images we build for connectors and UI (cc: @davidradl @sarbull @cmgrote @wbittles ) - some of these being based on other images
Also to note - as part of the work on the k8s operator, it too will be making TLS certs available via secrets, and in addition to documentation for the operator, it would be useful to document this technique for the benefit of those deploying with their own yaml/charts etc in k8s
The instructions for setting up microk8s on windows at https://odpi.github.io/egeria-docs/guides/operations/kubernetes/k8s/ refer the user to the official windows instructions.
Whilst that's initially appropriate -- to download and run the GUI installer, it then needs to be more explicit.
Following install/reboot the same instructions are used as for macOS (so this can be common) ie
Microsoft Windows [Version 10.0.19044.1348]
nigel@MEGA C:\Users\nigel
$ microk8s start
Started.
nigel@MEGA C:\Users\nigel
$ microk8s status --wait-ready
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
kata # Kata Containers is a secure runtime with lightweight VMS
keda # Kubernetes-based Event Driven Autoscaling
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
openebs # OpenEBS is the open-source storage solution for Kubernetes
openfaas # openfaas serverless framework
portainer # Portainer UI for your Kubernetes cluster
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
traefik # traefik Ingress controller for external access
nigel@MEGA C:\Users\nigel
$ microk8s enable dns storage helm3
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Enabling Helm 3
Fetching helm version v3.5.0.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.7M 100 11.7M 0 0 4935k 0 0:00:02 0:00:02 --:--:-- 4935k
Helm 3 is enabled
nigel@MEGA C:\Users\nigel
$ microk8s kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/calico-kube-controllers-578cfb99d5-zd9px 1/1 Running 0 5m17s
kube-system pod/coredns-7f9c69c78c-2pqxx 1/1 Running 0 29s
kube-system pod/calico-node-cxpfc 1/1 Running 0 5m17s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 5m24s
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 29s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 5m24s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 5m24s
kube-system deployment.apps/coredns 0/1 1 0 29s
kube-system deployment.apps/hostpath-provisioner 0/1 0 0 18s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-578cfb99d5 1 1 1 5m17s
kube-system replicaset.apps/coredns-7f9c69c78c 1 1 0 29s
nigel@MEGA C:\Users\nigel
$
A few other observations:
ON Helm....
A caveat is noted at https://odpi.github.io/egeria-docs/guides/operations/kubernetes/helm/ that the user needs to use 'microk8s.kubectl' or 'microk8s.helm'. This is non-standard. For the dojo we will explicitly use 'microk8s helm3 ....' but some users may still be confused by these joint instructions. An adominition, perhaps offering the variant may help as otherwise the user may pickup an old version or wrong version of helm, and for those new to k8s this could be confusing
And testing our charts - the instructions working ok on x86 windows 10
nigel@MEGA C:\Users\nigel
$ microk8s helm3 repo add egeria https://odpi.github.io/egeria-charts
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
"egeria" has been added to your repositories
nigel@MEGA C:\Users\nigel
$ microk8s helm3 search repo egeria
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
NAME CHART VERSION APP VERSION DESCRIPTION
egeria/egeria-base 3.3.0 3.3 Egeria simple deployment to Kubernetes
egeria/egeria-cts 3.3.0 3.3 Egeria Conformance Test Suite deployment to Kub...
egeria/egeria-pts 3.3.1 3.3 Egeria Performance Test Suite deployment to Kub...
egeria/odpi-egeria-lab 3.3.0 3.3 Egeria lab environment
nigel@MEGA C:\Users\nigel
$ microk8s helm3 install lab egeria/odpi-egeria-lab
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
NAME: lab
LAST DEPLOYED: Thu Nov 11 12:16:33 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
ODPi Egeria lab tutorial
---
The Egeria tutorials have now been deployed to Kubernetes.
It may take a minute or so for everything to start up.
Open your web browser and go to addressofmycluster:30888 to
get started
You may need to contact your cluster admin, or read your cloud provider helptext to understand
the correct 'addressofmycluster' to use.
If you experience problems, check memory consumption on your nodes. A minimum of
a 3 node cluster, 2GB per node; or a desktop environment with 6GB dedicated is recommended.
Please provide any feeback via a github issue at https://github.com/odpi/egeria or
join us on slack via https://http://slack.lfai.foundation
- The ODPi Egeria team
nigel@MEGA C:\Users\nigel
$
Overall the structure is looking excellent. Here are some specific comments ...
At https://odpi.github.io/egeria-docs/guides/admin/servers/configuring-a-metadata-access-point/#set-the-server-url-root we explain serverURL root
I have had a few people discuss this on slack & get confused when running in a container or complex/vpn environment
We need to clarify that this address is used for other servers in the cohort to call this server's REST endpoints - or at least that it is the default for that. Effectively it is how this cohort member advertizes its services to other members
Followon from odpi/egeria#4664 to document the process of making a release
In our 'egeria-base' helm chart there is information on adding additional libraries (such as connectors) for deployment at https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/charts/egeria-base
This has been tested previously and allows additional libraries to effectively be added to LOADER_PATH in the deployment
However when testing odpi/egeria-database-connectors#73 this fails
For example
However this then fails with:
$ helm install base egeria-base [12:57:55]
Error: create: failed to create: Request entity too large: limit is 3145728
$ ls -la egeria-base/libs [12:58:13]
total 2056
drwxr-xr-x 4 jonesn staff 128 15 Jun 12:57 .
drwxr-xr-x 10 jonesn staff 320 9 Jun 11:08 ..
-rw-r--r-- 1 jonesn staff 43350 15 Jun 12:56 egeria-connector-postgres-2.10-SNAPSHOT.jar
-rw-r--r--@ 1 jonesn staff 1005332 15 Jun 12:57 postgresql-42.2.21.jar
jonesn:charts/ (master)
These libraries are being encoded in base64, and as such will take more space than the binaries, quite likely exceeding the number given, which is the max size of a kubernetes API request...
This approach therefore needs rethinking. One viable alternative is to build a customized docker image with the required connector (+egeria) - indeed we already do this for the current SAS Viya connector at https://github.com/odpi/egeria-sas-viya-connector
Need to consider other options, but in either case it seems the current documented approach will not work well for k8s as this 3MB limit is very small (but was enough to pass my testing ....!)
Hi there, on the following page: https://odpi.github.io/egeria-docs/getting-started/dojo/egeria-dojo-day-1-3-1-2-configuring-the-platform/ just under the 2nd Postman screenshot is the following text:
"If the baseURL variable is set to a different value to the server platform then Postman can not connect. In the screen capture below, you can see the baseURL is set to the default of https://localhost:9443 when it should be https://localhost:9443 because the platform is running in docker."
Both locations stated here are identical so one must be wrong. I also note that the URL referenced in the screenshot underneath shows the port number 8080 for the baseURL.
Looks like something is amiss here?
PR Docs build allows the user to press merge when there are breaking errors in the code. It is therefore possible to think you have merged but you have not and the pr be closed.
The following links generate a 404 error:
Formatting Standards (https://odpi.github.io/egeria-docs/guides/documentation/guide/formatting) to keep the content of the documentation consistent.
The link should be (https://odpi.github.io/egeria-docs/guides/documentation/guide/)
Style Guide (https://odpi.github.io/egeria-docs/guides/documentation/guide/style) to keep the documentation clear and understandable.
The link should be (https://odpi.github.io/egeria-docs/guides/documentation/style/)
Hi, I spotted a typo in the docs.
https://odpi.github.io/egeria-docs/introduction/challenge/
In the 4 bullets at end of page, 2nd bullet 'envirohment'.
If there is a single ticket for typos like this let me know and I'll add to that rather than create a new ticket as I find any.
We have moved the webinar content from the data governance repository. We still point to workshops and presentations into the data governance site from the current repository. I suggest moving this content from the data governance site here. Also we agreed in a community meeting that the governance principles fro the data governance git repo should be moved into this repository, as they then would be searchable with the rest if the content.
Themain egeria docs need to be extended to cover running the Egeria UIs
Currently there is limited documentation
The community guid link on the top readme is broken.
https://github.com/odpi/egeria-docs/blob/main/README.md
Under the Key Concepts page, the 4 types of governance server should be listed since all of the other types of server are shown. Could also show them in the inheritance hierarchy structure. I would also suggest that nested under Key concepts, there should be:
Not all k8s implementations available for macOS work on the latest m1/ARM based macbooks
This can cause confusion for anyone trying to use our helm charts .
Though fundamentally egeria is agnostic to k8s implementation and platform, we are also trying to engage new users & help them learn about egeria, and a working k8s implementation is a crucial part of that.
We need to consider documenting an appropriate route, and add caveats so that users don't waste their time.
When using the docker container for egeria server-chassis (and ui-chassis), sometimes additional parameters need to be passed.
Since our container is based on Redhat UBI8 openjdk-11 image, this capability is already part of the image.
Some older (v3) docs are at Our container image is based on Redhat UBI8 openjdk-11 . There are some docs at
https://access.redhat.com/documentation/en-us/red_hat_jboss_middleware_for_openshift/3/html/red_hat_java_s2i_for_openshift/index
This covers the use of additional parms such as JAVA_ARGS
which can add additional parameters to the invocation of the selected executable. for example in the case of the ui-chassis this could include cors settings.
Will add to docs...
When swapping to dark mode for the main page of the MkDocs rendered site, the colours of the content (and the "hero" image) are not inverted to be readable in dark mode. All other pages seem to work fine in dark mode.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.