Allow additional connectors to be loaded into container image at runtime about egeria-docs HOT 7 CLOSED

The approach we're still taking for all of the other connectors is:

1. Create an emptyDir volume in the spec
2. initContainer that runs a bare image with retrieval command of your choice (curl / wget / etc), mounts the emptyDir volume, and uses the retrieval command(s) to download the necessary pre-built libraries to that volume
3. container that runs the core odpi/egeria image and mounts the emptyDir volume that (2) has initialized with the connector(s) to a location that an env variable LOADER_PATH also points to

The core Egeria image is then able to automatically pickup all the connectors for configuration, and run accordingly.

(If needed, this also allows (3) to be run directly as a side-car repository to some other image entirely: for example, for Apache Atlas we can run a vanilla Atlas docker container as a second container in the same deployment (side-car) right next to the Egeria container of (3) and the two are then "joined-at-the-hip" at runtime as well.)

This avoids any need to build a separate container for the connector, and doesn't touch the base Egeria image. Both can be configured at deployment time (e.g. to use latest SNAPSHOTs, releases, or some combination thereof), so injecting development code is as simple as making it available to those retrieval commands (e.g. checking it in / uploading it somewhere).

While I could see some concern with constantly retrieving the connectors in the initContainer step (2), this is what Docker is doing for images anyway (if the pull policy is Always), and I would expect the connector libraries themselves to be an order of magnitude smaller (5-10MBs) than Docker images (10-100MBs).

from egeria-docs.

As an aside it's noted that etcd (used for storing yaml in k8s) also has a limit around the 1.5MB size. Although this can be increased when using etcd directly, this is unlikely to be feasible in a k8s deployment. This may also mean files such as egeria configuration files cannot be managed as k8s resources.

from egeria-docs.

The approach taken by the postgres connector (building a separate image) works ok on top of a stable release. However during development it will rapidly go out of date, since the build is only triggered by changes to the connector, not the egeria code itself.... Note that in the case of this connector, the merge build as a workflow-dispatch trigger so can be refreshed manually as a workaround.

.. more thinking required as to best way to handle this.

from egeria-docs.

Agreed this is a flexible approach. I plan on adding this to the egeria-base, perhaps odpi-egeria-lab charts as time allows - but unlikely for a few weeks (other tasks got in the way & vacation) - or post here if you plan to do.
caching of images should be addressible easily when using statefulsets as pods retain an identity, so can be associated with specific storage.
The modified docker image is still a viable approach for others taking our images and rebuilding who want a more strict hard-baked configuration. But for development/demo/tutorials & indeed a flexible OOTB approach, dynamic downloading is superior imo.
On adding images, I don't have any concerns with this being an option to add in sidecars - there's so many different purposes they could be used for - ( like network proxies too), including this one, but beyond a simple demo (where it's useful), keeping to a distinct pod makes workload management/service definition easier.

from egeria-docs.

See also odpi/egeria#6340

from egeria-docs.

Connectors can now be loaded via URL in our helm charts.
The docs should be updated to capturing this - moving to docs repo

from egeria-docs.

Note the PRs that implemented this change are odpi/egeria-charts#181 & odpi/egeria-charts#184

An example fragment is

cat ~/etc/pg.yaml
extralibs:
  - url: https://search.maven.org/remotecontent?filepath=org/odpi/egeria/egeria-connector-xtdb/3.10/egeria-connector-xtdb-3.10-jar-with-dependencies.jar
    filename: egeria-connector-xtdb-3.10-jar-with-dependencies.jar
  - url: https://oss.sonatype.org/service/local/repositories/snapshots/content/org/odpi/egeria/egeria-connector-postgres/3.11-SNAPSHOT/egeria-connector-postgres-3.11-20220826.091931-1.jar
    #- url: ftp://download:[email protected]:49536/egeria-connector-postgres-3.11-SNAPSHOT.jar
    filename: egeria-connector-postgres.jar
  - url: https://jdbc.postgresql.org/download/postgresql-42.5.0.jar
    filename: postgresql.jar

from egeria-docs.