airshipit / airshipui Goto Github PK

● Design and justification - airshipui running as a long running container
● Design and justification - Should it support multiple concurrent users

Related to #57

As part of the end to end lifecycle management, provide users the ability to execute baremetal & image build operations through Airship UI.
This will likely be dependent on airshipctl issues: #246, #248

As a operator I need to ensure that any user is authenticated before he can access the Airshipui to access a particular Cluster and is restricted to do operations restricted to his role.

So a typical use case would be a tenant user authenticates to the airshipui and depending on his roles he is able to select only specific context (in the airshipui as dropdown list) and deploy or read artifacts restricted to those specific Cluster and nodes.

This also means there should be a mechanism to integrate the airshipui with the backend via some mechanism like Oath/Dex.

Need the ability to navigate through graphical representation of the topology expressed in the YAML files. This UI should represent the relationships that exist between the various components of the topology, and should offer some way of visualizing the customization and layers that are present therein.
Depends on #9

Description

As a consumer of the AirshipUI, I want it to be launchable as a single executable called airshipui that starts up the Octant UI with all of the Airship specific plugins enabled. This should be done in such a way that subcommands could be added in the future airshipui <foo> or airshipui help. The default behavior for airshipui should be to launch the Octant UI with the airship content loaded

Acceptance Criteria

• The build for the airshipui repo creates an “airshipui” Go binary that launches the Octant UI
• the airshipui binary should support subcommands (such as help or version as a PoC)
• airshipui run without and sub-command should launch the Octant UI. The Airship content may exist in separate plugins (which are separate) that may reside in a separate location
• CI system should validate that the airshipui binary is created, and ideally should do a headless browser run to validate that it launches

This issue was migrated from AIR-172

Airshipui should have a mechanism to initiate co-located dashboards. I.E. Initiate dashboard process if, identified in the list of plugins, and proper information is found.

AirshipUI needs to integrate with airshipctl and its config structure.

A Kubernetes Etcd plugin on the UI that shall enable Operations to debug the environment by enabling user to get the value of a key in the cluster. This Plugin provide equivalency to what can be done via teh etcdctl command line tool.

• etcdctl get [options] [range_end]
• etcdctl watch [options] [key or prefix] [range_end] [--] [exec-command arg1 arg2 ...]
• etcdctl lease list
• etcdctl member list
• etcdctl endpoint health
• etcdctl endpoint status
• etcdctl alarm list
• etcdctl role get
• etcdctl role list
• etcdctl user get [options]
• etcdctl user list
• etcdctl version

Plugin logging seems to be swallowed by octant or otherwise not making it out of the plugins for console logging. We need to figure out:

1. If octant is swallowing the logs
   a. How to get octant to not swallow the logs
   b. Or an alternative path to logging
2. If octant is not swallowing the logs
   a. What is the best approach to logging for the plugins

May be related to #18, #19, #20, and #21 .

Milestone: alphav1

We should have the ability to explore baremetal host definitions, add, and edit them directly in the Airship UI - namely BareMetalHost and Machine definitions (and possibly others). This should keep close to our philosophy that the YAML maintains our declared state so this simply operates on local YAML for the current site context with potentially some ability to incorporate some live state from the site by optionally reading existing CRDs in the site to enhance the editing process. It would require pushing/submitting that YAML to the site to effect any changes (through the standard flow).

See https://github.com/openshift-metal3/facet

This might be construed as a generic way of delivering CR’s to a site? Perhaps this is about integrating with airshipctl workflow?

We already have the normal CR visibility out of the box behavior from Octant

This issue was migrated from AIR-18

Develop a base Octant , tekton like interactive CR plugin , that will serve as the basis for all other Airship interactive CR capabilities.

Leverage Electron's ability to integrate with existing product dashboards to provide support for Logging, Monitoring & Alerting. Users will be able to launch these dashboards from AirshipUI. This would include such elements as Grafana, ElasticSearch, FluentD, etc. (each may get its own issue).

As a user I need to filter the portion of the topology that is currently being displayed, restricting it to a specific site; the “slice” of data shown in the topology should then be restricted to that particular site

Prerequisite: #13

Migrated from AIR-223

As a operator I should have the ability to execute the performance test against a deployed Target Cluster and compare the results.
The following benchmark test can be considered as scope with ability to add more as required :

1. Common set: CPU, Disk, Network and Application performance
2. Advanced set : Affinity, Anti-Affinity, Tolerations, Storage Classes , Node Selection,etc
3. External system API tests (like external Kafka system, Ticketing systems, etc)

Reference Links :
https://github.com/xridge/kubestone
https://github.com/kubernetes/perf-tests/tree/master/perfdash
https://github.com/vmware/weathervane

As database components become more align with operator framework, teh notion of expressing its configuration and intentions via CRD / CR will become the norm.

A Plugin or enhancements to this database CR, to introduce Database health management operationalizing is what this epic introduces.

Ability to interact with the database to :

• Shutdown
• Verify connectivity from a client point of view
• Database cli interactions

Specifically common databases in the infrastructure such as :

• mysqlDb
• postgreDb
• mariaDb

The ability of a plugin to do this, might of course be impacted by the external exposure of the database.

As a prerequisite for developing the YAML explorer, we have to understand what that set of topology yaml documents looks like and how they will be inter-related. This task is really not specific to airshipui, and arguably should be a task for the airshipctl project.

It would be preferable to use GraphViz or some similar tool that permits the source for the diagram to be checked into the airshipui repo

It might be necessary to also consult the airship configuration in order to determine the parent directory in which the individual files are stored

Some useful links:

• sig-airship-yaml notes
• treasuremap prototype with example documents and directory structure (currently devoid of content)

This was migrated from AIR-212 story and tasks.

As a user of airshipUI, I should be able to invoke the airshipui command using a superset of the command line options that are currently available on the underlying octant command. Currently the airshipui command exposes 6 command line options, two of which (--airshipconf and --kubeconfig) are implemented to integrate maximally with airshipctl. But there are about a dozen other octant options including --context, --namespace, that are not exposed.
The full list of these can be obtained by using 'octant --help'
Note that some options are described as "[DEV]" and others are not. All should be implemented, but the priority should lie with those that are not marked "[DEV]"

This was migrated from AIR-208

Users should be able to see the status of their sites, including the basics like workflow summaries, document revisions, and machine provisioning status, as well as the basic health of a site (remains to be seen whether this requires logging, monitoring, and alerting integration to report back or whether it is simpler than that).

This issue was migrated from AIR-59

Recommend we pull liberally from https://github.com/openshift-metal3/facet to build an interface to create, manage, and view baremetal machines in an environment.

This issue was migrated from AIR-56

As an operator I need to have the capability to view/modify site-level manifest so that I have a user-friendly view of rendered site documents and am able to customize site documents directly via the UI.

The operator will be able to view rendered manifests directly in the UI by phase. The rendered view would show the rendered documents with all kustomizations/replacements processed.

The operator would be allowed to make site-level changes to the rendered phase that would be written back to a new or existing site-level document for use in kustomization and/or replacement changes to rendering.

As a user I want to be able to use the UI to bootstrap a new cluster. This will use the airshipctl functionality to perform the various airshipctl bootstrap subcommands including

• generating the base configuration
• generating the bootstrap ISO
• bootstrapping the ephemeral node

This issue was migrated from AIR-224

Clearly most of the functionality or configuration of the CNI is achieved via CRD's.
Which implies we have the ability to identify NEtwork Policies etc via the standard ui functionality.

Tigera has introduced a UI component and an API Server component

This issue is the desired to either :

• Integrate the UI into Airship UI
• Integrate with the Calico API Server potentially

Expose CNI operation management from the UI.

In example do we want to observer and interact with NetworkAttached Definitions , separate from treating them like a normal CRD?.

Are there operations or audit-able functzonlality we can achieve via this UI.

Of clear interest will be troubleshooting abilities such as :
https://www.tigera.io/tigera-products/visibility-and-troubleshooting/

Argo UI is currently an iframe plugin. This has significant hurdles to being able to properly render an iframe across real or virtual network segments. It should be reworked to allow for a more portable utilization of the web standards.

Identify Openstack GO library that can be used to build Openstack Plugin

Related to #22

How do we manage configuration amongst plugin?

• Some of it might apply to multiple plugins
• Some of it might be unique .to a plugin

Need to define and design the relationship between these different configuration aspects :

• Airship config
• Kubernetes config
• Plugin Config's

Using the interactive CR framework developed in #41, provide visibility into deployment activities via AirshipUI. These may include the following:

• What has been deployed
• Software Versions
• Last update date
• Last backup date
• Who performed last update
• Expiration dates for certificates and secrets

Similar to the argo-ui plugin we would like to create a Nagios plugin to integrate its dashboard into airshipui

UI for Openstack will enable Operations to trigger a well defined command set for Compute, Network, Identity, Image, Block Storage, Queueing service APIs together from within the UI .

Some examples of things we might want to allow them to do :

• View tenant quota details
• View Images
• View Volumes
• View VM status
• Troubleshoot VM startup issues
• Troubleshoot Heat stack issues
• View Users/project

In general the ask is for a plugin that integrates AirshipUI with Openstack API's, so we can then rationalize what explicit . functionality we want to expose via it.

Stories for this Epic:

• #24

Improve the README to explain the purpose of Airship UI.

1. Explain goals
2. Include content from the Airship blog
3. Point to the developer contributing guide
4. Explain the process of a developer installation

Provide an authentication mechanism that allows users to sign on once & have access to the various integrated dashboards which can be launched from AirshipUI (e.g. LMA dashboards).

AirshipUI should be able to support one or more authentication mechanisms. These may include OAuth, basic authentication & cookie based authentication (may need to further break each one out into its own issue).

There should also be an option to not have authentication enabled.

Similar to the argo-ui plugin we would like to create a Kibana plugin to integrate its dashboard into airshipui

https://docs.google.com/document/d/11JXXNhzDXNO1mIh64glb7Cfy97_-jGDYej8QcVqQx20/edit#

Milestone: alphav1

Besides the Kubernetes storage artifacts themselves which the Overview plugin addresses for the most part. Some gaps need to be addressed.

Specifically this issue is about going beyond.
For something like CEPH how can we allow for this functionality :

Sunny day operational activities such as :

• Cluster health check
• OSD status
• Ceph Pool status
• Ceph volume size and usage data
• Ceph replication status
• Ceph PVC backup
• Provide k8s PVC mapping with Ceph volume

Octant allows for the starting of the binary without popping the browser:

$ octant --disable-open-browser
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "deployment/configuration", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/containerEditor", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/serviceEditor", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/startPortForward", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/stopPortForward", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/commandExec", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/deleteTerminal", "module-name": "overview"}
2020-01-29T21:14:16.172Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "octant/deleteObject", "module-name": "configuration"}
2020-01-29T21:14:16.198Z        INFO    plugin/manager.go:397   registered plugin "argo-ui"     {"plugin-name": "argoui", "cmd": "/root/.config/octant/plugins/argoui", "metadata": {"Name":"argo-ui","Description":"Argo UI version e93fe60","Capabilities":{"IsModule":true}}}
2020-01-29T21:14:16.198Z        INFO    plugin/manager.go:405   plugin supports navigation      {"plugin-name": "argoui"}
2020-01-29T21:14:16.198Z        INFO    dash/dash.go:391        Using embedded Octant frontend
2020-01-29T21:14:16.204Z        INFO    dash/dash.go:370        Dashboard is available at http://127.0.0.1:7777

Airship UI does not which results in an error showing when you have to tunnel the connection over ssh for the airshipui:

$ bin/airshipui
2020-01-29T21:15:44.246Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/deleteTerminal", "module-name": "overview"}
2020-01-29T21:15:44.246Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "deployment/configuration", "module-name": "overview"}
2020-01-29T21:15:44.246Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/containerEditor", "module-name": "overview"}
2020-01-29T21:15:44.246Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/serviceEditor", "module-name": "overview"}
2020-01-29T21:15:44.247Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/startPortForward", "module-name": "overview"}
2020-01-29T21:15:44.247Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/stopPortForward", "module-name": "overview"}
2020-01-29T21:15:44.247Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "overview/commandExec", "module-name": "overview"}
2020-01-29T21:15:44.247Z        INFO    module/manager.go:79    registering action      {"component": "module-manager", "actionPath": "octant/deleteObject", "module-name": "configuration"}
2020-01-29T21:15:44.275Z        INFO    plugin/manager.go:397   registered plugin "argo-ui"     {"plugin-name": "argoui", "cmd": "/root/.config/octant/plugins/argoui", "metadata": {"Name":"argo-ui","Description":"Argo UI version e93fe60","Capabilities":{"IsModule":true}}}
2020-01-29T21:15:44.275Z        INFO    plugin/manager.go:405   plugin supports navigation      {"plugin-name": "argoui"}
2020-01-29T21:15:44.276Z        INFO    dash/dash.go:391        Using embedded Octant frontend
2020-01-29T21:15:44.284Z        INFO    dash/dash.go:370        Dashboard is available at http://127.0.0.1:7777

2020-01-29T21:15:44.284Z        WARN    dash/dash.go:374        unable to open browser: exec: "xdg-open": executable file not found in $PATH
github.com/vmware-tanzu/octant/pkg/dash.(*dash).Run
        /go/src/github.com/vmware-tanzu/octant/pkg/dash/dash.go:374
github.com/vmware-tanzu/octant/pkg/dash.Run.func1
        /go/src/github.com/vmware-tanzu/octant/pkg/dash/dash.go:203

Using the interactive CR framework developed in #41, provide visibility into deployment status via AirshipUI. These may include the following:

• Current deployment step
  -- What command is running
  -- Step details
  -- Log access, etc.
• Percent complete (based on time and steps remaining)

Users should have the ability to execute the various phase commands via AirshipUI.

The current command set includes: phase apply & phase render.

Similar to the argo-ui plugin we would like to create a Grafana plugin to integrate its dashboard into airshipui

As a user of airshipUI, I need a way to obtain the binary. This epic will have stories to ensure the multiple mechanisms to deliver airshipui as a binary are in place.
A Developer that takes on these stories can create the individual stories and their details.

• Story for including documentation
• Story for delivery of airshipui binary as a URl such as curl -LO https://.../airship/airshipui/release/release/v2.0/bin/linux/amd64/airshipui
• Story for delivery of arshipui as a a debian pkg : sudo apt-get install airshipui
• Story for CI/CD changes to build appropriate binary and or packages.

This is the airshipUI equivalent of AIR-202

The original though of taking kustomize render documents, and producing a topology is not feasible without extending some of the core kustomize notions.

These Epic proposes an alternative Topology Manager notion .

There will be 2 Topology that will live side by side.

• Topology 1 - _Airshipctl Phase Bundle Kubernetes View _
  will represent the document set associated with an airshipctl phase, and represent them as a Kubernetes artifact topology.
  Cluster --> HAs Namespaces --> Have Kind --> Have Groups--> have an Artifact

• Topology 2 - Airshipctl Phase Bundle Source Yaml View _
  represent the document structure obtained by navigating through the resources defined by the airshipctl phase entry point kustomization.yaml.

As an AirshipUI developer, I want a basic airship branding style to be part of the airship plugin that I can override in my own corporate packaging if desired. This should include putting the airship logo and title in the UI and at least a prototype of a different color scheme (the default can be the baseline Octant branding)

Acceptance Criteria

• When run with the airship plugin , the Octant UI has an an Airship logo and title
• scss or other style files necessary to provide a basic branding scheme are in place in the airshipui repo, possibly with the styles commented out to maintain the default color scheme by default
• Stretch Goal: Support the disablement of out of the box Octant features. Not necessary today but in the future, Octant may add so much functionality that we drown out what we’re trying to provide users with the AirshipUI.
• CI should be updated to do basic validation of the airship branding

This issue was migrated from AIR-175

I75d55dc9842ca03453be28d0b6bab9b7dfc63549 made a change to use Client.List to retrieve the argo-ui endpoint to avoid a grpc server error. With the closing of vmware-archive/octant#540 due to vmware-archive/octant#643 we can make that change back and simplify that code.

attempting to run the default build on windows does not work correctly:

C:\aic\airshipui>bin\airshipui help
'bin\airshipui' is not recognized as an internal or external command,
operable program or batch file.

If I use cygwin I can execute the airshipui binary but it does not find the kube config even when specified on the command line:

$ bin/airshipui --kubeconfig /cygdrive/c/Users/as3525/kube/config
2020-02-11T09:01:08.634-0600    ERROR   commands/dash.go:109    dashboard failed        {"err": "failed to init cluster client: invalid configuration: no configuration has been provided"}
github.com/vmware-tanzu/octant/internal/commands.newOctantCmd.func1.2
        C:/aic/octant/internal/commands/dash.go:109

Ideally this would be cross compiled so that it is nativly executable on windows much in the same way that octant is.

This plugin shall allow access to services running on each compute node. Operator personnel should be able to get the appropriate data from a host by specifying the node and respective service command within the local cluster.

This plugin shall allow access to following service utilities on the hosts , but potentially this framework or plugin should be extendable to add other functionality.

• ovs-client
• libvirt-client
• ipmi-client
• perccli-client
• numa-client
• sos-client

Operators using these plugin should be able to :

Support Sunny day scenario such as :

• view running VM on specific node
• view iDRAC IPs
• view numa settings
• view OVS settings on host

Support Rainy day scenario

• Generate SOS report
• perform troubleshooting on tenant VM using virsh
• perform troubleshooting on OVS running on compute

As a tenant-admin (and other tenant users) I should have the ability to manage the tenant resources :

• Manage the tenant users, roles, rolebindings
• Manage the tenant namespace resources - ResourceQuota, IPPools,etc

As an operator using the Airship UI, I would like a view that shows a tree of YAML documents associated to a phase so that I can get a complete view of the YAML document types, composites, and functions, etc. associated to the phase. This view would be an entry point into doing a direct edit to the YAML files.

This view would provide visibility into the kustomize configuration for a specified phase by traversing the kustomize structure. The tree traversal should probably be provided by the airshipctl module.

This view could also provide viewing into the image builder configuration for the site depending upon how that is implemented in the image builder.

Document files can be edited directly on the file system and then pushed to a repo by the user.

With the merging of vmware-archive/octant#645 there is now a possibility of testing the argoui plugin route handler function which was removed in e384d17.

A sample plugin with tests was created as a part of the issue above which can be drawn from.