submariner-io / enhancements Goto Github PK
View Code? Open in Web Editor NEWEnhancement proposals for Submariner projects.
Home Page: https://submariner.io/
License: Apache License 2.0
Enhancement proposals for Submariner projects.
Home Page: https://submariner.io/
License: Apache License 2.0
We would like to create tooling that allows running multiple kind clusters in scale testing on one or more machines, and run scale testing to identify any issues.
Tasks:
make scale-test
Stretch goals:
This tracker epic will also be updated with any issues identified during the scale & longevity testing
OVN-Kubernetes is one of the two CNI's supported by RedHat Openshift, and will soon become the sole CNI for the Kubernetes distribution. Submariner supports the CNI via the OVN-Kubernetes specific NetworkPluginSyncer, however the design is more of a POC and currently non-functional with recent OVN-K versions. Many more steps need to be taken in order to ensure the integration is bug-free and ready for production environments, this epic will track the work that is needed to get the integration to such a state.
Submariner's ability to connect clusters utilizing the OVN-Kubernetes CNI is 100% functional with current OVN-K builds, and the correct CI infrastructure has been put into place in order to ensure the continued functionality of the integration.
What would you like to be added:
Proposal is to fully support connecting multiple OCP clusters on VMWare using Submariner.
Why is this needed:
Aim of this proposal is to list and track steps needed to fill any gaps and allow full support for such deployments.
UPDATE
Investigations have confirmed that there is nothing that needs to be done to prepare cloud for VM Ware, in subctl or OCM. Expectation is that VMWare should work without any changes, except when using NSX which is for now outside scope of this EPIC.
Work Items
Based on results of investigations, only effort required is testing.
We want to enable certificate rotation, especially for accessing the broker. Rotated certificates will need to be propagated somehow, or we’ll need to find some other (secure) way of authenticating and authorising access to the broker. This will have to include broker-info.subm
files — we want subctl join
to be possible throughout the lifetime of a broker.
subctl join
continues working with broker-info.subm
files produced during initial broker deployment (but not necessarily with an older version of subctl
); any additional requirements are documentedCurrently, ACM/OCM only supports deploying Vanilla Submariner, and it's not possible to deploy Globalnet.
This EPIC is to track the individual tasks associated to implement Globalnet in ACM.
Along with this, we also want to address the remaining bits from the Globalnet 2.0 feature, primarily the metrics support and removing the dependency with the kube-proxy driver.
One should be able to deploy Globalnet with ACM with all the e2e tests passing.
The following are the individual work items for this feature.
Similar to how Kubernetes uses "KEP-####", we should give unique IDs to enhancement proposals so we can easily reference them elsewhere.
It would be obvious to use "SEP-####", and that's okay with me, but it does unfortunately also sometimes stand for "Somebody Else's Problem". 😅
Periodic link aliveness CI detected a broken link. Please see the periodic job
results for details.
Support connecting multiple OCP clusters on ROSA using Submariner.
Why is this needed:
The aim of this proposal is to list and track steps needed to fill any gaps and allow full support for such deployments.
Should be able to prepare and deploy submariner in ROSA using subctl and cloud-prepare.
We need a service dedicated to syncing the broker information from the local cluster to the broker, and the broker info to the local cluster.
We have seen on scale testing that having the broker syncer inside the gateway causes issues to connectivity when there's a high enough scale.
Additionally, multiple projects (Submariner, Lighthouse) are using a specific broker syncer to sync over their resources.
From an architectural POV it would be simpler if there was a "broker syncer" service which can sync resources to/from the remote broker, and the local services would only consume local resources.
This decentralizes the information further, and decouples the services allowing the local services to easily keep working even if the broker is temporarily down.
In addition, this opens up possibilities of easy broker HA since the data can be easily replicated.
This can also benefit Axon as it decouples the control plane information dissemination from Submariner, allowing alternative methods to be used.
submariner-brokersyncer
repository in Quay.iosubmariner-brokersyncer
in submariner
repository (Would admiral
be a better place?)subctl diagnose
subctl gather
subctl uninstall
submariner
lighthouse
submariner-brokersyncer
image to releases
processWe want to provide out-of-the-box support for ARM64. Submariner should just work on ARM64 nodes, with no special configuration by the administrator or end-user.
Depends on submariner-io/submariner#1772
Depends on submariner-io/lighthouse#694
Depends on submariner-io/submariner-operator#2021
Depends on submariner-io/subctl#66
The Submariner operator currently has wide-ranging privileges. It doesn’t need to be able to access anything outside the namespaces it manages, so this should be reduced. See https://hackmd.io/wVfLKpxtSN-P0n07Kx4J8Q for background.
Depends on submariner-io/submariner-operator#1105
RBAC generation will affect this, we should wait to have a better idea of that before starting work on designing this.
The operator is de-scoped, ideally with no ClusterRole, at minimum with justifications for every permission in its ClusterRole.
See also submariner-io/submariner-operator#1105 which overlaps with this; auto-generation should be used if the SDK supports it for namespace-scoped Roles.
subctl diagnose
is a CLI tool used to troubleshoot submariner deployments. But no such tools are available from the OCM UI. Add option in OCM UI to run diagnostics.
Refer submariner-io/submariner-operator#88
TBD
What would you like to be added:
We need to either find or develop some tools to make actions across multiple repos in our org:
Tooling could live under releases
repo
Why is this needed:
We often need to do these manually which is a bit error prone and time consuming, would be better to have some tooling for this.
The current implementation of Globalnet Controller works with iptables based kubeproxy. We have to add support for clusters that use OVN as the network-plugin driver.
The Globalnet feature functions correctly when using Submariner with the OVN-Kubernetes CNI.
The goal of this epic is to implement ServiceImport merging.
Once an mcs-controller has been introduced, the reconciliation logic will be extended to address
the following open issues:
ServiceExport
objects with the ServiceExportConflict
conditionServiceExport
objects into a single ServiceImport
objectRelated enhancement proposal: #111
Lighthouse behaviour aligns with MCS specification:
This is a proposal to introduce policy CRDs to filter the ServiceImports
that get added to
each cluster. The policy would primarily make it possible to limit what gets imported to a
cluster:
The policy would additionally support namespace mapping so that ServiceImports
can be added
into a different namepsace from the originating namespace in the event that there is a namespace
conflict between exporting and importing clusters.
This would avoid the necessity of port mapping in a great deal of cases:
https://en.wikipedia.org/wiki/UDP_hole_punching
Once the UDP hole is punched (if the router would allow that, of course),
then we can advertise that port as the port to be used for NATT discovery or UDP traffic.
subctl, soon to be a library itself, should be moved out of submariner-operator repo to its own repo. Everything specific to the subctl is moved to the subctl repo. Operator repo depends on the subctl repo. Operator repo should contain things that the operator framework generates/ needs. This is an alternative to the previous proposal of moving bundles out of the operator repo.
Pros:
go install
-ed currently because the operator dependencies don't allow it.Cons:
All of the subctl related code/GHAs are out of operator repo and in the new subctl repo.
Depends on #114
subctl
repository and copy all the contents from submariner-operator
repository.subctl
repositorysubmariner-operator
repositoryDepends on submariner-io/submariner-operator#2030
get.submariner.io
repositorysubmariner-website
repositorysubctl
dependency in submariner-addon
.Currently, E2E tests run the full test suite, even when tests are failing. The failures often show up as time-outs, and a failing test suite thus takes a very long time; but in most cases, we’re only interested in whether the tests pass or fail, not the details of the failure (after the first one). This is particularly visible with flaky tests and causes significant delays in the release process.
We should add an option to abort on the first failure, and enable it when run from GHAs.
This would involve:
make e2e
runs all tests or aborts on the first failure (at the Ginkgo level); this should default to run all tests, to preserve existing behaviour for local invocations.make e2e
with the setting to abort on first failure; this might involve changes in all projects.This shouldn’t affect the GHA test matrices at all — we still want to run E2E in all desired configurations, the only change is that within each run, the first failure should abort that run.
Add support to automatically "unjoin" a previously connected cluster by deleting/uninstalling all the Submariner-related components and removing dataplane rules programmed on the route agent and gateway nodes.
This is a common request from users. A manual process is documented here.
The uninstall process should be able to be initiated via a subctl
command and also via the Submariner ACM add-on.
For dataplane cleanup, this blog describes how to run a one-time task on each node using a DaemonSet
with an init container to run the task. We can create cleanup DaemonSets
corresponding to each component (route agent, gateway engine etc). We can reuse the same images and pass an UNINSTALL environment variable to initiate the cleanup.
The tricky part is how to invoke the cleanup DaemonSets
. The logical place would be from the Operator
controller. We could put a finalizer on the Submariner
resource and, when it's being deleted, delete the normal component DaemonSets
and create the cleanup DaemonSets
. After the cleanup DaemonSets
have completed (ie are ready), remove the finalizer and let deletion of the Submariner
resource proceed.
If cleanup fails for a component, the DaemonSet
init container pod would return a non-zero exit code causing K8s to restart the pod. The Operator
controller can wait a certain amount of time, like 2 min, for a cleanup DaemonSet
to be ready. After that time it’s reasonable to assume the error isn’t transient so the Operator
controller can remove the finalizer and let deletion of the Submariner
resource proceed. It could leave the cleanup DaemonSet
so the user can inspect the DaemonSet
status to see what failed. Or it could delete the DaemonSet
and we could create a K8s Event
to report the failure, either from the Operator
or the failing pod. In subctl
, after Submariner
is deleted, we can report the failure to the user. Any cleanup DaemonSet
that failed will eventually get deleted along with the namespace regardless.
The one potential downside with this approach is if the Operator
pod is deleted first and thus the controller is never triggered. This would also leave the finalizer and prevent the Submariner
resource from being deleted. But we can document the proper sequence.
Invoking the uninstall process should remove all Submariner-related components and artifacts in the cluster.
Event
if an uninstall pod failsEvents
in submariner NS and copy to the event recorderSubmariner currently assumes full mesh by default, and provision VPN connections between all participating clusters. For some use cases, it makes sense to offer finer-grained connectivity options. This could also improve overall scalability/state maintained, for those cases in which full mesh is not required.
Ovn-kubernetes is not so convenient to use. Please consider supporting Kube-OVN.
Submariner currently offers three cable drivers: VXLAN, IPSec or wireguard. However only one of these cable drivers
is enabled across all the clusters in a ClusterSet; That is all clusters that join a broker must use the same cable
driver. The goal of this proposal is to extend the current model to support the selection of different cable types
between connected clusters. For example, an IPSec cable between public and private clouds but a VXLAN cable between
the clusters in the same cloud.
More details can be found here: #110
Multiple cables can be deployed with Submariner Gateways
The various subctl
commands support a mixture of KUBECONFIG
environment variables, --kubeconfig
flags, --kubecontext
flags, kubeconfig files etc.
We should make this consistent, ideally using clientcmd
’s tools to handle kubeconfig. skitt/submariner-operator#23 has an old attempt at this, using https://pkg.go.dev/k8s.io/[email protected]/tools/clientcmd#BindOverrideFlags to provide access to all the features supported by clientcmd
(see kubectl options
).
For commands using a single context, all the following should work, with kubeconfigs containing one or more clusters and/or contexts (this uses a kube
prefix when setting up clientcmd
):
# Use the default context in the kubeconfig
KUBECONFIG=/path/to/kubeconfig subctl foo
subctl foo --kubeconfig /path/to/kubeconfig
subctl --kubeconfig /path/to/kubeconfig foo
# Use the specified context from the kubeconfig
KUBECONFIG=/path/to/kubeconfig subctl foo --kubecontext bar
subctl foo --kubeconfig /path/to/kubeconfig --kubecontext bar
subctl --kubeconfig /path/to/kubeconfig --kubecontext bar foo
For commands using multiple contexts, the prefixes should make the purpose of each context clear, using appropriate prefixes when setting up clientcmd
:
KUBECONFIG=/path/to/kubeconfig subctl benchmark latency --fromcontext foo --tocontext bar
subctl benchmark latency --kubeconfig /path/to/kubeconfig --fromcontext foo --tocontext bar
--kubeconfig
is preserved as-is, since users can use a single file combining all their contexts. However such commands should also support separate kubeconfigs:
subctl benchmark latency --fromconfig /path/to/fromconfig --toconfig /path/to/toconfig
clientcmd
doesn’t handle --kubeconfig
itself, that needs to be handled by subctl
.
This enhancement should also include a test suite covering all the different possibilities. As far as possible, the 0.12 options should keep their existing behaviour, and be marked for deprecation; they will be removed after 0.13.
This would subsume submariner-io/subctl#49
name: Debuggability and Troubleshooting
about: Improving debuggability and troubleshooting for submariner deplyments in general, OCM in particular
labels: epic
Current documentation on Sumariner.io - Troubleshooting docs covers the basic manual steps and how to run subctl diagnose and gather. What more needs to be done? Aim is to make it easy for support teams to troubleshoot and resolve the problems, or capture relevant information if unable to resolve.
TBD
subctl
code to reuse gather
and diagnose
like a library
must-gather
- Add gather
to OCP must-gathersubctl diagnose
to CI/CD in case of any test failure
subctl gather
in CI/CD post-mortem in case of any test failureFull node mesh can be support with the wireguard VPN implementation.
Full node mesh adds the following:
Support connecting multiple OCP clusters on RHOS using Submariner.
Why is this needed:
The aim of this proposal is to list and track steps needed to fill any gaps and allow full support for such deployments.
Should be able to prepare and deploy submariner in RHOS using subctl and cloud-prepare.
Using a subnet router would theoretically make it possible:
https://tailscale.com/kb/1019/subnets/
It's a managed wireguard on steroids, which would allow connecting also to existing legacy workloads
with ease.
It supports HTTP relay via their cloud servers for cases where NAT traversal is impossible, as well as
NAT hole punching techniques, etc...
See [1] for reference.
Broker Endpoints and cluster could expire if the timestamp is not updated in T, while submariner
would update the objects every T/2 time.
Alternatively, instead of auto-cleanup, this could be an admin triggered action on the broker, to avoid
for example downtime to cluster who can't contact the broker temporarily.
Cleanup of local copies should not happen if the broker is unreachable to avoid dataplane downtime
while the broker is down.
[1] https://github.com/submariner-io/submariner/blob/devel/pkg/cableengine/syncer/syncer.go#L142
Submariner addon should use submariner operator to manage submariner deployment on managed clusters. This allows for reusing of operator, automated management using operator and a single point of development efforts.
addon uses operator to deploy broker and submariner on hub and spokes.
What would you like to be added:
Proposal is to fully support connecting multiple OCP clusters on Azure using Submariner.
Why is this needed:
Aim of this proposal is to list and track steps needed to fill any gaps and allow full support for such deployments.
Work Items
Does subctl show command returns non-zero exit code, if there's a missing/failed resource (e.g connections, networks, gateways, etc.) ?
If not, can you add this feature ?
subctl gather
is a CLI tool used to gather logs and other data from submariner deployments. But no such tools are available from the OCM UI. Add option in OCM UI to run gather logs.
Refer #88
TBD
subctl gather
Job on the clusterPeriodic link aliveness CI detected a broken link. Please see the periodic job
results for details.
The goal of this Epic is to enable a number of new features for Submariner Cable Drivers:
additional cable driver encapsulation protocols, including: IP in IP and GRE. Enabling these encapsulation protocols could allow for performance improvements over the current configuration.
Enabling IPSec transport mode - coupled with VXLAN, GRE or IP in IP.
It should be possible to do finer grained configurations for any of these cable drivers, for example in the case of IPSec, selecting the encryption algorithm, or for VXLAN selecting the UDP port (separately to the NAT port)...
New Cable Drivers are functional and deployable with the Submariner Gateways.
Submariner load-balancer support
Currently, the load-balancer mode is an experimental feature in Submariner and is not well tested across various public and onPrem clusters. This EPIC is to understand the load-balancer implementations across AWS and GCP clouds to start with, finding any gaps and also its implications on HA-failover times.
Identify and where applicable fix the missing gaps in order to make load balancer a supported feature.
The following are the individual work items for this feature.
This will ensure that submariner-operator repo structure follows go standards and is optimised.
submariner-operator code is as much optimised and modular as possible.
What would you like to be added:
The reporter API is generic so move it to admiral.
Why is this needed:
This will allow the API to be used by other projects, specifically in cloud-prepare
to replace a similar API.
What would you like to be added:
Proposal is to add support for VPC Peering between clusters deployed with same cloud provider.
Why is this needed:
When all cluster are deployed within the same cloud provider, VPC Peering provides secure networking between them. In such scenario one can use VxLan Cable driver for better performance without the IPSec overhead.
Work Items
What would you like to be added:
Proposal is to fully support connecting multiple OCP clusters on IBM Cloud using Submariner.
Why is this needed:
Aim of this proposal is to list and track steps needed to fill any gaps and allow full support for such deployments.
Work Items
multiple active gateways can increase the bandwidth. Some cloud provider have limited instance-to-instance bandwidth between regions, so this feature is an enabler for all throughput-sensitive workloads.
Enhancement document: #69
Improve current troubleshooting documentation on Sumariner.io - Troubleshooting docs
Aim is to make it easy for support teams to troubleshoot and resolve the problems, or capture relevant information if unable to resolve.
subctl diagnose
diagnose
outputsubctl gather
optionsdiagnose
diagnose
gather
and diagnose
subctl
- Handle multiple kubeconfigs/contexts better, avoid gathering information from same cluster repeatedlysubctl diagnose
from a podsubctl diagnose
should be capture in CRsubctl diagnose
running resultsdiagnose
- Detect errors in cloud deploymentdiagnose
- Detect service discovery issuesdiagnose
- Detect cabledriver issuesgather
- Gather logs from submariner-addon
if OCM deploymentgather
- Gather information from cloud e.g. ports opened etc.subctl diagnose
to CI/CD in case of any test failuresubctl gather
in CI/CD post-mortem in case of any test failurePeriodic link aliveness CI detected a broken link. Please see the periodic job
results for details.
https://github.com/cloudflare/boringtun It's a rust based, very performant implementation of wireguard.
https://github.com/WireGuard/wireguard-go is a go-based inplementation (performance to be checked)
The operator was built using an old version of the Operator SDK. This causes a number of problems:
We need to upgrade our operator to use the current Operator SDK, following the migration guide.
Depends on #89
stolostron/submariner-addon#318
submariner-io/submariner-operator#1105
submariner-io/submariner-operator#1415
submariner-io/submariner-operator#1824
The operator’s Operator framework dependencies are up-to-date, and all tests pass.
An operator was written from scratch following the official operator tutorial. The code was pushed on a fork as a PR. This PR was then used as a reference to make required changes.
The operator-sdk version is upgraded to 1.23.0. Migration guide was followed to pin versions for dependencies.
Currently OCM UI doesn't show connection status in its own UI. It shows up as part of ManagedClusterAddOn status. It is not very intuitive for user to know which cluster connections are in connecting or error states. With topology view available in OCM 2.6, it should be used to display multicluster connection information to user.
Refer #88
This requires topology view to be available in OCM UI. Also requires some design discussions with UI team on how to model ConnectionStatus CRD so it can be used by UI in a scaleable way.
Prepare to apply for CNCF Incubation. Use CNCF Graduation and CII Silver/Gold as guidance for exceeding Incubation minimums and preparing for Graduating in less than two years from Incubating (per requirements).
What would you like to be added:
Support for different cable drivers inside the same supercluster.
Why is this needed:
Once we start supporting cable drivers with meaningfully different features (sorry if I’m insulting any existing cable driver), e.g. submariner-io/submariner#674, various scenarios involve mixing cable drivers. For example, a supercluster could contain three clusters, one of which should only be connected to using encrypted connections, but two of which may be connected to using unencrypted connections (because they’re connected by a private link). So we’d have three connections, one unencrypted and two encrypted, presumably using different cable drivers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.