redhat-cop / openshift-toolkit Goto Github PK

View Code? Open in Web Editor NEW

232.0 47.0 150.0 650 KB

A collection of code samples to help you get started with OpenShift

License: Apache License 2.0

Python 65.54% Shell 33.91% Dockerfile 0.55%

container-cop

openshift-toolkit's Introduction

The OpenShift Toolkit

A collection of code samples to help you get started with OpenShift.

Solutions

Custom Autoscaler
Syncing a Registry for Bootstrapping Disconnected Installs
Expanded Health Checks for EFK and Hawkular
Quota Management - A sample strategy for managing Quotas
NetworkPolicy Management - A sample strategy for managing Network Policy
Pre and Post install OpenShift validation - Automation to ensure clusters are ready for use.
Sample Install Inventories
Capacity Planning Tooling - A Dashboard to help you make capacity decisions.
Software Delivery Metrics Dashboard - A Dashboard to Measure the Flow of Software Features through a Value Stream
OpenShift Applier - An Automation Framework for Kubernetes
Containers Quickstarts - A set of quickstarts to help you get started building and deploying software on OpenShift
Container Pipelines - A set of sample CI/CD Piplines for OpenShift

Ansible Playbooks

openshift-toolkit's People

Contributors

Stargazers

Watchers

Forkers

etsauer makentenza mmckinst themoosman sabre1041 samleonard93 lpsantil canit00 liornabat saurabhtandon13 orgadk codrinbucur genadipost benxar oats87 warmchang profisien duritong flydragonle jooho ironicbadger pabrahamsson bbeaudoin jaredburck maniacs-ops zer0glitch bevinhex kanedafromparisfriends christianh814 pallav8990 startxfr zeldi fabiomartinelli therevoman b43646 muheric grimkroton glowdb keerthivel28 jchraibi aland-zhang jtudelag npoyant felipetux prasenforu georgegoh pk-vungle rgupta1234 dzungdo huddlesj maci0 dizzythermal bumplzz69 pichuang klaas- dalitun dineshtripathi30 uasau midora arylwen ptanloc afcollins cdieng aizuddin85 harddian trent-melcher mwitzenm faridsaad ngaacb pekramp epasham shah-zobair talkedkestrel adlerfleurant tomgeorge clutchbeyers ianwatsonrh jkupferer jmanuelcalvo pittar deepakvpatil fabiovolpini raffaelespazzoli binnyrs pytoshka manngupta cheungkaho jimdillon ldsanches deskoh jkersbergen-redhat ramius345 sadpdtchr surajnayak88 bszeti princesayn john2912 tosin2013 oybed alimuratunsal

openshift-toolkit's Issues

Update NetworkPolicy solution to work with Applier, and provide examples for enabling service to service traffic

@sabre1041 @oybed @raffaelespazzoli since we are investing more heavily in adopting Network Policy, I think its a good time to do a refresh on this solution.

Is this toolkit applicable to OCP3.10?

For https://github.com/redhat-cop/openshift-toolkit/tree/master/networkpolicy, as most OCP components are containerized, such as kube-system, openshift-node and openshift-sdn, is the basline networkpolicy applicable still?

Add a stack for building custom dashboards

we need this desperately.

docker-registry-sync.py does not support registry.redhat.io

We would like the disconnected registry to pull from the new registry.redhat.io site, appear the python script line 68 has hardcoded registry.access.redhat.com, so we changed it to registry.redhat.io as seen below, but now we get the error below.

Code snippet from docker-registry-sync.py

 67         for image in config_file_dict[dictionary_key][namespace]:
 68             docker_json_link = "https://registry.redhat.io/v2/%s/%s/tags/list" % (namespace, image)
 69             list_to_populate.append(docker_json_link)
 70

We successfully logged into registry.redhat.io using our Service Account

docker login -u='#####|username' -p THATS_A_REALLY_LONG_HASH  registry.redhat.io

And ran the script...

./docker-registry-sync.py --from=registry.redhat.io \
--to=myregistry.myoffice.com:5000 \
--file=./docker_tags.json \
--openshift-version=3.11

Error after shell script ran

Traceback (most recent call last):
  File "./docker-registry-sync.py", line 179, in <module>
    get_latest_tag_from_api(retrieve_v_tags_from_redhat_list, latest_tag_list, failed_images)
  File "./docker-registry-sync.py", line 89, in get_latest_tag_from_api
    image_name = image_tag_dictionary['name']
KeyError: 'name'

fluentd base image

Create UBI fluentd base image for custom logging solutions.

need a license

Can an open source license be put on the repo?

How do I restart the service gracefully

I have deployed the OSE 3.7 based on docker container image not rpm or atomic.
I executed the playbook as per your instructions for change the branding logo.
How do I restart the service as gracefully to take this changes effort in container.

It's having 6 node setup (1 master, 2 infra, 1 nfs, 2 worker)
Thanks in advance.

Pre-reqs validation script doesn't check for `rhel-7-fast-datapath-rpms` in OCP 3.5

Per the docs: https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html#host-registration

3.5 added a new channel. we should check for that.

@stratus-ss

Possible redundancies between cluster-management/ and maintenance_cleanup/

It looks like we have an opportunity to do some refactoring and consolidation of a couple of our solutions. The cluster-management folder contains instructions and playbooks to deploy docker-gc, which I believe does some similar things to some of the cleanup scripts in the maintenance_cleanup/ folder. Additionally, I think maybe there is some opportunity to make these solutions more "OpenShift-y" by combining the capabilities of those two solutions and deploying them as ScheduledJobs rather than linux cron jobs.

@sal2fcVir @themoosman what do you think?

unable to pull / sync images for 3.10 utilzing docker-registry-sync.py

root        : ERROR    Unable to properly parse the version for image: openshift3/ose-ansible
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-cluster-capacity
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-deployer
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-docker-builder
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-docker-registry
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-egress-http-proxy
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-egress-router
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-f5-router
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-federation
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-haproxy-router
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-keepalived-ipfailover
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-pod
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-sti-builder
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/container-engine
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/node
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/openvswitch
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/logging-auth-proxy
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/logging-curator
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/logging-elasticsearch
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/logging-fluentd
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/logging-kibana
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/metrics-cassandra
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/metrics-hawkular-metrics
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/metrics-hawkular-openshift-agent
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/metrics-heapster
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-service-catalog
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/ose-ansible-service-broker
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/mediawiki-apb
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/postgresql-apb
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/registry-console
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/prometheus
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/oauth-proxy
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/prometheus-alertmanager
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : ERROR    Unable to properly parse the version for image: openshift3/prometheus-alert-buffer
root        : ERROR    Are you sure that the version exists in the RedHat registry?
root        : INFO     
root        : INFO     Total images to download: 1
root        : INFO     
root        : INFO     Downloading image 1/1
root        : INFO     Pulling registry.access.redhat.com/rhel7/etcd:3.2.22
root        : INFO     Trying to pull repository registry.access.redhat.com/rhel7/etcd ...
root        : INFO     3.2.22: Pulling from registry.access.redhat.com/rhel7/etcd
root        : INFO     92058cf44053: Pulling fs layer
root        : INFO     8d846904080d: Pulling fs layer
root        : INFO     e72f5c3da95a: Pulling fs layer
root        : INFO     8d846904080d: Verifying Checksum
root        : INFO     8d846904080d: Download complete
root        : INFO     e72f5c3da95a: Verifying Checksum
root        : INFO     e72f5c3da95a: Download complete
root        : INFO     92058cf44053: Download complete
root        : INFO     92058cf44053: Pull complete
root        : INFO     8d846904080d: Pull complete
root        : INFO     e72f5c3da95a: Pull complete
root        : INFO     Digest: sha256:c67c6b125e47a68ad12c361f137f4c314a81d64ba64486f99e690d5022dde9b3
root        : INFO     Status: Downloaded newer image for registry.access.redhat.com/rhel7/etcd:3.2.22
root        : INFO     Tagging for this registry: registry.mgt.devsecops.gov:5000
root        : INFO     Pushing into the local registry...
root        : INFO     The push refers to a repository [registry.mgt.devsecops.gov:5000/rhel7/etcd]
root        : INFO     68377f9620ff: Preparing
root        : INFO     8a325f8a1c47: Preparing
root        : INFO     20a1b4e9f75b: Preparing
root        : INFO     8a325f8a1c47: Pushed
root        : INFO     68377f9620ff: Pushed
root        : INFO     20a1b4e9f75b: Pushed
root        : INFO     3.2.22: digest: sha256:bc0c5ce2a55ef1dd7454b43d339f27aaa47f7831186a3713e4fe4e5dcb5913b4 size: 949
root        : WARNING  
root        : WARNING  35/36 failed to download: [u'openshift3/ose-ansible', u'openshift3/ose-cluster-capacity', u'openshift3/ose-deployer', u'openshift3/ose-docker-builder', u'openshift3/ose-docker-registry', u'openshift3/ose-egress-http-proxy', u'openshift3/ose-egress-router', u'openshift3/ose-f5-router', u'openshift3/ose-federation', u'openshift3/ose-haproxy-router', u'openshift3/ose-keepalived-ipfailover', u'openshift3/ose-pod', u'openshift3/ose-sti-builder', u'openshift3/ose', u'openshift3/container-engine', u'openshift3/node', u'openshift3/openvswitch', u'openshift3/logging-auth-proxy', u'openshift3/logging-curator', u'openshift3/logging-elasticsearch', u'openshift3/logging-fluentd', u'openshift3/logging-kibana', u'openshift3/metrics-cassandra', u'openshift3/metrics-hawkular-metrics', u'openshift3/metrics-hawkular-openshift-agent', u'openshift3/metrics-heapster', u'openshift3/ose-service-catalog', u'openshift3/ose-ansible-service-broker', u'openshift3/mediawiki-apb', u'openshift3/postgresql-apb', u'openshift3/registry-console', u'openshift3/prometheus', u'openshift3/oauth-proxy', u'openshift3/prometheus-alertmanager', u'openshift3/prometheus-alert-buffer']
[root@registry ~]#

Registry sync script does not work with web proxies

When attempting to run this script via an http/s proxy, the script bombs at:

./docker-registry-sync.py --from=registry.access.redhat.com --to=10.138.211.99:5000 --file=./docker_tags.json --openshift-version=3.4
Traceback (most recent call last):
  File "./docker-registry-sync.py", line 159, in <module>
    get_latest_tag_from_api(retrieve_v_tags_from_redhat_list, latest_tag_list, failed_images)
  File "./docker-registry-sync.py", line 75, in get_latest_tag_from_api
    image_tag_dictionary = json.loads(redhat_registry.read())
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

According to http://stackoverflow.com/questions/3168171/how-can-i-open-a-website-with-urllib-via-proxy-in-python , urllib.urlopen() should use proxy environment variables it finds. However this doesn't seem to be the case as:

export http_proxy=blah
export https_proxy=blah

Doesn't appear to change the behavior.

@stratus-ss @redhat-cop/cant-contain-this

Branding playbooks need to be grouped under a soluition.

@Jooho could you update the two Branding playbooks to be under a directory called branding?

Add node backup helper

As an openshift administrator, I want an executable that implements the node backup procedure as documented in the Day2 guide.
https://docs.openshift.com/container-platform/3.11/day_two_guide/environment_backup.html#backing-up-node_environment-backup

"helper" to mean either script or playbook, based on preference.

Feature request: Extend cluster_capacity.py script to show top projects or pods

Today I heard a request to show the top name spaces using the most CPU or memory. I think this could be added to the cluster_capacity.py script.

Script does not respect a "no_proxy" env in a proxied environment

If running this script behind a proxy, the image pull is running, but the image push to a local registry, which is reachable without a proxy, fails.

The push also uses the proxy, but the local registry is reachable without a proxy.

If a "no_proxy" env is set, the push should not use a proxy.

Update Cluster Validation scripts for Openshift 4

Update Cluster Validation scripts for Openshift 4.

[K8s/Compute Resources/Cluster Grafana Dashboard] CPU Quota is pulling in terminated pods in the CPU Limits count

I noticed some skewed numbers in some of our projects for the CPU Quota. The calculations are factoring in 0/1 Completed pods as well as active running, I dont think that is the intent with these numbers.

Example CPU Quota Numbers
Namespace CPU Usage CPU Reqeusts CPU Req % CPU Limitis CPU Limits %
example-project | 0.01 | 2.70 | 0.25% | 108.00 | 0.01%

Requests/Limits Per Pod
Limits:
cpu: 2
memory: 3Gi
Requests:
cpu: 50m
memory: 256Mi

There was 1 Running 1/1 Pod, the other 53 were build pods that were in 0/1 Completed state

54x2 limit per pod = 108.00 CPU limits

It seems like these numbers should filter out Completed/Failed non-running pods in a project for accurate utilization reporting??

docker-registry-sync tools don't sync hosted components images (ocp v3.6 / v3.7)

Hello,

The docker-registry-sync.py was not able to pull the host_components images listed on the docker_tags.json.

This is happen when, I tried to sync a disconnected registry for openshift-version=3.6 and 3.7 with registry.access.redhat.com.

Kindest Regards,

PS: I tested with openshift-version=3.5 and it works fine.

Enhancement ideas for post-install validation script

With #71 , we introduced an initial form of post-install validation for openshift. We also captured some ideas for future enhancements, which I will move to this issue for tracking.

Fine tune the execution through command line parameters
- Ability to skip certain tests
- Set the expected values for certain tests (ie: number of masters)
  - We can look into the addoption functionality. https://docs.pytest.org/en/latest/example/simple.html

Feature Request: docker-registry-sync.py only pulls x.y "latest" and cannot be configured for "x.y.z"

The function "get_latest_tag_from_api" will only pull, for example, the latest packages for 3.11. There are times that one environment is deployed and the current version is 3.11.69 (and that environment has its own disconnected-docker-registry), and a new environment is deployed in the future. Sometimes, new '.z' versions are released into Red Hat's container library and is newer than 3.11.69 (for example 3.11.92). It would be beneficial to be able to set a variable, via command line, to specify the full version that you'd like to download.

Perhaps both 'latest' AND the x.y.z release can both be downloaded, asynchronously, first latest, and then x.y.z.

Perhaps via this method:

Duplicate the function "get_latest_tag_from_api", and make modifications in the new function to be able to pull x.y.z (perhaps "get_my_tag_from_api"). x.y.z is defined via command line attribute, if blank/not defined, then only pull latest and do not execute the function to pull x.y.z

Master backup to follow documentation

The master backup script only archives the certificates.

Ideally, the script (or playbook) would implement the documentation backup procedure (backing up all configs, etc.):

https://docs.openshift.com/container-platform/3.11/day_two_guide/environment_backup.html#creating-master-backup_environment-backup

3.11 ocp-master-cert-backup.sh

Hi,
ocp-master-cert-backup.sh is not working on a containerized 3.11. It checks for atomic-openshift-master-api which is not a service here.

Only running service (but that does not define a master) is atomic-openshift-node.service.

I would suggest to check for /etc/origin/master/ which only exists on masters.

Greetings
Klaas

The hyperlink of "add_etcd_node.adoc" is broken.

Hi,
First, this repo is very useful for daily maintenance of the OCP.
We I read the article, found the hyperlink of "add_etcd_node.adoc" in README is broken, there is a "docs" repeats:
https://github.com/redhat-cop/openshift-toolkit/blob/master/etcd-procedures/docs/docs/add_etcd_node.adoc

OCP 4 OLM Mirroring and sample operator managed image sync script

Lack of reliable scripts to do OLM content mirroring for OCP 4 restricted network installation. We should have one from redhat-cop. This also can be extended to ocp 4 sample operator managed imagestream.

3.11 ocp-master-cert-backup.sh date format %k%M%S make script fail on RHEL

The date format %k%M%S make it has space when hour is 1 digit as
20190221- 60545

Need to change to %H%M%S to make fix 2 digit format for hour
20190221-060545

Add RBAC capability to Custom Grafana so that we can have namespace level dashboards

Pre-install validation playbooks needs a README

@stratus-ss

Build UI layer for capacity planner tool

We would like to build an "executive level" UI around our new capacity planner tool.

Currently the tool traverses an openshift cluster and collects quota, limit, and request data (real time usage data coming soon), and exposes it via a simple API.

The API presents 3 different layers of scope:

Cluster level
- Total resource allocations for the entire cluster
Namespace level:
- Total resources allocation per namespace
Pod level:
- same as above for pods

We would like to be able to visualize this information for leadership to give them some assistance in making decisions.

By default, we would want to display the cluster level metrics, but provide the ability to drill down into the namespace and pod. We get a lot of asks to "show be the top 10 largest namespaces" so the ability to sort based on any of the given metrics would be really helpful. Here is a sample of what we should by default for the cluster.


Total Cluster Size:
----
	Memory:	151.23707342147827 Gi
	CPU:	37.2 Cores

Amount Claimed By Quota:
----
	Memory:	12.0 Gi (7.934562424755168%)
	CPU:	2 Cores (5.376344086021505%)

Total Limits:
----
	Memory:	108.569046119228 Gi (71.78732281909478%)
	CPU:	11.444999999999997 Cores (30.766129032258053%)

Total Requests:
----
	Memory:	47.17162685096264 Gi (31.190518160518344%)
	CPU:	10.724999999999987 Cores (28.830645161290285%)

docker-registry-sync.py fills local /var partition when pushing to private registry

Hi, I'm trying to use your docker-registry-sync.py script to populate a disconnected registry for an OCP 3.11 install at a customer site.

I noticed that the script caches the images locally on the server running the script and is storing the images in /var/lib/docker/overlay2

We don't have enough storage on the box running the script to handle this and it would be real convenient if the script had an option to delete each local image after it pushes to the remote registry.

The script also doesn't seem to have any functionality to clean up after itself at the end of the run. This will lead to eventual filling of any partition as the registry images come out with newer versions/etc. All the old versions will remain on the filesystem until manually deleted via "docker image rm..."

Thanks,

-- Dave Wujcik
(dwujcik) at (red hat)

docker_tags.json missing registry-console image

@redhat-cop/cant-contain-this