Author: @mikedep333 ([email protected])

Redmine Issue: 5592, https://pulp.plan.io/issues/5592

None

ansible/awx-operator#753
ansible/awx-operator#775

When using cert-manager to generate TLS certs for pulp-operator, we need to 'copy' the data from the secret and reformat it for the container_token_secret.

What we should do, is replace our format with cert-manager format. We can preserve backwards compat but should recommend cert-manager as the default.

apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls

Author: @mikedep333 ([email protected])

Redmine Issue: 5587, https://pulp.plan.io/issues/5587

None

Create new Pulp instance using operator, expand "Advanced Configuration", then expand "Database resource requirements". Set a limit for CPU, RAM, and Storage

Leave all other default options and create.

Postgres StatefulSet fails with this message:

create Claim postgres-pulp-postgres-0 for Pod pulp-postgres-0 in StatefulSet pulp-postgres failed error: PersistentVolumeClaim "postgres-pulp-postgres-0" is invalid: spec.resources[storage]: Required value

Author: @mikedep333 ([email protected])

Redmine Issue: 5598, https://pulp.plan.io/issues/5598

None

Author: @mikedep333 ([email protected])

Redmine Issue: 5593, https://pulp.plan.io/issues/5593

pulp/pulpcore#127 (comment)

[localhost] => {\"ansible_facts\": {\"api_groups\": [\"\", \"apiregistration.k8s.io\", \"apps\", \"events.k8s.io\", \"authentication.k8s.io\", \"authorization.k8s.io\", \"autoscaling\", \"batch\", \"certificates.k8s.io\", \"networking.k8s.io\", \"extensions\", \"policy\", \"rbac.authorization.k8s.io\", \"storage.k8s.io\", \"admissionregistration.k8s.io\", \"apiexten
sions.k8s.io\", \"scheduling.k8s.io\", \"coordination.k8s.io\", \"node.k8s.io\", \"discovery.k8s.io\", \"flowcontrol.apiserver.k8s.io\", \"pulp.pulpproject.org\"]}, \"changed\": false}\n\r\nTASK [pulp-api : Determine the cluster type] ***********************************\r\ntask path: /opt/ansible/roles/pulp-api/tasks/main.yml:91\nok: [localhost] => {\"ansible_facts\": {\"is_k8s\": true, \"is_openshift\": false}, \"changed\": false}\n\r\nTASK [pulp-api : debug] ********************************************************\r\ntask path: /opt/ansible/roles/pulp-api/tasks/main.yml:100\nok: [localhost] => {\r\n    \"msg\": \"CLUSTER TYPE: is_openshift=False; is_k8s=True\"\r\n}\n\r\nTASK [pulp-api : fail] *********************************************************\r\ntask path: /opt/ansible/roles/pulp-api/tasks/main.yml:102\nskipping: [localhost] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\n\r\nTASK [pulp-api deployment] *****************************************************\r\ntask path: /opt/ansible/roles/pulp-api/tasks/main.yml:108\nfatal: [localhost]: FAILED! => {\"msg\": \"The task includes an option with an undefined variable. The error was: 'image_pull_policy' is undefined\\n\\nThe error appears to be in '/opt/ansible/roles/pulp-api/tasks/main.yml': line 108, column 3, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n- name: pulp-api deployment\\n  ^ here\\n\"}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=30   changed=0    unreachable=0    failed=1    skipped=35   rescued=0    ignored=0   \r\n\n","job":"2775422040480279449","name":"example-pulp","namespace":"default","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:239"}


--------------------------- Ansible Task Status Event StdOut  -----------------

PLAY RECAP *********************************************************************
localhost                  : ok=30   changed=0    unreachable=0    failed=1    skipped=35   rescued=0    ignored=0   

Steps to reproduce:

1. Clone operator repository
2. Run ./up.sh with no changes to the repository.

If CRD is modified with the 'stable' tag, deployment works. I believe this behaviour was introduced by #119 - lacking default values for the image_pull_policy var.

From: https://pulp.plan.io/issues/8965

Author: @mikedep333 ([email protected])

Redmine Issue: 5595, https://pulp.plan.io/issues/5595

pulp/pulpcore#127 (comment)

Author: @mikedep333 ([email protected])

Redmine Issue: 5140, https://pulp.plan.io/issues/5140

See:
pulp/pulpcore#127 (comment)

Need to figure out how to implement.

Author: @mikedep333 ([email protected])

Redmine Issue: 5132, https://pulp.plan.io/issues/5132

Synopsis

Kubernetes Operators are the the new preferred way to deploy and manage applications on top of Kubernetes / OpenShift.

Once ready for the community to consume, they are published on https://OperatorHub.io

TODO

Before our operator can be published, it needs to be stable enough in certain ways:

• Ansible Molecule and Pulp-Smash need to run against it
• Users need to be able to specify a consistent/stable port to access content (& API) against, rather than just the random port that K8s assigns.
• Storage needs to be expandable
• In general, a user's interaction with the containers should be stable. It shouldn't randomly stop working without interaction because we re-worked something in the containers / operator.
• Some security improvements should be implemented

Not required

The following is strongly desirable, but is an acceptable limitation for this Epic:

• Support for K8s clusters that do not support ReadWriteMany Persistent Volume Claims. This can be documented as a limitation if not implemented in time.

Author: @mikedep333 ([email protected])

Redmine Issue: 5603, https://pulp.plan.io/issues/5603

None

Version
quay.io/pulp/pulp-operator:v0.7.0

Describe the bug
If I invoke make deploy with a namespace other than pulp-operator-system, the operator will not work properly.

export NAMESPACE=my-pulp
make deploy

The status of the Pod will be Running,

$ kubectl -n my-pulp get pod
NAME                                               READY   STATUS    RESTARTS   AGE
pulp-operator-controller-manager-9b8644f46-cw59d   2/2     Running   0          12m

however, the log shows errors repeatedly and continuously.

$ kubectl -n my-pulp logs deployments/pulp-operator-controller-manager -c pulp-manager | tail -n 10
E0204 14:41:29.324437       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:33.674177       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:37.899865       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:41.273833       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:45.492735       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:49.333856       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:52.762954       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:56.118074       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:41:59.256319       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"
E0204 14:42:02.443896       7 leaderelection.go:325] error retrieving resource lock my-pulp/pulp-operator: configmaps "pulp-operator" is forbidden: User "system:serviceaccount:my-pulp:pulp-operator-sa" cannot get resource "configmaps" in API group "" in the namespace "my-pulp"

In this state, nothing happens even if I create new kind: Pulp resources in my-pulp namespace.

To Reproduce

# Preparation
git clone https://github.com/pulp/pulp-operator.git
cd pulp-operator
git checkout 0.7.0

# Set namespace other than pulp-operator-system and deploy
export NAMESPACE=my-pulp
make deploy

# Check logs
kubectl -n my-pulp logs -f deployments/pulp-operator-controller-manager -c pulp-manager

Expected behavior
Pulp Operator works in my-pulp namespace.

Additional context
The hard-coded namespace in some config files for RBAC e.g. auth_proxy_role_binding.yaml apears to be the root cause.
The RoleBindings in my-pulp namespace subject to the SA in pulp-operator-system namespace which does not exist.

$ kubectl -n my-pulp get rolebinding -o wide
NAME                                        ROLE                                      AGE   USERS   GROUPS   SERVICEACCOUNTS
pulp-operator-leader-election-rolebinding   Role/pulp-operator-leader-election-role   20m                    pulp-operator-system/pulp-operator-sa
pulp-operator-proxy-rolebinding             Role/pulp-operator-proxy-role             20m                    pulp-operator-system/pulp-operator-sa
pulp-operator-pulp-manager-rolebinding      Role/pulp-operator-pulp-manager-role      20m                    pulp-operator-system/pulp-operator-sa

Author: @mikedep333 ([email protected])

Redmine Issue: 5597, https://pulp.plan.io/issues/5597

None

For pulpcore >= 3.16

From: https://pulp.plan.io/issues/9172

Author: @mikedep333 ([email protected])

Redmine Issue: 5588, https://pulp.plan.io/issues/5588

Many environments don't provide the needed ReadWriteMany such as:
openshift dedicated environment backing galaxy
Metakube environment
Mike Adamz environments

Currently the container fails to start:

$ podman run --rm quay.io/pulp/pulp-web:3.16.1
nginx: [emerg] "location" directive is not allowed here in
/opt/app-root/etc/nginx.d/pulp_ansible.conf:1

Those custom nginx config files should be present in the
/opt/app-root/etc/nginx.default.d directory instead.

NGINX_CONFIGURATION_PATH -> config under the http section.
NGINX_DEFAULT_CONF_PATH -> config under the server section.

The location directive should be used in the server section.

Author: @mikedep333 ([email protected])

Redmine Issue: 5864, https://pulp.plan.io/issues/5864

None

Author: @mikedep333 ([email protected])

Redmine Issue: 5404, https://pulp.plan.io/issues/5404

Currently in order to do CI testing, plugin-template installs RPM and python packages into the pulp-api container.

This has 3 problems:
1. It is brittle. If pulp-api is restarted, they are lost.
2. It means the container image is being modified; which could possibly invalidate tests.
3. It is probably not the correct design pattern.

This task is to reach the correct design pattern, and implement it.
A possible alternative pattern is running tests from a separate sidecar container in the same pod.

Author: @mikedep333 ([email protected])

Redmine Issue: 5601, https://pulp.plan.io/issues/5601

None

Hello,

While trying to set up an instance of pulp on pulp-operator, we are facing an issue, as we don't have storage that supports the ReadWriteMany with volume mode file-system. We thought of using the alternative and switch to ReadwriteOnce mode [1]
and use node affinity to make the deployment stay on one single node , so all pods can interact with the storage(as the ReadWriteOnce on same node kinda behaves as ReadWriteMany)

However, when stating the node affinity, it is not bounding the pods to the same node.
Is this the right way of using the pulp node affinity?
https://github.com/operate-first/apps/blob/f67c2ae1556633acdad02f0771800ce488e62e83/pulp/overlays/moc/smaug/opf-pulp.yaml#L31

Related-to: operate-first/support#456

Hello,
We are using the pulp-operator: v0.6.1.
After deploying the pulp instance with the following config: link.

we are facing pulp-worker pod getting crash looped due to the following error:

+ /usr/bin/wait_on_postgres.py
Waiting on postgresql to start...
Checking postgres host pulp-postgres
Checking postgres port 5432
Postgres started!
+ /usr/bin/wait_on_database_migrations.sh
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "pulp"

complete log file:
pulp-worker-74c67b4895-brzc6-worker.log

Author: @dkliban ([email protected])

Redmine Issue: 5784, https://pulp.plan.io/issues/5784

Pulp operator should use the JSON format for pulp settings. This way the 'to_json' filter will work as expected.

The settings file should be called settings.json.

https://github.com/pulp/pulp-operator/blob/master/roles/pulp-api/templates/pulp-server.config-map.yaml

Author: @mikedep333 ([email protected])

Redmine Issue: 5137, https://pulp.plan.io/issues/5137

pulp-operator currently uses static sized Persistent Volume Claims for both the PostgreSQL database and for /var/lib/pulp .

This is unacceptable because by adding/syncing more content, users can exceed the capacity. Also, the current values may be too large.

As a developer, I use github issues instead of redmine

under certain circumstances syncing content(console.redhat.com) is causing the sync task to get canceled because of OOM conditions. Raise the limit to something greater than we expect.

When there are e.g. multiple authenticated container registries used
we need to be able to add multiple imagePullSecrets to the k8s resource

ansible/awx-operator#701

Error: writing blob: uploading layer chunked: StatusCode: 413,

Seems related to nginx config. we need a bigger max_body_size for containers

Author: @mikedep333 ([email protected])

Redmine Issue: 5136, https://pulp.plan.io/issues/5136

pulp-operator needs a stable & configurable port for the pulp webserver (nginx/apache) to be accessed on.

This way, sysadmins can say update DNS records for their pulp server to redirect clients to a new Pulp server, without changing the client config on every single system.

Currently, pulp-api and pulp-content are exposed directly without the webserver, and the port is randomized.

Author: @mikedep333 ([email protected])

Redmine Issue: 5599, https://pulp.plan.io/issues/5599

Notes/convo previous about this:
Rollback? (Yes, often will a backup operator. An operator that shuts down services & what not for backups.)

we see failures creating an Ingress. Specifically, the Application Gateway Ingress Controller (AGIC) reports the following:

E0211 19:59:20.564021 1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 – Original Error: Code="ApplicationGatewayProbePathIsInvalid" Message="Path specified for Probe /subscriptions/3f7e29ba-24e0-42f6-8d9c-5149a14bda37/resourceGroups/rg-aapg56lg5t7hy2ko-nodepools-eastus/providers/Microsoft.Network/applicationGateways/rybappgwui-nodepool/probes/pb-ansible-automation-platform-automation-hub-s-002a92caa9cf3a5dae90b90838a0c505 is not valid. Path must start with a forward slash  and meet the format requirements." Details=[]

OSBS has the capability to inject image urls at build time if they are specified in a specific format, which happens to be as environment variables that start with RELATED_IMAGES_.

We should conform to this standard so that we can start taking advantage of this in order to support deployments to disconnected environments.

Author: spredzy (spredzy)

Redmine Issue: 8207, https://pulp.plan.io/issues/8207

As a user, I would like to specify an external database when deploying my operator

Meaning, a way should be provided to specify the needed variables.

Ideally two ways should be provided, in order of priority:

• Deployer provides a secret that follows a template, where the informations are provided
• Deployer provides N variables with a 1:1 match for configuration that is supported (like, pg_host, pg_database, pg_user, pg_password, pg_port, ...)

This was originally found on pulp-installer, but I believe operator also needs this

Nginx

(pulp) [vagrant@pulp3-source-fedora33 pulp_ansible]$ ansible-galaxy collection publish -c -s http://pulp3-source-fedora33.localhost.example.com/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api /tmp/tmpajfv5by1/cisco-nxos-1.4.0.tar.gz
Publishing collection artifact '/tmp/tmpajfv5by1/cisco-nxos-1.4.0.tar.gz' to cmd_arg http://pulp3-source-fedora33.localhost.example.com/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api
ERROR! Error when publishing collection to cmd_arg (http://pulp3-source-fedora33.localhost.example.com/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api) (HTTP Code: 413, Message: Request Entity Too Large Code: Unknown)

Localhost:

(pulp) [vagrant@pulp3-source-fedora33 pulp_ansible]$ ansible-galaxy collection publish -c -s http://localhost:24817/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api /tmp/tmpajfv5
by1/cisco-nxos-1.4.0.tar.gz
Publishing collection artifact '/tmp/tmpajfv5by1/cisco-nxos-1.4.0.tar.gz' to cmd_arg http://localhost:24817/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api
Collection has been published to the Galaxy server cmd_arg http://localhost:24817/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api
Waiting until Galaxy import task http://localhost:24817/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api/v3/imports/collections/1f16da54-d073-42cc-8447-42d875954048/ has completed
Collection has been successfully published and imported to the Galaxy server cmd_arg http://localhost:24817/pulp_ansible/galaxy/555bb1ec-6076-4523-a79e-cb9ae4d14b3c/api

https://pulp.plan.io/issues/8414

Author: @mikedep333 ([email protected])

Redmine Issue: 5591, https://pulp.plan.io/issues/5591

None

If overriding tag under spec, it overrides them for all containers for the automationhub object. I propose that we just omit tag and let the users set the tag as part of the custom image.

Example:

It looks like we should be fine getting rid of this tag var considering we don't use it on awx-operator, example:

https://github.com/ansible/awx-operator/blob/ebe5d1e2f4afd7fd329c18420947ff5840670293/roles/installer/templates/deployment.yaml.j2#L265

The upstream default tags for pulp-operator is latest, which it will pull anyways if the tag is not set.

Right now, as is, if users try to override the image with image or image_web, they cannot set the tag. If they set the tag, it'll override the tag for both.

Author: @mikedep333 ([email protected])

Redmine Issue: 5589, https://pulp.plan.io/issues/5589

None

Related awx-operator PR: ansible/awx-operator#541

I propose we move the pulp-operator from a cluster-scoped operator to a namespace-scoped operator. At a high level this means only Roles for service accounts, no ClusterRoles, and it also means that operators & Pulp deployments will have a 1-to-1 relationship.

• This means one operator can only deploy to a single namespace
• This increases security, the pulp-operator SA has less cluster-wide
  access
• Safer upgrades: upgrade is sandboxed to a single namespace, so this will help with conflicts with other pulp-operators (or operators in general) in a cluster.
• If Pulp deployments are needed in multiple namespaces, multiple pulp-operators can be deployed to accomplish this.

https://www.youtube.com/watch?v=gX8UuLRfGec

Author: @mikedep333 ([email protected])

Redmine Issue: 5590, https://pulp.plan.io/issues/5590

None

Author: @mikedep333 ([email protected])

Redmine Issue: 5602, https://pulp.plan.io/issues/5602

None

Author: @mikedep333 ([email protected])

Redmine Issue: 5683, https://pulp.plan.io/issues/5683

None

Ref: pulp/pulpcore#2057

the pod limit is currently set to 1Gi, should it need more than 1Gi or is there a memory issue somewhere in the sync process?

The pulp-operator currently supports backup and restore via custom resources. The CI/CD pipeline should have a test that validates this continues to function.

Test steps:

• Install operator and create a Pulp custom resource
• Add content to Pulp (e.g. Ansible Collection)
• Create a backup by defining a PulpBackup custom resource
• Delete the previous Pulp custom resource to remove the Pulp instance
• Create a PulpRestore custom resource which will create a Pulp custom resource and restore the db and files
• Validate that the content is present in the "restored" Pulp instance

from: https://pulp.plan.io/issues/9145

Author: @mikedep333 ([email protected])

Redmine Issue: 5139, https://pulp.plan.io/issues/5139

See:
pulp/pulpcore#127 (comment) pulp/pulpcore#127 (comment)

Hello

we have pulp-operator:v0.6.1 deployed.
we are noticing the following error in the pulp-api
pulp [eff4343985714d8bb0b891b0b6e3e5ce]: pulpcore.app.views.status:ERROR: Connection to Redis failed during status check!
The redis status seems to be ready for connection.
please can someone help to resolve this issue, due to this the API is not getting ready for serving

Additional info:
pulp-api log:
pulp-api-7b98f5878-j7bst-api.log

pulp-redis log:
pulp-redis-7494b78744-j8mvk-redis.log

pulp status: https://pulp.operate-first.cloud/pulp/api/v3/status/

Related-to: operate-first/support#176 (comment)