Postgres database statefulset created/deployed even if external database settings are defined about awx-operator HOT 18 CLOSED

I am seeing the same issue.

from awx-operator.

Seeing the same issue. External database is up and running yet a postgres-0 pod is running after deployment

from awx-operator.

Hello all,

My guess is that you might be affected by this #124

TL;DR: Current devel of the CRDs and logic does not necessarily work with 0.6.0. If you want to fix this now while the above issue is addressed, follow this pattern:

1. Download and install operator-sdk==0.19.4 https://v0-19-x.sdk.operatorframework.io/docs/install-operator-sdk/
2. Build the operator container operator-sdk build <url>:<tag> (at the root of this repo)
3. Push the container docker push <url>:<tag>
4. Retrieve locally https://raw.githubusercontent.com/ansible/awx-operator/devel/deploy/awx-operator.yaml and update the url to the operator container
5. Deploy.

Hope this helps,

from awx-operator.

I did this, and now awx can't connect to my DB at all. It's connecting to the right DB, but It seems to be ignoring ca_trust_bundle: /etc/ca-bundle/ca_bundle.pem

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 81, in _ctit_db_wrapper
    yield
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 441, in __getattr__
    value = self._get_local(name)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 383, in _get_local
    setting = Setting.objects.filter(key=name, user__isnull=True).order_by('pk').first()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 653, in first
    for obj in (self if self.ordered else self.order_by('pk'))[:1]:
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
    cursor = self.connection.cursor()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  password authentication failed for user "awx"
FATAL:  no pg_hba.conf entry for host "172.30.239.68", user "awx", database "awx", SSL off

2021-03-16 19:37:30,631 ERROR    [-] awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "awx"
FATAL:  no pg_hba.conf entry for host "172.30.92.194", user "awx", database "awx", SSL off

from awx-operator.

Hello @adammike, is it fixed? how did you manage to attach the .crt file inside pods. i am also having same issue.

from awx-operator.

I tested it today and it worked for me when using an external database. I basically created the secret <resource_name>-postgres-configuraton as noted at https://github.com/ansible/awx-operator#external-postgresql-service and then created the awx kind.

If you connect to your -web container, what are the contents you see at the /etc/tower/conf.d/environment.sh and /etc/tower/conf.d/credentials.py. Does it match with your <resource_name>-postgres-conrfiguration secret?

from awx-operator.

I did this, and now awx can't connect to my DB at all. It's connecting to the right DB, but It seems to be ignoring ca_trust_bundle: /etc/ca-bundle/ca_bundle.pem

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 81, in _ctit_db_wrapper
    yield
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 441, in __getattr__
    value = self._get_local(name)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 383, in _get_local
    setting = Setting.objects.filter(key=name, user__isnull=True).order_by('pk').first()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 653, in first
    for obj in (self if self.ordered else self.order_by('pk'))[:1]:
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
    cursor = self.connection.cursor()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  password authentication failed for user "awx"
FATAL:  no pg_hba.conf entry for host "172.30.239.68", user "awx", database "awx", SSL off

2021-03-16 19:37:30,631 ERROR    [-] awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "awx"
FATAL:  no pg_hba.conf entry for host "172.30.92.194", user "awx", database "awx", SSL off

@adammike there is a chance of your PostgreSQL using the same PVC from an old deployment, therefore the awx credentials will fail. I had the same issue when using a storageClass on my lab that had retention enabled.

from awx-operator.

@adammike are you still hitting this issue?

from awx-operator.

from awx-operator.

Any updates on this @adammike? Thanks

from awx-operator.

I'm also seeing this, deployed awx-operator 0.9.0 with an external postgres db.
AWX is connecting to the given external database, but a awx-postgres-0 container is also deployed.
Secret configuration stored:

---
apiVersion: v1
kind: Secret
metadata:
  name: awx-postgres-configuration
  namespace: default
stringData:
  host: 'x.x.x.x'
  port: '5432'
  database: awx
  username: awx
  password: xxxx
  sslmode: prefer
  type: managed
type: Opaque

When type is set to unmanaged, the awx-postgres-0 container is not being deployed, but then the awx database is not populated either.

awx-postgres-0 container:

Name:         awx-postgres-0
Namespace:    default
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Thu, 06 May 2021 13:02:56 +0200
Labels:       app.kubernetes.io/component=database
              app.kubernetes.io/managed-by=awx-operator
              app.kubernetes.io/name=awx-postgres
              app.kubernetes.io/part-of=awx
              controller-revision-hash=awx-postgres-cc4c66c8d
              statefulset.kubernetes.io/pod-name=awx-postgres-0
Annotations:  <none>
Status:       Running
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  StatefulSet/awx-postgres
Containers:
  postgres:
    Container ID:   docker://0be74fa74c9ee3d3f1af5a44d08a568c6ff598e78379c8deec084858b1c707d2
    Image:          postgres:12
    Image ID:       docker-pullable://postgres@sha256:06277995d7028f4455e56f21864146ee2425c83308dc96283c7a96e6881cc826
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 06 May 2021 13:03:11 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      POSTGRESQL_DATABASE:        <set to the key 'database' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRESQL_USER:            <set to the key 'username' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRESQL_PASSWORD:        <set to the key 'password' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_DB:                <set to the key 'database' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_USER:              <set to the key 'username' in secret 'awx-postgres-configuration'>  Optional: false
      POSTGRES_PASSWORD:          <set to the key 'password' in secret 'awx-postgres-configuration'>  Optional: false
      PGDATA:                     /var/lib/postgresql/data/pgdata
      POSTGRES_INITDB_ARGS:       --auth-host=scram-sha-256
      POSTGRES_HOST_AUTH_METHOD:  scram-sha-256
    Mounts:
      /var/lib/postgresql/data from postgres (rw,path="data")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mbghf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  postgres:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-awx-postgres-0
    ReadOnly:   false
  default-token-mbghf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mbghf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  17m (x2 over 17m)  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         17m                default-scheduler  Successfully assigned default/awx-postgres-0 to minikube
  Normal   Pulling           17m                kubelet            Pulling image "postgres:12"
  Normal   Pulled            17m                kubelet            Successfully pulled image "postgres:12" in 11.350880082s
  Normal   Created           17m                kubelet            Created container postgres
  Normal   Started           17m                kubelet            Started container postgres

from awx-operator.

I'm also seeing this, deployed awx-operator 0.9.0 with an external postgres db.
AWX is connecting to the given external database, but a awx-postgres-0 container is also deployed.
Secret configuration stored:

---
apiVersion: v1
kind: Secret
metadata:
  name: awx-postgres-configuration
  namespace: default
stringData:
  host: 'x.x.x.x'
  port: '5432'
  database: awx
  username: awx
  password: xxxx
  sslmode: prefer
  type: managed
type: Opaque

When type is set to unmanaged, the awx-postgres-0 container is not being deployed, but then the awx database is not populated either.

@gstorme could you send us the /etc/tower/conf.d/credentials.py from the awx-task container? Also the output from awx-manage check_db?

from awx-operator.

DATABASES = {
    'default': {
        'ATOMIC_REQUESTS': True,
        'ENGINE': 'awx.main.db.profiled_pg',
        'NAME': "awx",
        'USER': "awx",
        'PASSWORD': "xxxxx",
        'HOST': '192.168.3.90',
        'PORT': "5432",
        'OPTIONS': { 'sslmode': 'prefer',
                     'sslrootcert': '/etc/pki/tls/certs/ca-bundle.crt',
        },
    }
}

BROADCAST_WEBSOCKET_SECRET = "ul1DzvGxe15etiZQo0a18ZtfgqkSbIL5"

Database Version: PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit

from awx-operator.

@gstorme I'm a little bit confuse. You mentioned the database is being deployed even when the external database is set. Let me try to clarify a few things.

1. When the type=managed that means that statefulset will be created. For external databases, you will want that to be set as type=unmanaged.
2. The database seems to be communicating correctly as the awx-manage check_db returned correctly.

So if you need that be an external database. you can do the following:

• Modify your secret to type=unmanaged, then if you want to get rid of the statefulset you can delete the current awx instance and re-create it. That should do the trick for you.

from awx-operator.

This is working fine if you use an external database which is already populated from a previous awx installation.

Is there a way to use an external database and have that newly database populated, without having the postgres statefulset?
Like if you want to start from scratch without migrating from a previous awx database.
Or is the only way to do this, by setting an external db with type=managed, and afterwards setting it to type=unmanaged and recreate the awx instance?

from awx-operator.

@gstorme, yes you can do that. Basically, you will need to create a secret of the type=unmanaged. Once you do it, on your awx kind, you have to specify the secret to be used as explained at https://github.com/ansible/awx-operator#external-postgresql-service

So basically, let's assume this is the secret for the external database

---
apiVersion: v1
kind: Secret
metadata:
  name: external-postgres-configuration
  namespace: default
stringData:
  host: 192.168.1.1
  port: 5432
  database: awx-demo
  username: awx-demo
  password: secret
  sslmode: prefer
  type: unmanaged
type: Opaque

Now on your awx kind, you could do:

[...SNIP..]
spec:
    tower_postgres_configuration_secret: external-postgres-configuration
[...SNIP..]

Make sure the database already exists on the remote PostgreSQL server and the credentials are correct.

That should do the trick for you!

from awx-operator.

Ok this is working as expected indeed, no postgres statefulset is created and the external database is populated.
Don't know why it went wrong before, probably an error on my side then.
Thanks

from awx-operator.

Thanks! I'll close this for now as it worked for both of us.

from awx-operator.