Issue connecting to any data source about contrib-helm-chart HOT 9 OPEN

@AndrewChubatiuk Sounds like one for you? 😄

from contrib-helm-chart.

Hi, I got a similar issue. When testing a datasource, the UI timeouts after a while and this appears on logs :

[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:49 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:49,196][PID:9][INFO][metrics] method=POST path=/api/data_sources/1/test endpoint=datasourcetestresource status=500 content_type=? content_length=-1 duration=60422.93 query_count=4 query_duration=7.50
[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:49 +0000] [9] [ERROR] Error handling request /api/data_sources/1/test
[redash-7c94b854dc-8p2h8] Traceback (most recent call last):
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 135, in handle
[redash-7c94b854dc-8p2h8] self.handle_request(listener, req, client, addr)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 178, in handle_request
[redash-7c94b854dc-8p2h8] respiter = self.wsgi(environ, resp.start_response)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2213, in call
[redash-7c94b854dc-8p2h8] return self.wsgi_app(environ, start_response)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in call
[redash-7c94b854dc-8p2h8] return self.app(environ, start_response)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2190, in wsgi_app
[redash-7c94b854dc-8p2h8] response = self.full_dispatch_request()
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1484, in full_dispatch_request
[redash-7c94b854dc-8p2h8] rv = self.dispatch_request()
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1469, in dispatch_request
[redash-7c94b854dc-8p2h8] return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask_restful/init.py", line 489, in wrapper
[redash-7c94b854dc-8p2h8] resp = resource(*args, **kwargs)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask_login/utils.py", line 277, in decorated_view
[redash-7c94b854dc-8p2h8] return current_app.ensure_sync(func)(*args, **kwargs)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 109, in view
[redash-7c94b854dc-8p2h8] return current_app.ensure_sync(self.dispatch_request)(**kwargs)
[redash-7c94b854dc-8p2h8] File "/app/redash/handlers/base.py", line 31, in dispatch_request
[redash-7c94b854dc-8p2h8] return super(BaseResource, self).dispatch_request(*args, **kwargs)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/flask_restful/init.py", line 604, in dispatch_request
[redash-7c94b854dc-8p2h8] resp = meth(*args, **kwargs)
[redash-7c94b854dc-8p2h8] File "/app/redash/permissions.py", line 71, in decorated
[redash-7c94b854dc-8p2h8] return fn(*args, **kwargs)
[redash-7c94b854dc-8p2h8] File "/app/redash/handlers/data_sources.py", line 232, in post
[redash-7c94b854dc-8p2h8] time.sleep(1)
[redash-7c94b854dc-8p2h8] File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
[redash-7c94b854dc-8p2h8] sys.exit(1)
[redash-7c94b854dc-8p2h8] SystemExit: 1
[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:49 +0000] [9] [INFO] Worker exiting (pid: 9)
[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:50 +0000] [1] [ERROR] Worker (pid:9) was sent SIGKILL! Perhaps out of memory?
[redash-7c94b854dc-8p2h8] [2024-04-22 07:16:50 +0000] [103] [INFO] Booting worker with pid: 103

from contrib-helm-chart.

@iRakic are you using image provided with chart version 3.0.1 or do you set a custom one?

from contrib-helm-chart.

@iRakic are you using image provided with chart version 3.0.1 or do you set a custom one?

I'm using the one from the chart.

from contrib-helm-chart.

which datasource type are you trying to query? does it return lots of data during these attempts to query?
have you tried to set server.resources?

from contrib-helm-chart.

@AndrewChubatiuk I guess this last reply is for @chenchuk77
In my tests, I didn't encounter out of memory since server.resources is not set.

which datasource type are you trying to query?
I have tried multiple databases, such as MSSQL, Postgres, and MariaDB, as a test.

I have also noticed this when I have tried to run a test under data source: [2024-04-22 16:21:55 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:2170) . It is from redash pod

from contrib-helm-chart.

you may have resources not set but do you have enough free ram on machine?

from contrib-helm-chart.

you may have resources not set but do you have enough free ram on machine?

Yes, like at least 64GB of RAM at all times. It's a cluster of multiple bare-metal servers.

from contrib-helm-chart.

Hi @AndrewChubatiuk
Here is my values :

  resources:
    limits:
      cpu: 500m
      memory: 3Gi
    requests:
      cpu: 500m
      memory: 2Gi

I tried a datasource of type postgres (which is accessible from the pod shell). i also try JSON from a url. both timedout and i see similiar output in logs

from contrib-helm-chart.