Webserver component not working about kube-opex-analytics HOT 17 CLOSED

When switching uwsgi off, back to direct call in entrypoint
python3 -u ./backend.py

and call the flask app directly in backend.py
if __name__ == '__main__': app.run(host='0.0.0.0', port=5483)

Frontend is loading for me.
So it seems to be related to uwsgi and/or the prometheus middleware part. But how to investigate that further?

from kube-opex-analytics.

Hi @choseh,
Sorry for the late reply.
Did you try to access the service through port forwarding instead?

Run this command on your local machine and open a browser at http://127.0.0.1:9080

kubectl -n kube-opex-analytics port-forward service/kube-opex-analytics 9080:80

This command assumes that you're running the service in the namespace kube-opex-analytics

from kube-opex-analytics.

Also which version do you use?

from kube-opex-analytics.

We're running it in an existing k8s cluster. It's behind traefik in this case. We're on 0.4.5, same issue with prior versions. I think curling it from the pod itself should work, shouldn't it?

from kube-opex-analytics.

Can you please try curl http://kube-opex-analytics/ from the pod' console (instead of using http://127.0.0.1:5483)

from kube-opex-analytics.

root@kube-opex-analytics-69989c4b8-ppjmj:/koa# curl http://kube-opex-analytics/
^C

same thing. timeout

from kube-opex-analytics.

netstat:
tcp 3 0 0.0.0.0:5483 0.0.0.0:* LISTEN 0 2374429 8/uwsgi

svc:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-11-12T09:10:28Z"
labels:
app: kube-opex-analytics
app.kubernetes.io/instance: kube-opex-analytics
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: kube-opex-analytics
app.kubernetes.io/version: "1.0"
helm.sh/chart: kube-opex-analytics-0.4.5
name: kube-opex-analytics
namespace: monitoring
resourceVersion: "92627935"
selfLink: /api/v1/namespaces/monitoring/services/kube-opex-analytics
uid: 4512c147-052c-11ea-a1bb-0289a95f0f74
spec:
clusterIP: 100.64.93.142
ports:

• name: http
  port: 80
  protocol: TCP
  targetPort: 5483
  selector:
  app.kubernetes.io/instance: kube-opex-analytics
  app.kubernetes.io/name: kube-opex-analytics
  sessionAffinity: None
  type: ClusterIP
  status:
  loadBalancer: {}

ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
creationTimestamp: "2019-11-12T09:10:28Z"
generation: 1
labels:
app: kube-opex-analytics
app.kubernetes.io/instance: kube-opex-analytics
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: kube-opex-analytics
app.kubernetes.io/version: "1.0"
helm.sh/chart: kube-opex-analytics-0.4.5
traffic-type: private
name: kube-opex-analytics
namespace: monitoring
resourceVersion: "92627938"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring/ingresses/kube-opex-analytics
uid: 4513ff28-052c-11ea-a1bb-0289a95f0f74
spec:
rules:

• host: kube-opex-analytics.travel-test.invia.lan
  http:
  paths:
  • backend:
    serviceName: kube-opex-analytics
    servicePort: http
    path: /
    status:
    loadBalancer: {}

from kube-opex-analytics.

I don't know if it's related, perhaps you may try it in unprivileged mode?
Set securityContext to true in values.yaml file

from kube-opex-analytics.

We had that set to true before, in order to allow more debugging we switched it to false. I just set it to true again, but unfortunately this didn't change anything.

*** Starting uWSGI 2.0.18 (64bit) on [Tue Nov 12 21:10:17 2019] ***
compiled with version: 7.4.0 on 10 November 2019 22:22:21
os: Linux-4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11)
nodename: kube-opex-analytics-7d755c8d46-2qptx
machine: x86_64
clock source: unix
detected number of CPU cores: 16
current working directory: /koa
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 1048576
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
*** RRDtool library available at 0x560d87c3aa50 ***
uwsgi socket 0 bound to TCP address :5483 fd 3
Python version: 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0]
Python main interpreter initialized at 0x560d87c6bf00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145808 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x560d87c6bf00 pid: 8 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8)
spawned uWSGI worker 1 (pid: 14, cores: 1)
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,

from kube-opex-analytics.

How do you use traefik, as service mesh?
Perhaps you need a sidecar or additional config to make the routing works with traefik?

This is not something we're already experienced, I just make a test on GKE and everything looks good.
Unfortunately I don't have a cluster with traefik to try to reproduce the problem in your context.

from kube-opex-analytics.

can you curl http://127.0.0.1:5483 from within the pod?

from kube-opex-analytics.

And no, we're on traefik <2.0 so no service mesh stuff yet.

from kube-opex-analytics.

Here is what I did in GKE :

kubectl run -i --tty busybox --image=busybox -- sh
wget -o - http://kube-opex-analytics.kube-opex-analytics

and the output is

saving to 'index.html'
index.html           100% |******************************************************************************************************************************************************************************| 14604  0:00:00 ETA
'index.html' saved
/ # 

So everything is working well.

I don't know where you have problem in your cluster, is it the only component that is experiencing such issue?

from kube-opex-analytics.

Hi, when i run it locally in Docker Container (build with Dockerfile from master branch) with Debug Mode and token to a small test cluster im getting timeout to 127.0.0.1:5483 too.
So seems not be related to k8s at all. My Docker Version 18.06.1-ce-rc1

from kube-opex-analytics.

Hi,
@atze234 many thanks for this feedback.

@choseh are you able to try this trick? If it's confirmed that it fix the issue we'll manage to introduce it as alternative (through a startup option) in our upcoming releases.

from kube-opex-analytics.

hey @rchakode
@atze234 is actually on my team and we were discussing this already today, so we're working on the same thing :) We found an alternative that offers aws billing optiona out of the box and I don't think we'll investigate any further. We can help if anyone else has the same issue though.

from kube-opex-analytics.

Right, thank you for the feedback.

from kube-opex-analytics.