Giter VIP home page Giter VIP logo

Comments (14)

stewartshea avatar stewartshea commented on June 1, 2024 1

@MuckerMayhem Thanks for sharing - and apologies for the issue.

The version you have should hav the code responsible for handling the scenario where namespaces could not be listed but projects could (a common tenant configuration in OpenShift). I will test against our own development cluster and get back to you if I have further questions.

from runwhen-local.

MuckerMayhem avatar MuckerMayhem commented on June 1, 2024 1

@MuckerMayhem just checking in - did you manage to make any progress or are you still stuck?

Hi, unfortunately I haven't had time to look at this, but my goal is to sit down and look through it tomorrow. I'll also pop in Slack, just in case I have questions.

from runwhen-local.

jon-funk avatar jon-funk commented on June 1, 2024

Adding some notes to this issue from a recent troubleshooting session with a user - enhanced exception handling would be great.

  • Kubeconfig was valid and able to list projects but not namespaces
  • user was on openshift

Initial error when running:
"Error 500 from Workspace Builder service for command "run": None
Logs contained:
'Internal Server Error: /run/'

from runwhen-local.

MuckerMayhem avatar MuckerMayhem commented on June 1, 2024

Hi, I'm the mysterious user in this case.

This is what I get for the sha on the image:

~$sudo docker inspect --format='{{index .RepoDigests 0}}' 8d750494f2c1
ghcr.io/runwhen-contrib/runwhen-local@sha256:9a0337c97e7a6edc066a3ae1fcacc882949ade9c9f84d22ff75378b07375024e

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

FWIW, additional error handling is on the list, but for now you can docker attach to the container to see more details;

The following is an example from my ocp demo host;

# Check access to projects
$ KUBECONFIG=openshift-demo-sa-kubeconfig oc get projects
NAME      DISPLAY NAME   STATUS
default                  Active

# Verify I don't have permission to list namespaces
$ KUBECONFIG=openshift-demo-sa-kubeconfig oc get ns      
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:default:demo-sa" cannot list resource "namespaces" in API group "" at the cluster scope

# docker attach RunWhenLocal 
Resetting neo4j models
Error scanning for namespace instances; skipping and continuing; error: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e25350e6-7cc7-4f9e-b811-6cbbed9c2611', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f12e0a43-09f3-43e6-8910-d858a7fd1295', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5341b65f-8932-4cce-881b-99952354ea3d', 'Date': 'Thu, 13 Jul 2023 13:47:33 GMT', 'Content-Length': '292'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:serviceaccount:default:demo-sa\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}


Scanning for Kubernetes resources in namespace "default"
Level of detail lookup failed; defaulting to full LOD; resource type=<class 'models.KubernetesCluster'>
Level of detail lookup failed; defaulting to full LOD; resource type=<class 'models.KubernetesCluster'>
Rendering output item: workspaces/my-workspace/slxs/default-pod-resources/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pod-resources/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/adservice-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/adservice-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/cartservice-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/cartservice-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/chcktsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/chcktsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/crrncysrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/crrncysrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/emlsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/emlsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/frontend-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/frontend-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/ldgnrtr-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ldgnrtr-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/pymntsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/pymntsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/prdctctlgsr-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/prdctctlgsr-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/rcmmndtnsrv-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/rcmmndtnsrv-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/shppngsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/shppngsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-image-check/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-image-check/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/slo.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-redis-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-redis-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-sa-check/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-sa-check/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pvc-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pvc-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-cpu/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-cpu/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-mem/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-mem/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-disk/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-disk/sli.yaml
Rendering output item: workspaces/my-workspace/workspace.yaml
[13/Jul/2023 13:47:36] "POST /run/ HTTP/1.1" 200 22386
[13/Jul/2023 13:50:52] "POST /run/ HTTP/1.1" 200 22390
INFO     -  [13:50:52] Detected file changes
INFO     -  Building documentation...
WARNING  -  Config value 'build': Unrecognised configuration name: build
WARNING  -  Config value 'dev_addr': The use of the IP address '0.0.0.0' suggests a production environment or the use of a proxy to connect to the MkDocs server. However, the MkDocs' server is intended for local development purposes only. Please use a third party production-ready server instead.
INFO     -  Documentation built in 0.18 seconds
INFO     -  [13:50:53] Reloading browsers
INFO     -  [13:50:53] Browser connected: http://localhost:8081/
INFO     -  [13:50:56] Detected file changes
INFO     -  Building documentation...

image

@MuckerMayhem can you docker attach to the container and share any helpful output there? My test indicates that the code should properly scan your cluster - though maybe we are having a problem with the kubeconfig / access? No need to share sensitive data, just generic output. If you want, can you also share your workspaceInfo.yaml file? Feel free to jump into slack if you want to debug together, otherwise we can keep using this thread. The link to join slack is in the top header of https://docs.runwhen.com

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

@MuckerMayhem just checking in - did you manage to make any progress or are you still stuck?

from runwhen-local.

MuckerMayhem avatar MuckerMayhem commented on June 1, 2024

So it seems my issue appears to be network related:

Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1d11280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1d118b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1b15a00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Internal Server Error: /run/
[21/Jul/2023 18:53:46] "POST /run/ HTTP/1.1" 500 11703

I tried running the container with --network=host, but no luck there either. I'm going to guess this is an issue on my end, and I need to create some iptables rules to allow a proper connection.

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

Can you exec into the container and read the contents of /shared/kubeconfig (don't paste the details here)? We just want to make sure that it's there and specifies a valid host - though I think Jon said he checked this.

Otherwise, what's your container setup? Docker/podman version and OS? There might be something amiss in the network as you suspect preventing the outbound connection.

from runwhen-local.

MuckerMayhem avatar MuckerMayhem commented on June 1, 2024

The config present seems valid in yaml, vscode doesn't see any issues.

As for my setup, I'm on Ubuntu:

5.4.0-150-generic #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

And running docker:

Docker version 23.0.5, build bc4487a

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

Internally the issue is tracked here: https://github.com/runwhen/platform-core/issues/1051 and is being worked on in this sprint.

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

@MuckerMayhem Are you able to get to the internet at all or access the cluster api endpoint by using wget?

https://docs.runwhen.com/public/runwhen-local/stuck-read-this.#check-network-connectivity

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

Feel free to also jump into Slack or our very fresh and new Discord server to chat live. Otherwise we can keep the thread going here. Totally up to you.

from runwhen-local.

MuckerMayhem avatar MuckerMayhem commented on June 1, 2024

Feel free to also jump into Slack or our very fresh and new Discord server to chat live. Otherwise we can keep the thread going here. Totally up to you.

Thanks! Apologies for the late reply as I've been on vacation until now. I plan to join the discord, was not able to join Slack as I don't have an account that can do so, and I'll check the internet connectivity when I'm back at work tomorrow.

from runwhen-local.

stewartshea avatar stewartshea commented on June 1, 2024

We've rolled additional error handling in 0.1.9 (https://github.com/runwhen-contrib/runwhen-local/releases/tag/v0.1.9) which should provide a better indication as to why a run might fail. Closing this issue now and we can re-open specific issues based on the output of any errors.

from runwhen-local.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.