Comments (14)
@MuckerMayhem Thanks for sharing - and apologies for the issue.
The version you have should hav the code responsible for handling the scenario where namespaces could not be listed but projects could (a common tenant configuration in OpenShift). I will test against our own development cluster and get back to you if I have further questions.
from runwhen-local.
@MuckerMayhem just checking in - did you manage to make any progress or are you still stuck?
Hi, unfortunately I haven't had time to look at this, but my goal is to sit down and look through it tomorrow. I'll also pop in Slack, just in case I have questions.
from runwhen-local.
Adding some notes to this issue from a recent troubleshooting session with a user - enhanced exception handling would be great.
- Kubeconfig was valid and able to list projects but not namespaces
- user was on openshift
Initial error when running:
"Error 500 from Workspace Builder service for command "run": None
Logs contained:
'Internal Server Error: /run/'
from runwhen-local.
Hi, I'm the mysterious user in this case.
This is what I get for the sha on the image:
~$sudo docker inspect --format='{{index .RepoDigests 0}}' 8d750494f2c1
ghcr.io/runwhen-contrib/runwhen-local@sha256:9a0337c97e7a6edc066a3ae1fcacc882949ade9c9f84d22ff75378b07375024e
from runwhen-local.
FWIW, additional error handling is on the list, but for now you can docker attach to the container to see more details;
The following is an example from my ocp demo host;
# Check access to projects
$ KUBECONFIG=openshift-demo-sa-kubeconfig oc get projects
NAME DISPLAY NAME STATUS
default Active
# Verify I don't have permission to list namespaces
$ KUBECONFIG=openshift-demo-sa-kubeconfig oc get ns
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:default:demo-sa" cannot list resource "namespaces" in API group "" at the cluster scope
# docker attach RunWhenLocal
Resetting neo4j models
Error scanning for namespace instances; skipping and continuing; error: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e25350e6-7cc7-4f9e-b811-6cbbed9c2611', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f12e0a43-09f3-43e6-8910-d858a7fd1295', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5341b65f-8932-4cce-881b-99952354ea3d', 'Date': 'Thu, 13 Jul 2023 13:47:33 GMT', 'Content-Length': '292'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"system:serviceaccount:default:demo-sa\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}
Scanning for Kubernetes resources in namespace "default"
Level of detail lookup failed; defaulting to full LOD; resource type=<class 'models.KubernetesCluster'>
Level of detail lookup failed; defaulting to full LOD; resource type=<class 'models.KubernetesCluster'>
Rendering output item: workspaces/my-workspace/slxs/default-pod-resources/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pod-resources/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/adservice-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/adservice-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/cartservice-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/cartservice-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/chcktsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/chcktsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/crrncysrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/crrncysrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/emlsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/emlsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/frontend-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/frontend-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/ldgnrtr-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ldgnrtr-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/pymntsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/pymntsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/prdctctlgsr-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/prdctctlgsr-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/rcmmndtnsrv-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/rcmmndtnsrv-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/shppngsrvc-depl-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/shppngsrvc-depl-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-image-check/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-image-check/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/slo.yaml
Rendering output item: workspaces/my-workspace/slxs/default-ns-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-redis-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/redis-cart-redis-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-sa-check/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-sa-check/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pvc-health/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/default-pvc-health/runbook.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-cpu/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-cpu/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-mem/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-mem/sli.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-disk/slx.yaml
Rendering output item: workspaces/my-workspace/slxs/ocp-disk/sli.yaml
Rendering output item: workspaces/my-workspace/workspace.yaml
[13/Jul/2023 13:47:36] "POST /run/ HTTP/1.1" 200 22386
[13/Jul/2023 13:50:52] "POST /run/ HTTP/1.1" 200 22390
INFO - [13:50:52] Detected file changes
INFO - Building documentation...
WARNING - Config value 'build': Unrecognised configuration name: build
WARNING - Config value 'dev_addr': The use of the IP address '0.0.0.0' suggests a production environment or the use of a proxy to connect to the MkDocs server. However, the MkDocs' server is intended for local development purposes only. Please use a third party production-ready server instead.
INFO - Documentation built in 0.18 seconds
INFO - [13:50:53] Reloading browsers
INFO - [13:50:53] Browser connected: http://localhost:8081/
INFO - [13:50:56] Detected file changes
INFO - Building documentation...
@MuckerMayhem can you docker attach to the container and share any helpful output there? My test indicates that the code should properly scan your cluster - though maybe we are having a problem with the kubeconfig / access? No need to share sensitive data, just generic output. If you want, can you also share your workspaceInfo.yaml file? Feel free to jump into slack if you want to debug together, otherwise we can keep using this thread. The link to join slack is in the top header of https://docs.runwhen.com
from runwhen-local.
@MuckerMayhem just checking in - did you manage to make any progress or are you still stuck?
from runwhen-local.
So it seems my issue appears to be network related:
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1d11280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1d118b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f03b1b15a00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /apis/
Internal Server Error: /run/
[21/Jul/2023 18:53:46] "POST /run/ HTTP/1.1" 500 11703
I tried running the container with --network=host
, but no luck there either. I'm going to guess this is an issue on my end, and I need to create some iptables rules to allow a proper connection.
from runwhen-local.
Can you exec into the container and read the contents of /shared/kubeconfig (don't paste the details here)? We just want to make sure that it's there and specifies a valid host - though I think Jon said he checked this.
Otherwise, what's your container setup? Docker/podman version and OS? There might be something amiss in the network as you suspect preventing the outbound connection.
from runwhen-local.
The config present seems valid in yaml, vscode doesn't see any issues.
As for my setup, I'm on Ubuntu:
5.4.0-150-generic #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
And running docker:
Docker version 23.0.5, build bc4487a
from runwhen-local.
Internally the issue is tracked here: https://github.com/runwhen/platform-core/issues/1051 and is being worked on in this sprint.
from runwhen-local.
@MuckerMayhem Are you able to get to the internet at all or access the cluster api endpoint by using wget?
https://docs.runwhen.com/public/runwhen-local/stuck-read-this.#check-network-connectivity
from runwhen-local.
Feel free to also jump into Slack or our very fresh and new Discord server to chat live. Otherwise we can keep the thread going here. Totally up to you.
from runwhen-local.
Feel free to also jump into Slack or our very fresh and new Discord server to chat live. Otherwise we can keep the thread going here. Totally up to you.
Thanks! Apologies for the late reply as I've been on vacation until now. I plan to join the discord, was not able to join Slack as I don't have an account that can do so, and I'll check the internet connectivity when I'm back at work tomorrow.
from runwhen-local.
We've rolled additional error handling in 0.1.9 (https://github.com/runwhen-contrib/runwhen-local/releases/tag/v0.1.9) which should provide a better indication as to why a run might fail. Closing this issue now and we can re-open specific issues based on the output of any errors.
from runwhen-local.
Related Issues (20)
- Doc Bug Architecture.md HOT 3
- Doc Bug Getting_Started-Running_Locally.md HOT 2
- Enhancement - Getting_Started-Running_Locally.md - SELinux HOT 2
- Make workspace builder aware of CloudQuery premium vs free tables
- Design document for handling CloudQuery premium tables in open core source plugins
- Include information about gen rules that are suppressed in warning for disallowed access to a CQ premium table
- Add mechanism to report warnings during successful execution of workspace builder
- Dump/load support for resource database in workspace builder
- Unit test framework for the generation rules code in the workspace builder
- Tool to validate all of the generation rule files in a code collection against the JSON/YAML schema
- [runwhen-local-feedback] Test and document proxy configuration HOT 11
- [runwhen-local-feedback] Cannot run in OpenShift - errors with port 8081
- [runwhen-local-feedback] SLX names include "|" HOT 1
- [runwhen-local-feedback] Upload to RunWhen Platform / Missing secrets / configProvided
- [runwhen-local-feedback] Create slim image
- [runwhen-local-feedback] Rewrite homepage - align with the codecollection registry at registry.runwhen.com
- [runwhen-local-feedback] Fix dark mode css on new UI
- [runwhen-local-feedback] Error 500 from Workspace Builder service for command "run": join() argument must be str, bytes, or os.PathLike object, not 'list'
- [runwhen-local-feedback] fix kubeconfig generation with helm chart customized name
- Support for assuming roles in the AWS indexer for the workspace builder
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from runwhen-local.