Comments (6)
willgraf
commented on June 15, 2024
Idea: The cluster could not properly delete the bucket during destruction.
from kiosk-console.
dylanbannon
commented on June 15, 2024
kubectl logs frontend-deployment-[blahblah]-[blah] shows this:
2018-11-12T17:02:48.276Z [ERROR] [email protected] does not have storage.objects.delete access to deepcell-output/uploads/img_000000000_FITC_001.png.
I had just uploaded the FITC image. Same thing happens if I upload the far-red image.
from kiosk-console.
willgraf
commented on June 15, 2024
Just saw this myself, deploying on GKE:
2018-11-12T19:51:13.619Z [ERROR] [email protected] does not have storage.objects.delete access to deepcell-output/uploads/mibi_nuclear.png.
Mon, 12 Nov 2018 19:51:13 GMT express:router <anonymous> : /api/upload
2018-11-12T19:51:13.620Z [ERROR] middlewareError
Mon, 12 Nov 2018 19:51:13 GMT express:router <anonymous> : /api/upload
2018-11-12T19:51:13.622Z [INFO] ::ffff:10.24.2.7 - - "POST /api/upload HTTP/1.1" 500 148 "http://104.196.231.104/predict" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
RangeError: Invalid status code: undefined
at ServerResponse.writeHead (_http_server.js:199:11)
at ServerResponse.writeHead (/usr/src/app/node_modules/on-headers/index.js:55:19)
at ServerResponse.writeHead (/usr/src/app/node_modules/on-headers/index.js:55:19)
at ServerResponse._implicitHeader (_http_server.js:190:8)
at ServerResponse.end (/usr/src/app/node_modules/compression/index.js:103:14)
at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:221:10)
at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:267:15)
at app.use (/usr/src/app/dist/server/config/express.js:95:26)
at Layer.handle_error (/usr/src/app/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:315:13)
After changing the name of the file, the upload worked as expected.
After examining the service accounts and permissions via gcloud projects get-iam-policy $GKE_PROJECT_ID I found that indeed the service account created for the kiosk deployment (serviceAccount:[email protected]) has: role: roles/storage.admin
storage.admin should give the service account access to all bucket and object permissions. Why does this service account get a permission error?
from kiosk-console.
willgraf
commented on June 15, 2024
Deploying on AWS, I get a 500 error for all upload requests, even those that are new files.
Update: After a totally fresh install, I did not receive an error with AWS. Looks like @dylanbannon is right and there is some issue with teardown and service accounts that does not provide the proper storage access.
from kiosk-console.
dylanbannon
commented on June 15, 2024
Yeah, @willgraf, I'm not seeing this on AWS either, so I'm going to move forward with the understanding that this is a GKE-only issue.
from kiosk-console.
dylanbannon
commented on June 15, 2024
Fixed by #75. See there for full explanation of bug.
from kiosk-console.
Related Issues (20)
- Move Prometheus rules into a separate ConfigMap. HOT 2
- Deepcell.org deployment returns error on Predict console HOT 1
- New workflows may require changes to tf-serving and consumer settings.
- Separate configuration and tensorflow-serving containers in the tf-serving chart. HOT 1
- Docker run command in readthedocs
- Multizone clusters with GPU_NODE_MIN_SIZE=1 must have a GPU quota of at least 2. HOT 1
- Cluster created but HPAs failed to deploy HOT 1
- Change default name/options for available models HOT 2
- Error messages are not displayed for zips of images
- Creation screen sometimes gets disconnected from the process it's tailing.
- dnsSolver01 Service Account can have a maximum number of keys. HOT 1
- GKE 1.16.X is no longer supported
- Outdated instructions for installing Docker on google cloud Ubuntu VM
- Maximum batch size set by tf_serving pod instead of consumer HOT 1
- Consumer uses cached prediction if uploaded file name has been used previously
- Upgrade kiosk to run tensorflow 2.8 models HOT 1
- Credential authentication fails during GKE configuration HOT 2
- Bump Nuclear Segmentation and Tracking to version 7 HOT 1
- Read the Docs build requirements
- Google Cloud authorization failed during GKE configuration
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kiosk-console.