ahmetb / cloud-run-faq Goto Github PK

Hi, some secrets are being loaded on startup, but I did not set the right permissions for the secret resources. I want to restart the revision/service/container so that it would try to get the secrets again.

App Engine Flexible has a way to "restart" it by deleting the instance. I wonder if Cloud Run has something similar, but I could not find it. Wondering if you guys have better luck figuring out how to do it.

How can I serve traffic multiple revisions? is written as below.

However, Cloud Run (currently) only supports serving traffic from the last healthy revision of your service. Therefore, it currently does not support revision based traffic splitting and canary deployments.

There is no description of healthy revision in Cloud Run FAQ.
I think that it is defined as Listening for requests on PORT in Cloud Run's container runtime contract and Cloud Run doesn't have L7 health checks.

The container must listen for requests on 0.0.0.0 on the port defined by the PORT environment variable.

Meta Requests of Knative's container runtime contract has more description but I am not sure that Cloud Run does support deploymentProbe/livenessProbe setting in REST API.

refs: https://cloud.google.com/run/docs/troubleshooting?hl=en

I am really new to cloud, can i get a tutorial, which shows step by step, how to deploy my django app backend to cloud run and then my front end to firebase

I'm pretty sure it's not currently possible but that's my goal : multi-region with Cloud Run, managed by Google infra + Google Cloud LB.

Cold start latency depends on many factors, however many users observe additional ~2 seconds latency on cold starts. [more user data needed!]

I just want to inform that with my container, which was 1.13GB, I experienced a cold start time of 40 seconds. The image was slimmed down with docker-slim and was hosted at the same region as the cloud run server.

I want to make my own API server from docker, but the user who can use this API server should be restricted by some conditions. For example, by specific URL referrer, by specific Android App and by specific iOS App like image below.

For example,

If my app has OAuth 2.0 + PKCE authorization code flow against a third party identity provider, can I guarantee that after the user has logged in on the third party's site and is redirected back they are redirected back to the same instance?

If they are not, the new instance they are redirected back to will know nothing about the code_verifier.

Some of my applications need to store configuration files on volumes inside the cluster (the Grafana container, for example) and I sometimes use volumes to store service accounts (Pub/Sub publisher and reader, storage owned and so on). I do that by configuring my volumeMounts and volumes on my deployment.yaml.

Without running Docker In Docker? Maybe by setting "concurrency" to 1 and stopping my container after handling 1 http request?

Anyone tried that? :D

The usecase is that I have a case that needs more "sandboxing" than usual, and running each http request in its own container seems useful (but slow, of course).

Does Cloud Run timeout apply on GKE-Cluster? If it does, will it always apply, or is this being looked in to?

Something along the lines of

language: node
install:
- npm install -g markdown-cli
script:
- markdown-toc -i README.md
- test -z "$(git diff | tee /dev/stderr)"

I was trying to find out how to set up test environment with Cloud Run to be available for selected users, for example i have front end application deployed to CloudRun, how is it possible to restrict permissions?
There is some information about logging in using curl and so on, but it all looks more like back-end authentication for specific routes, what about testing your app on mobile before releasing, google only mentions log in users sending requests with auth token headers and so on.
P.S Maybe there is another better solution to deploy to test environment and have a whitelisted IP addresses?

Thanks

I'm developing an image processing app for iOS and would like to keep the actual processing algorithm in the cloud. Based on my research Cloud Run looks like a perfect prospect.

I need to be able to send an image file from iOS app to Cloud Run container. Container should process the image and return processed image. I don't need to store the processed and the original image in the cloud.

I would really appreciate if somebody can clear up the below:

1. Is it possible to do this with just Cloud Run or I have to store the image in the Cloud storage?
2. How to send image file as a parameter to Cloud Run function and how to return processed image back to swift.
3. Do I have to use Vapor, Perfect or similar framework? If yes which is better?
4. The processing algorithm is divided into two parts:
   Part1: Image Segmentation written in python/pytorch, requires cuda/gpu support.
   Part2: Image Processing written in Swift5
   Do I need to have two separate containers for different codes? How would they communicate with each other?

Thank you.

Hi,

First or all, thanks for this useful GCR guide.

I'm currently looking for the best way to deploy microservices into GCE and thought GCR could be the best alternative for my use case, until I found this 32MB limitation.

Basically, what my microservices do is:

• Receive a video file between 5-100 MB
• Apply some filters / conversions
• Return a new video file between 5-100 MB

The conversion takes between 20 and 150 seconds.

At this moment, I have this service deployed on a GCE instance. However, I only receive about 50 requests / day, so I was looking for some way to save money an resources, and the scaling to zero feature of GCR attracted me.

Do you think GCR is a good alternative for my use case? If so, is there anything I can do to avoid the 32 MB limitation?

Thanks in advance.

As there is upper limit of running instances, it’s great if we could know the current number.

Hi @ahmetb,

I've got my own thoughts about this but I thought I'd get some second opinions about what is ideally containerised and deployed to Cloud Run. We currently have a large REST API, which is composed of numerous Lambdas / Cloud Functions. We want to move all our functions to Docker containers, as we don't like the vendor lock-in and have a need to use custom runtimes.

The question we're still deliberating over is whether it's good practice to package a single function inside one container, or to combine multiple functions (within the scope of their microservice) into one container. In other words, keeping the granularity of services at the function level, or packaging all related functions as a group of routes (for example, Express).

I understand this question does not apply solely to Cloud Run, but we've obviously considered things like cold starts, minimal dependencies, global variables, image sizes, and the maximum service quota for Cloud Run.

If what I've mentioned doesn't really make a difference at scale, and is really just an architectural decision, I'd be happy to know.

Cheers!

How to get client IP when using GRPC?

For example :
"X-Forwarded-For" header can be read to get client IP when using HTTP, Cloud Run HTTP proxy injects it.

https://twitter.com/ibrunotome/status/1117204999581388802

many users observe additional ~2 seconds latency on cold starts

You need actual data for this claim. Otherwise, I think it might hurt more than help, as many would assume there is always a 2 second overhead. That's not true: the infrastructure barely adds a few 100ms and a go binary will start without any issues under 1s.

Hi @ahmetb @steren ,

I have an actual domain but would like to map individual Cloud Run Services to a wildcard e.g. *.mydomain.com (so that I don't have to add individual records for each Cloud Run Service)

I am a bit confused as at why this is not working fine as I have added the correct DNS records.

Steren created https://github.com/steren/awesome-cloudrun

I understand both the repos are different, but let's make sure to link them somewhere. Both are great resources.

Hi

Is it possible to invoke a service from the same project with service accounts without having to implement this https://cloud.google.com/run/docs/authenticating/service-to-service?

I would like to run db migrations in cloud run entrypoint to avoid having to use an external tool run this before promoting a revision.

In kubernetes this functionality can be achieved by having an init container on the deployment and setting max surge to 1, ensuring only one pod attempts the migration before rolling out to other containers.

Is the rollout strategy of CloudRun defined anywhere? It would be nice if CloudRun waited for one container in a revision to be healthy before cutting over wholesale, which would achieve this purpose.

I understand that Cloud Run does not support Web Sockets, which makes it unusable for my application. But I could rearchitect using HTTP Server Sent Events or perhaps the Web Push API.

I have found some information on the GCP slack channel indicating Web Push API would be supported, but I was unable to find any information on whether Cloud Run SSE or any other push technologies are supported. Including them here would be awesome!

This describes the question: https://stackoverflow.com/questions/57689876/google-cloud-run-container-networking

My domains seem to now take forever to map, I tried using a VM Instance and setup SSL but Let's Encrypt tells me I have used up my rate limit which I believe is from my Cloud Run domain mappings.
Is there an option when this happens?

Hi,

At present I have a service in google cloud run that connected to others REST API/services to retrieve specific information. The other services need whitelist the public/external ip address from the source service (Google Cloud Run).

Can I assign a reserved public/external ip address from Google Cloud Networking to my Google Cloud Run service?

Regards,
Arsenio

cc: @steren per twitter thread.
I think we should have this behavior documented, especially if there's magic around it rather than just being unbounded concurrency.

Hi Ahmet, any idea when this feature will arrive (if at all?). We're managing our own database that whitelists IP address for obvious security purposes. I read the workaround, but the solution still feels hack-ish.

So, any idea if any official static ip will come by to Cloud Run? My alternative is to just host the docker instance in an instance group with static ip. Let me know if you think there's a better way. Thanks!

I saw in the documentation that there was a TODO section for Tracing. Has anyone figured out how to do this? I have created a Flask app and run it using gunicorn inside a docker container. I have added tracing for the flask application. I can run the container locally and ping the endpoints, they show up in Trace. The problem arises when I deploy to Cloud run, if I ping the endpoints there then I do not see the Traces appear.

I am curious as to if anyone here has got Tracing working within Cloud Run?

I've transitioned all of our frontend services to Cloud Run and love it. It'd be great to also move the API there as well but is using web sockets heavily.

Bumped into this FAQ repository which states:

WebSockets are currently not supported on Cloud Run. However, Cloud Run on GKE supports running applications capable of doing WebSockets.

It says that GKE supports running applications capable of doing websockets but doesn't quite clarify how.

What limitation does managed Run have that a GKE hosted solution does not suffer from?

The information on network pricing suggests, that outgoing network traffic costs (for NA-NA, EU-Eu as an example) ~0.1$ per GiB.
I'm wondering if that also applies to downloading from cloud storage via the official library?

I intend to use cloud run to render videos created by users (via ffmpeg and pupeteer).
These videos can contain custom assets, like further videos, from my current understanding I would have to download all of these assets on each render call, even if only minimal changes where made. This could become extremely costly, especially since cloud run instances only offer 2gb memory at most.

Is my understanding correct, and is compute engine a viable alternative?

Hi,

What are the alternatives to manage background or scheduled processes? For instances, I have a cron to run a process all days since 18:00.

Regards,
Arsenio

Hi, I'm not that familiar with Cloud Run, so I hope this isn't a dumb question. I see the README says:

Since you currently can’t use [Cloud HTTP(S) Load Balancer][https-lb] with Cloud Run, you cannot use Cloud CDN. However, this is subject to change soon.

I'm wondering if this has already changed? I see the following doc which makes it look like it is possible:

https://cloud.google.com/solutions/integrating-https-load-balancing-with-istio-and-cloud-run-for-anthos-deployed-on-gke

If it's not possible yet, is there a ticket or article we could link to with more details about it potentially coming soon?

I periodically see Cloud Run respond with 429s as it's spinning up new instances to serve requests. My cold start may take a couple seconds, it's not incredibly slow...

Anyone else experience this? How should I fix / diagnose this?

Hi,
I have several questions to understand scale to zero.

• How to approach best practices for cold start?
• If no traffic, does it kill the container, and spin up the new one if it's needed?
• How long the container up and running before it's stated idle and scale to zero? And how do I know this container is scaled to zero?

Thanks

Http/2 clients can send http2 to the google frontend which translates to http1.1 requests to my stateless server over http with special header. How can I tell google to send my server http2 requests instead?

thanks,
Dean

Hi,

Is there a way to create a security group to limit access to the container's network?

I would like to execute user-generated code and thus avoid serving as a proxy for any kind of abuse or attack.

Is there a kind of security group feature somewhere that will allow to apply network rules on my container ?

Thanks

This is Beta, but it looks like you can serve dynamic content via Firebase Hosting -> Cloud Run:

https://firebase.google.com/docs/hosting/cloud-run#direct_requests_to_container

It might be a good alternative since Cloud Run doesn't work with HTTP(S) Load Balancer.

Question: Is GCP IAP supported? Or support planned?

My goal is to host a reverse proxy like Envoy on Cloud Run, and proxy traffic behind that to other services. This seems cool because it can scale up and down dynamically, and it feels lighter weight than spinning up k8s just to host a reverse proxy.

So far I've only found one person that has tried this and written about it: https://blog.getambassador.io/experimenting-with-a-serverless-service-mesh-with-ambassador-on-google-cloud-run-c0b0da2130d1

Questions:

• is there already a hosted service that would be a better fit for this, without needing to run my own reverse proxy? Am I re-inventing the wheel? (I looked at Cloud Endpoints, but unless I misinterpreted it seemed pretty weak for what I'm hoping for. Traffic shadowing is a major feature I want, for example, and it didn't seem to exist there.)
• are there any API gateways or reverse proxies that might be a particularly good fit for this? I'm looking at stuff like Envoy, Nginx, Gloo, Ambassador, etc.
• any considerations that I might be missing here?

Now you can attach you cloud run to VPC using Serverless VPC access:
https://cloud.google.com/run/docs/configuring/connecting-vpc
(as of today this is a beta feature)

So I've made a cloud run revision that is executed when there's a new notification (OBJECT_FINALIZE) from a specific bucket (pubsub).
However, when I get in the revision's logs, every few seconds I see a new POST request even though no new objects were created/overwritten in that bucket.
https://i.imgur.com/ZreOtKd.png

And when there is a new object => I see that my code is executed like 5 times at the same time. Why is that?
https://i.imgur.com/SoL9JfT.png

I checked the output of gsutil notification list for the bucket (there is only one notification), and i have checked gcloud pubsub subscriptions list (there is only one subscription).
So what is causing this? I am honestly lost and annoyed because my code executes INSERT queries to the database and having duplicate values is silly...

If this is a result of the upscaling, then why does it upscale? I only tried having one request, this shouldn't cause my instance to create more instances...
CPU allocation is 2, fully managed, 512MiB memory
Max instances and concurrency settings are default

My service exports prometheus metrics on /_metrics but visiting that route will obviously pick a single instance.
I need to be able to query every instance every n seconds to gather all the metrics.

That being said, we recommend it strongly for portability,.

Hi,

I like the idea of Cloud Run but what if we get a ddos attack? Cost could explode.

Any recommendation for internet exposed service ?

Regards,
Maxence