Container revisions will not start about azure-container-apps HOT 21 CLOSED

An update from the original reporter. We have mitigated this issue for now by forcing our containers to never scale to zero, but we are fearful of those containers being force restarted by Azure.

If it helps, the only way we seem to be able to get those services back online, was to repeatedly scale the revision up and down in terms of minimum instances. Even then, it took sometime 15 minute plus for the revision to activate.

From our perspective, this was a really challenging issue because we had no way to diagnose what was happening other than looking at the limited logs that were (sometimes) being generated by the system console of a container, even then we didn't get any information that pointed to what was happening.

When a container eventually failed to activate, the azure portal had no information in any of the logs, meaning we were pretty stuck.

This seems like a capacity issue, which is (I think) why restarting the containers again and again eventually got our containers online.

from azure-container-apps.

This issue is happening on our various container apps environments as well (dev, quality, production). We are also using the consumption model and are hosting this in West Europe. We opened a Microsoft support case, hopefully it gets resolved soon.

from azure-container-apps.

There might be a few more regions which exhibited this behavior we're mostly cleaned this up across the globe. Please open a new issue if it should pop up again. Thank you all for being patient/diligent/friendly with us.

from azure-container-apps.

Thanks for raising this. Can you please send the blocked out content to acasupport at microsoft.com. We would need your subscription, app and environment name please.

from azure-container-apps.

acasupport at microsoft.com

Hi @simonjj I've sent that detail across. Thanks

from azure-container-apps.

I'm having the same issue, also in UK South.

from azure-container-apps.

I am also seeing the same problem on multiple container apps in the uksouth region

from azure-container-apps.

Just to update the thread. We're investigating this issue and will update once it's resolved.

from azure-container-apps.

same issue here in West Europe

from azure-container-apps.

in our case reducing the max number of container instances resolves the problem. With 3 instances we would end up with 6 cores, which is more than the allowed 4:

With max 2 container instances, both container instances start without any problems.

from azure-container-apps.

scratch that thought, 3 instances running now:

Looks like something got fixed in Azure

from azure-container-apps.

Any update on this? This is a major problem..

from azure-container-apps.

I'm seeing something very similar.. specifically errors such as this:

{"TimeStamp":"2024-07-12 14:46:40 \u002B0000 UTC","Type":"Normal","ContainerAppName":"{CONTAINERAPPNAME}","RevisionName":"{REVISIONNAME}","ReplicaName":"{REPLICANAME}","Msg":"Replica {REPLICANAME} has been scheduled to run on a node.","Reason":"AssigningReplica","EventSource":"ContainerAppController","Count":0}
{"TimeStamp":"2024-07-12 14:47:23 \u002B0000 UTC","Type":"Warning","ContainerAppName":"{CONTAINERNAME}","RevisionName":"{REVISIONNAME}","ReplicaName":"","Msg":"ScaledObject doesn\u0027t have correct triggers specification","Reason":"ScaledObjectCheckFailed","EventSource":"KEDA","Count":9}

While terraform is used to deploy the containers, nothing has been changed with min/max replicas. Everything that has been deployed/changed since July 9th, the new revision isn't deploying and sits in a state of activating showing 0/0 ready with -Infinity restarts.

Increasing to 1 min/2 max did not fix my issue

Edit: If it matters, our CAE and containers are on the consumption model.

from azure-container-apps.

The only way we were able to get the container to activate was to delete the container app and redeploy, the issue did eventually reappear when the container scaled to 0 and then tried to scale back up, but it's been intermittent, its been working "ok" today

from azure-container-apps.

This issue should be resolved now. The impact should have been limited to uksouth. Please notify us here if there continue to be issues with spinning up new revisions or replicas.

from azure-container-apps.

This impacted us in Central US as well. Looks to be resolved now

from azure-container-apps.

We are facing this issue here in Western Europe.

from azure-container-apps.

Same issue on all environments in West-Europe. Consumption workload profiles

from azure-container-apps.

We have the same issue in all environments in West Europe. New replicas are not activating.

from azure-container-apps.

I'm also having this issue in West-Europe, all my replica's are down and won't start. Scaling up and down like somebody suggested seems to work sometimes.

Seeing this error:

"ScaledObject doesn't have correct triggers specification","Reason":"ScaledObjectCheckFailed","EventSource":"KEDA"

from azure-container-apps.

I also had this issue with a customer's prod deployment in West Europe yesterday. Only thing that helped was throwing the whole CA away and redeploying (which I know might not be feasible for some unfortunately).

from azure-container-apps.