Comments (28)
In our case, serverless container app jobs were neither writing logs nor working but execution summary shows running state. Ran for many hours with no progress.
However, when I changed the queue name event is tied to and started the job manually, everything worked fine. This is a major hurdle at this point
Hope this gets patched over weekend as all our environments face this issue.
from azure-container-apps.
@ruvintri yes we do. The container is in fact running just fine - it interacts with Azure storage for example to store its results and we see that as expected. Just nothing in the Console logs since 6/12/2024. Very weird.
It seems almost like when you trigger them manually, there's some sort of context set that is not set when triggered by a queue.
from azure-container-apps.
Having the same issue. Deployed a simple container app job this morning and only the manually-triggered jobs show stdout or stderr logs.
My job doesn't (yet) contain logic. It has a few simple Python print() statements for testing purposes. It's triggering as expected based on the KEDA scaling rule for a Service Bus queue. It runs in the expected amount of time, but the output can't be found anywhere for the auto-triggered jobs.
from azure-container-apps.
Same here. When I start the job manually it works perfectly and finishes in success. All the log outputs works as intended. When the job is starts with the servicebus queue trigger it runs until timeout and fails with no console log. Only the system logs are shown and they report no unexpected errors except timeout.
I am actually questioning if I can trust this kind of service to deploy an application.
from azure-container-apps.
@GuiUzeda - questioning the same thing. Wondering if I should just go old school with a VM for these jobs. Ugh.
from azure-container-apps.
@GuiUzeda - questioning the same thing. Wondering if I should just go old school with a VM for these jobs. Ugh.
It is a little bit out there. There is a lot of unanswered issues here with triage tag. I have noticed that your original report is 3 days old. I am deploying an app in development for tests so no big issues for me but if that was to happen in production we would be in big trouble here.
from azure-container-apps.
We have fixed the issue, a hotfix is deploying, we expect to deploy to all impacted environments within 1 day.
from azure-container-apps.
I have patched all impacted environment, let me know anyone still meet issues.
from azure-container-apps.
@jedjohan - I noticed it yesterday too. No system logs, but console logs are back. I went to look for system logs because a job randomly crashed in the evening. I notice things seem to work much better during the day (PST) - I hope it's just because they're doing maintenance at night. I'm using the West US data center.
@jason-berk-k1x - Yea, this service is way not ready for prime time. It reminds me of when Azure first started 20 years ago, and they couldn't keep VMs up and running. Hopefully they'll figure this out too - glad I'm not in production, yet.
Summarizing, I've noticed the following issues:
- No way to create a support ticket, documentation is wrong or misleading. This repo is the only place I could find to log errors. Some serious tickets remain in "need triage" forever. Guess since this isn't "AI", Microsoft doesn't want to put any resources on it.
- "Max. Executions" doesn't work and relatedly, Container App Jobs can't scale beyond a dozen (or so) concurrent replicas. Documented in #1216. Try starting 20 jobs that each peg the CPU. Easy as pie to reproduce.
- Randomly missing logs (sometimes console, sometimes system).
- Unexplained execution failures, at night (I'm in PST, West US DC). No system logs to diagnose.
- Backoff errors in system logs do not explain the parameter that triggered them - no way to find out what caused the backoff.
- Replicas do not appear to use dedicated resources. 10 1-vCPU replicas starting at the same time running the same loop should run just as fast (actual time) as 1-vCPU replica. This is not the case. The 10-replica case runs much slower. This is really bad because it means that Microsoft is likely way overcharging for vCPU time. This may be related to the concurrent replica problem mentioned above.
- When starting many (e.g., 20) replicas (i.e., Container App Jobs), errors occur in pulling the replicas from the container registry.
Again, like 20 years ago, I'm left wondering... will Microsoft ever get its act together on this?
from azure-container-apps.
we're seeing something similar. seems to have started around tuesday. manual runs go correctly and produce logs, anything that is triggered from the queue reports as a failure and produces no logs for console.writelines or our custom logging client.
the queue activated runs appears to stop dropping system logs somewhere around the time it should be trying to pull the container. the delivery count is not incremented on our queue message on the failed runs, so it's unclear at what point this failing, but it seems to be pretty early on.
we tried recreating the app from the ground up, remade the queue, stripped out functionality and redeployed, but no luck. if i run the code directly on my local it functions as expected. we also downloaded the container image from our acr and ran it on a local docker and it also performed as expected.
from azure-container-apps.
@oligarchy - June 11 2024 was the last time we saw Console messages.
I've also seen failures in starting "too many" (e.g. 40) replicas simultaneously. You'll see 401 errors in the container system log coming from the container registry - it fails because I guess the container registry can only server a certain number of replicas starting simultaneously at a time.
We're compiling Rust code into an Alpine OS to create our container app btw.
from azure-container-apps.
@robrennie Do you see 'Created container .. ' and 'Started container ..' in your queue triggered job system logs?
We are seeing a similar issue in multiple environments including production and realizing that the containers are not actually starting, hence the empty console logs.
The only difference we can see in our logs is that the 'msi-transition' image that is pulled when starting the container has rolled back from tag 1.39.26-m to 1.0.8-m
This rollback occurred in the last 24 hours has caused all event triggered jobs to fail starting.
from azure-container-apps.
Hi, we have root-caused the missing logs to a platform regression: #1211
from azure-container-apps.
@vinisoto - #1211 did not fix this. I just ran a container app job, system log says everything was fine, no Console Log. In fact, ContainerAppConsoleLogs_CL still isn't even created.
'where' operator: Failed to resolve table or column expression named 'ContainerAppConsoleLogs_CL'
Request id: 14fab04f-1582-4890-8c99-93f0ef01319a
from azure-container-apps.
The amount of open issue, closed issues that never actually got fixed and overall horrible communication around contain apps makes me question how this made it out of preview.
The lack of communication and accountability is telling
from azure-container-apps.
As part of the retrospective, can anyone share where these changes are publicly discussed so we can be better prepared and have a chance at correlating platform changes to regressions. My entire non-prod set of subscriptions have been dead in the water since last Thursday
from azure-container-apps.
I have patched all impacted environment, let me know anyone still meet issues.
Thanks for your answer! Any redeploy needed? I am still having issues with container not starting:
from azure-container-apps.
from azure-container-apps.
from azure-container-apps.
I get no system or console logs:
from azure-container-apps.
It's been resolved for my jobs. Thank you!
from azure-container-apps.
Looking good for me too now. I can see Console Logs as expected. Thank you.
from azure-container-apps.
@jason-berk-k1x I do notice a significant delay between the replicas completing and the Console logs appearing - fooled me a couple times. I think it's due to the verbosity of my app though - and will fix that anyways. Also, I tend to remove the filter on the default Log Analytics query and rerun it and that's usually when everything (or something) appears.
from azure-container-apps.
@robrennie did you have to recreate the job or anything?
from azure-container-apps.
@jason-berk-k1x I did not, just ran it again and it worked.
from azure-container-apps.
Since yesterday 14:00 UTC my Systemlogs are no more :( Hundreds of executions for 4 ACJs in one specific environment with no data for ContainerAppSystemLogs_CL
from azure-container-apps.
@jedjohan isn't it funny you can see console logs....meaning the container is running, but get no system logs! I've been fighting this issue for literally MONTHS!!!!
from azure-container-apps.
@jedjohan isn't it funny you can see console logs....meaning the container is running, but get no system logs! I've been fighting this issue for literally MONTHS!!!!
Mine are back again wohoo. Just went missing for about 24h.
from azure-container-apps.
Related Issues (20)
- Container App Jobs do not scale beyond 10-12 replicas concurrently reliably HOT 4
- Error when a new revision aprovisioning HOT 2
- Feature Request: Smaller/Focused CI/CD Tooling (Azure CLI size is unreasonable for CI/CD usage) HOT 6
- Error deploying new revision HOT 11
- Error `ManagedEnvironmentInvalidNetworkConfiguration` when creating internal Azure Container App Environment HOT 2
- unable to get github runner agent to run HOT 1
- The traffic is not happening between Azure Application Gateway(Public) to the Internal Azure Container Apps. HOT 7
- Container App Job Replica Scales Down In Middle of Processing HOT 1
- Can't enable aspire dashboard HOT 5
- [KEDA][AzureEventHub] App not scaling to zero HOT 6
- Cannot create more than 10 containerapp environment in a subscription HOT 4
- Container revisions will not start HOT 21
- az containerapp update --yaml fails with scale rule with user-assigned identity HOT 5
- Azure Portal not working correctly when routing all traffic to Azure Firewall HOT 1
- Aspire Dashboard: Could not authenticate user with requested resource. HOT 35
- Feature Request: AppArmor, Seccomp profiles, and capability dropping support HOT 2
- Provisioning state failed from private ACR with User managed identity HOT 4
- Crashed Container App Job Causes Infinite Loop in KEDA Scaler HOT 6
- Missing System Logs (yet again), Container App Jobs HOT 2
- Aspire Authorization: Microsoft Entra External ID HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-container-apps.