[BUG] Companion crashing with ERR_STREAM_DESTROYED error about companion HOT 10 CLOSED

3.4.0+7035-main-5f79fdbc or later in the betas. v3.3.1 is in the pipeline for release by the end of the week.

from companion.

No worries, we will run up the beta build and confirm it is solved.

Kudos both for the speedy attention.

from companion.

Any idea on what could have been going on when this happened?
Unfortunately this error is incredibly vague (it almost just says 'an error occured'), so it might require a lot of trial and error to figure out where this is coming from

from companion.

At the moment, no. The companion logs I looked at afterwards werent especially revealing. There were some logged errors from modules, but nothing scary, and some entries about a streamdeck being disconnected (via satellite). I am not especially convinced any of these are contributing factors. I suppose the other potentially noteworthy factors:

• Its running on a VM inside a Windows Server instance
• It is a system we are testing with a centralized companion instance and satellite for stream decks around the facility. So its not in production yet, and fingers crossed we haven't seen this issue in our production systems.
• It has quite a few more modules than any of our production systems, and some of those modules are in dev, so probably have their own bugs.
• The error message was identical both times it crashed today, with the same code line references.

As a test and dev system for now we can keep running and monitoring to see how it goes. But does Companion write a stack trace anywhere on crash that we can access and share? Or can we switch on a higher level of internal logging?

2024-06-04T01:03:58.872Z error Surface/Handler/CL21L2A02850 Satellite StreamDeck: xl disconnected
2024-06-04T01:03:58.873Z info Service/Satellite/::ffff:172.20.145.69:49921 connection closed with 1 connected surfaces.

Logs, such as they are, attached.

Companion Logs.zip

from companion.

This would seemingly have to be during an attempted file write operation, possibly not having write permissions to a directory? Since this is a dev VM, I'd try launching Companion as an admin and see if that resolves it. Maybe also (before doing that) compare the files and folders in the config directory to those on a Win desktop and see if there's a noticeable difference or something missing on the dev server.

from companion.

With some help from @krocheck I think we have figured out where this is coming from.
So this is fixed in the betas and will be in 3.3.1 when that is done

from companion.

from companion.

@krocheck is there still value in testing Companion run at Admin level today? Happy to do that if it will be of any use.

from companion.

@phillipivan, @Julusian was able to reproduce in a test case once we found where this could come from (an outside module for log rotation). I'd prefer to see if it goes away with a patched version than muddy the waters, since running as admin could null the file system level condition that's causing normal operation to throw the error. At minimum, there should be a Companion log for the error condition rather than an exception thrown.

from companion.

We have not encountered this issue today running either the beta or 3.3.1.

from companion.