DeepStack Crashing about bi-aidetection HOT 27 CLOSED

I'm trying the latest push, and everything is good so far. I disabled the telegram uploading because it seemed to be slowing down the time it took for AITool to process the images.

Do you think telegram uploading should have its own queue in another thread so the deepstack detection routine finishes faster and the telegram upload works in the background?

How about ALL actions have their own queue? How big should the max queue be? Are there any downsides to doing it this way that you can think of?

from bi-aidetection.

If you look in the JSON settings file there are several settings that can be tweaked related to this:

• HTTPClientTimeoutSeconds (30) - How long the HTTPClient waits for a its initial connection. Throws error "HttpRequestException" OR "TaskCanceledException"
• AIDetectionTimeoutSeconds (60) - The 'Task' timeout for the connection to the AI server. This throws error "TaskCanceledException" - "A task was canceled" error.
• MaxQueueItemRetries (10) - How many times the connection to AI server fails before disabling the URL
• URLResetAfterDisabledMinutes (30) - How long to wait to try disabled URL's again.
• MaxImageQueueSize (100) - the maximum number of new images allowed in the queue

So, your deepstack server is timing out after "AIDetectionTimeoutSeconds" causing that error. You could try increasing that number, but I think the server just needs restarting. I do know for sure that the GPU accelerated version in docker does stop responding to requests after a while in any case. A good test would be to have only ONE URL defined to 192.168.1.146:1234 and see if it eventually stops responding in the same way. That will tell us for sure if it has to do with having multiple servers set up or not, but I would guess it doesnt.

from bi-aidetection.

Ok, see if the latest is more stable. Each deepstack URL now has its dedicated HttpClient and it gets reused now to prevent too many connections from build up behind the scenes.
AITOOL-VORLONCD.zip

from bi-aidetection.

I have briefly tested the latest version, and it seems that it works now. I will let it run for a little bit to confirm the issue has been resolved.

from bi-aidetection.

I'm running two DeepStack instances on my desktop. One from Docker and the other is the Windows version of DeepStack from AITool. I seem to be having trouble with the Windows version of DeepStack. I tried reinstalling it and it didn't help. I'm not sure if starting the DeepStack server from AITool vs from the DeepStack UI makes a difference, so I'll be testing that next.

from bi-aidetection.

Can you describe what kind of trouble on the Windows version? Does it work at all or die after a while? What do logs show?
Perhaps enable the debug option on the deepstack tab and look at the deepstack log entries from just before it fails.
I seem to remember I had to reset this folder to fix a windows issue a while back:
C:\Users\username.deepstack

from bi-aidetection.

I would let AITool run for a few hours and come back to see a lot of images in the queue and the program stops processing images. If I use two DeepStack URLs, then the Docker version still processes images and I just get errors in my log from the Windows version of DeepStack having issues. I've just enabled the debug option and I've gone ahead and tried to reset that folder to see if that solved it. I'll let the program run a little longer to see if I could catch the log entries before a failure.

from bi-aidetection.

same ..:[16.09.2020, 08:30:58]: TelegramText>> ERROR: Could not send text "[16.09.2020, 08:30:50]: TelegramText>> ERROR: Could not send text "[16.09.2020, 08:30:50]: TelegramText>> ERROR: Could not send text "[16.09.2020, 08:30:46]: ImageQueueLoop>> * Not showing further warnings about skipping disabled URL: [URL unfurl="true"]http://192.168.2.1:280/v1/vision/detection[/URL]" to Telegram: Response status code does not indicate success: 429 (Too Many Requests). [HttpRequestException] Mod: <TelegramText>d__26 Line:1196:25" to Telegram: Response status code does not indicate success: 429 (Too Many Requests). [HttpRequestException] Mod: <TelegramText>d__26 Line:1196:25" to Telegram: Response status code does not indicate success: 429 (Too Many Requests). [HttpRequestException] Mod: <TelegramText>d__26 Line:1196:25

from bi-aidetection.

Your errors seems to also be related to telegram, which reminded me. A few versions back, I tried telegram uploading which caused AITool to process very slowly (AITool would be stuck on trying to send an image and not continue to process images until it was done uploading).

from bi-aidetection.

@syberx - try increasing the telegram specific cooldown in AITOOL.SETTINGS.JSON - "telegram_cooldown_minutes"
Make sure and close the program before changing the setting.
If you have a regular cooldown set, in theory that should override the telegram specific cooldown.

from bi-aidetection.

For some reason, AITool now crashes and freezes for me when I try to upload an image to Telegram. Had to reset the log file because it created a 2-gigabyte file. Not sure what the problem is. I also have the " Could not send text" error in my log.

from bi-aidetection.

@syberx @petermai6655 - I just changed a number of things to try to prevent telegram from sending messages too often. And normally when you get the 'Too Many Requests' it gives you a number of seconds to 'retry after' but there is a bug in the current version of the Telegram.bot library, so it never tells you how long to wait. I am now ASSUMING 5 minutes after every failure. (Telegram_RetryAfterFailSeconds in settings json file). You can also set the telegram cooldown on the settings tab. See how this is working:

AITOOL-VORLONCD.zip

from bi-aidetection.

For me, I do not see the "Too Many Requests" error. Instead, I get:

TelegramUpload> ERROR: Could not upload image (directory) to Telegram: Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=(token) or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
[FileLoadException] Mod: d__29 Line:1225:33
Trigger> Error: Could not load file or assembly 'System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=(token) or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) [FileLoadException] Mod: d__31 Line:1443:29
Trigger> -> ERROR sending image to Telegram.

I am able to upload images just fine with gentlepumpkin's version of AITool, so I assume my Telegram token is working.

from bi-aidetection.

@petermai6655 - Probably some file still blocked (unblock the zip before extracting) or a version conflict with json dll file. The latest eases the version requirements for the json file and and updates a few other DLL's to the latest versions.

AITOOL-VORLONCD.zip

from bi-aidetection.

@syberx @petermai6655 - I just changed a number of things to try to prevent telegram from sending messages too often. And normally when you get the 'Too Many Requests' it gives you a number of seconds to 'retry after' but there is a bug in the current version of the Telegram.bot library, so it never tells you how long to wait. I am now ASSUMING 5 minutes after every failure. (Telegram_RetryAfterFailSeconds in settings json file). You can also set the telegram cooldown on the settings tab. See how this is working:

AITOOL-VORLONCD.zip

Thanks this is working no error.

from bi-aidetection.

Not sure if installing Visual Studio and compiling to fix the Telegram issue might've caused this, but now I'm receiving errors from DeepStack again.

ERROR: A task was canceled. [TaskCanceledException] Mod: d__28 Line:875:40
(Error: No predictions? JSON: '{"success":false,"error":"invalid image"}')
ERROR: Failure response from DeepStack. JSON: '{"success":false,"error":"invalid image"}'

from bi-aidetection.

Probably unrelated. How is deepstack running? Tried restarting it lately?

I wouldnt expect to see TaskCanceled and failure response in the same sequence of events. Assume TaskCanceled was from further up the log?

Did it put the same image back into the queue more than once? It should have tried a number of times.

The json 'AIDetectionTimeoutSeconds' setting defaults to 60. Had you previously increased that when it was working?

The next version will have pre-verification to make sure the image file can be loaded by the program before giving it to deepstack. That way we will know if deepstack is telling LIES :)

from bi-aidetection.

For DeepStack, I tried restarting and even reinstalling and resetting it. And yes, TaskCanceled was from a different part of the log. The log is long, with multiple errors repeating, so I just looked for the unique errors and listed them.

I don't remember putting the same image back into the queue more than once, but I sometimes restart AITool and/or empty the image input folder. Not sure if it would cause this error if that occurred while the program is processing? However, the error seems to come up for new images so I'm not sure that's the cause of the problem. I'm using a slightly different method of sending images to the tool though. I'm using my camera's built-in motion detection to reduce CPU usage from DeepStack. Not sure if that affects anything? (Also, my cameras use underscores instead of periods in the file name, so I am wondering if that can be changed in AITool? Right now, I'm using a workaround where I'm adding a period to the camera name, but it would be nice to not have to do that.)

As for the AIDetectionTimeoutSeconds, I've used the default setting because changing them didn't help with the problem last time.

from bi-aidetection.

If you want to try it, the latest push will now verify the image can be read into memory and is a valid jpeg file before sending to Deepstack

from bi-aidetection.

I'm trying the latest push, and everything is good so far. I disabled the telegram uploading because it seemed to be slowing down the time it took for AITool to process the images.

I also noticed that some of my camera URLs we corrupt. Might've been because I was transferring the settings file between versions? I got this error:

CallTriggerURLs>> ERROR: Could not trigger URL 'person', please check if 'person' is correct and reachable: Could not find file 'C:\Video Storage\AITOOL-VORLONCD\person'. [WebException] Mod: CallTriggerURLs(List`1 trigger_urls) Line:1350:21

I looked at my cameras and some of them changed to "person" in the URL field instead of the correct URL. I changed it to the correct URL and still had the same issue. However, I resolved it by erasing the camera completely and adding a new camera.

from bi-aidetection.

'person' was accidentally set as the default URL for NEW cameras in the last push, should be fixed in latest up there.

from bi-aidetection.

Sorry for not getting back to you. Is there a difference between how Telegram messages are uploaded compared to gentlepumpkin's version of AITool? I haven't had the chance to update to the latest version, but it seems like adding Telegram uploading on your version sometimes causes problems. This doesn't seem to happen on gentlepumkin's version.

from bi-aidetection.

Telegram should be stable with latest release posted a few weeks ago. Or look for an even more recent version posted in a few issue threads here.

from bi-aidetection.

Reverted back to using my Docker version last week because I just couldn't get it to work. But now, I'm back to troubleshooting it and giving it a second shot (because I think it processes images faster). I compiled the latest commit (10-04-2020), so hopefully, I'll have greater luck with the latest version. It might be a problem with my version of DeepStack on Windows or compatibility issues with running both Docker and Windows versions?

AITool's log (DeepStack debug) is giving me these errors when starting DeepStack:
AI Tool Log.txt

from bi-aidetection.

I get all those same errors with debug enabled with the Windows version of deepstack running on the same machine as AITOOL.

...
Python.exe> ERROR: 2020-10-05 10:36:12.0732622 [W:onnxruntime:InferenceSession, session_state_initializer.cc:500 onnxruntime::SaveInputOutputNamesToNodeMapping] Graph input with name 318 is not associated with a node.
Redis-server.exe> [15576] 05 Oct 11:36:12 - Accepted 127.0.0.1:60348
Redis-server.exe> [15576] 05 Oct 11:36:13 - Accepted 127.0.0.1:60349
Redis-server.exe> [15576] 05 Oct 11:36:13 - DB 0: 404884 keys (0 volatile) in 524288 slots HT.
Redis-server.exe> [15576] 05 Oct 11:36:13 - 2 clients connected (0 slaves), 136510104 bytes in use

And when something is posted to it:
DeepStack> Server.exe> [GIN] 2020/10/05 - 11:37:31 | 200 | 11.2904664s | 127.0.0.1 | POST /v1/vision/detection

First, make sure when you start that you have "Detection API" checked. If you do, then try reinstall, and remove c:\users\username.deepstack. Double check when you go directly to the URL that it says activated and doesnt give an error.

The commit you used probably still has issues with dynamic masks, hope to correct later today.

[GIN]?
Go home deepstack, you are drunk. :)

from bi-aidetection.

You may be interested in the latest version - In ACTIONS it has a queue option so all actions are queued in a background thread and the main detection routine is released faster for deepstack to process the next image. The only side effect I'm still trying to work out is that the telegram cooldown may behave funny. Let me know how it works.
AITOOL-VORLONCD.zip

from bi-aidetection.

The windows version is depreciated and DeepStack is moving towards Docker. I'm expericing no problems with Docker.

from bi-aidetection.