using IR LED and mono cameras for skeleton detection about depthai_blazepose HOT 15 OPEN

@stephanschulz

Since i do right.out.link(mono_manip.inputImage) to create the image blazepose gets handed i assume i should also do stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT).

That's what I would have tried too.

Do you think lower the threshold and setting subPIxels to True helps to improve the chance to find skeletons?
I don't think so. Modifying these values will impact only the stereo processing result but has no impact on the body detection or landmarks regression.

If you use the depthai demo, you can have a feeling about the influence of these values:
https://docs.luxonis.com/en/latest/pages/tutorials/first_steps/#depth-properties

from depthai_blazepose.

You have to modify the pipeline in order to replace the ColorCamera node with a MonoCamera node and slightly modify the ImageManip nodes to convert the mono frame to a color frame (because the NeuralNetwork nodes after the ImageManip nodes take color frame as input). I think you should need to add something like this:
manipRight.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p)

IBut it is not guaranted that the models work well on mono frames. I know for instance that hand palm detection does not work well on mono frames.

from depthai_blazepose.

Thanks for the hint.
This sounds pretty involved to make these changes.

I guess i can pick for MonoCamera the left or right camera.
They already get set in line 249.

right = pipeline.createMonoCamera() right.setBoardSocket(dai.CameraBoardSocket.RIGHT) right.setResolution(mono_resolution) right.setFps(self.internal_fps)

Would i need to comment out the right camera pipeline since i am going to use it as my main camera already?
But it is used in the right.out.link(stereo.right) stereo camera output link.

I can't just simple change cam = pipeline.create(dai.node.ColorCamera) to cam = pipeline.create(dai.nodeMonoCamera) because the MonoCamera does not have setInterleaved, setIspScale, setVideoSize, setPreviewSize, etc

As you already mentioning i need to convert one of the MonoCamera images in to a RGB to hand to balzepose.
I just do not know how to setup a manipRight image properly.

i get AttributeError: 'depthai.node.MonoCamera' object has no attribute 'initialConfig'

IBut it is not guaranted that the models work well on mono frames. I know for instance that hand palm detection does not work well on mono frames.
I have tested blazepose on greyscale video before and it works pretty well.

I wonder if your "DepthAI Pipeline Graph" can help me set this up easier?

from depthai_blazepose.

"DepthAI Pipeline Graph" will just help you to visualize the nodes and the connections between the nodes.
An easy way to check if blazepose works well on mono images is the following:

• make a color video (with any camera);
• convert the video to grayscale (for instance by using this method: https://davidwalsh.name/convert-video-grayscale);
• run blazepose in host mode with the grayscale video as input : ./demo.py -i video.mp4

I think the error you get (AttributeError: 'depthai.node.MonoCamera' object has no attribute 'initialConfig') because you are trying to apply .initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p) to a MonoCamera node instead of ImageManip node, no ?

from depthai_blazepose.

@geaxgx Hello, I also used the grayscale camera to load the algorithm of blazepose, updated the pipeline, and added color_ The cam node is changed to mono_ Cam, and added the operation of gray2bgr, but the output detection frame and posture point are not on my body, but on my upper left, and the posture point is too jittery. I changed the resolution of the internal frame to 720p, but the effect has not changed. May I ask if my changes are not complete enough? I hope to get an early reply. Thank you

from depthai_blazepose.

@shamus333 I can't say without having a look at your code.
Have you an output detection output to demonstrate the behaviour ?

from depthai_blazepose.

@geaxgx OK, I have modified part of the code, and now it is uploaded to you in the form of an attachment. The running instruction is Python 3 demo py -e --lm_ M Lite, depthai version is 2.17.0.0. The phenomenon is shown in the picture, （The source file is too large. I only uploaded the modified file to you. You can directly overwrite the modified code on the basis of the source code）

modify.zip

from depthai_blazepose.

It seems to be a problem with the padding. I am busy and can't spend time on your problem now but I had a quick glance to your code.
It is maybe related to this line:
mono_manip.initialConfig.setResize(300, 300) which crops the frame. Can you try without it ?

Also can you give me the result printed for this line:
print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}") ?
The call to mpu.find_isp_scale_params() does not make sense in the context of a monochrome camera.

from depthai_blazepose.

Thanks @shamus333 for posting your code. It is very helpful to see how ti use mono_manip.
here is my repo: https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono/BlazeposeDepthaiEdge.py#L270

print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}")
prints out
Internal camera image size: 1152 x 648 - pad_h: 252

I did remove mono_manip.initialConfig.setResize(300, 300)

FYI i am working on the OAK-D Pro POE not the USB version.

I commented everything related to cam = pipeline.create(dai.node.ColorCamera) since pose and depth detection is only done on the mono camera feeds.

 mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_400_P
            left = pipeline.createMonoCamera()
            left.setBoardSocket(dai.CameraBoardSocket.LEFT)
            left.setResolution(mono_resolution)
            left.setFps(self.internal_fps)

            right = pipeline.createMonoCamera()
            right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
            right.setResolution(mono_resolution)
            right.setFps(self.internal_fps)

            # Define mono camera Gray2BGR
            print("Creating Gray2BGR Camera...")
            mono_manip = pipeline.createImageManip()
#            mono_manip.initialConfig.setResize(300, 300)
            mono_manip.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
            right.out.link(mono_manip.inputImage)
            
            mono_manip_out = pipeline.create(dai.node.XLinkOut)
            mono_manip_out.setStreamName("mono_manip_out")
            mono_manip_out.input.setQueueSize(1)
            mono_manip_out.input.setBlocking(False)
            mono_manip.out.link(mono_manip_out.input)
            
            print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}")
        
            stereo = pipeline.createStereoDepth()
            stereo.setConfidenceThreshold(230)
            # LR-check is required for depth alignment
            stereo.setLeftRightCheck(True)
#            stereo.setDepthAlign(dai.CameraBoardSocket.RGB)
            stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT)
            stereo.setSubpixel(False)  # subpixel True -> latency

I too see the offset even though i am doing on right.out.link(mono_manip.inputImage) and output mono_manip.out.link(mono_manip_out.input)

I also see this SpatialLocationCalculator(6) error
`
Here as a graph

from depthai_blazepose.

i have tested the monoCamera + tracking in the dark outside and it works OKish. It works best when using the heavy model.'
But i am still not certain if the stereo calibration (know that i took out all the RGB camera stuff) is accurate. I see the Z values jump around a lot. And maybe the fact that the mono_manip image is a different size (which size?) than the left+right stereo images could also mean wrong depth data?

https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L323

from depthai_blazepose.

@stephanschulz I think you should align the stereo depth frame with the mono camera instead of the color camera:
https://github.com/stephanschulz/depthai_blazepose/blob/81f17065cc9c051ba0e69db16340396691083e77/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L342

from depthai_blazepose.

@geaxgx thanks for the tip.
Visually this still does not fix the offset between IR image and skeleton.
But i guess it results in better depth data?

The options seem to be AUTO, RGB, LEFT, RIGHT
https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.CameraBoardSocket

Since i do right.out.link(mono_manip.inputImage) to create the image blazepose gets handed i assume i should also do stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT).

I did notice that detection and tracking is much better on the RGB image, probably due to the fact that the model was trained on RGB data. But i need to use it outside in the dark.
Since the resolution of the right image is much lower than the RGB image
Do you think lower the threshold and setting subPIxels to True helps to improve the chance to find skeletons?
setConfidenceThreshold(50)
stereo.setSubpixel(False)

from depthai_blazepose.

I am now trying to increase the monocamera resolution, which i think is done with mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_400_P. THE_480_P also works.
But when i try the other options
mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_720_P
mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_800_P
it get this error message:

[18443010C121C61200] [10.100.0.21] [147.942] [ImageManip(3)] [error] Output image is bigger (2764800B) than maximum frame size specified in properties (1048576B) - skipping frame.
Please use the setMaxOutputFrameSize API to explicitly config the [maximum] output size.

Maybe the error does not come from the left or right camera but rather from mono_manip?
https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L318
But if i try to set the image size with mono_manip.initialConfig.setResize(self.frame_size)# setResize(1280, 800)
i get this error

Traceback (most recent call last):
  File "demo_osc.py", line 60, in <module>
    tracker = BlazeposeDepthai(input_src=args.input, 
  File "/Users/stephanschulz/Documents/depthai-python-2/depthai_blazepose/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py", line 226, in __init__
    self.device.startPipeline(self.create_pipeline())
  File "/Users/stephanschulz/Documents/depthai-python-2/depthai_blazepose/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py", line 320, in create_pipeline
    mono_manip.initialConfig.setResize(self.frame_size)# setResize(1280, 800)
TypeError: setResize(): incompatible function arguments. The following argument types are supported:
    1. (self: depthai.ImageManipConfig, w: int, h: int) -> depthai.ImageManipConfig
    2. (self: depthai.ImageManipConfig, size: Tuple[int, int]) -> depthai.ImageManipConfig

from depthai_blazepose.

@stephanschulz When you use an ImageManip node, you must specify the maximum output size, like it is done there:

from depthai_blazepose.

that works.
obvisouly it slows down everything. with the heavy model i am now getting 3.4 fps.
thanks again.

from depthai_blazepose.