pipeless-ai / pipeless Goto Github PK

When the frame processing time takes longer than expected the sockets queues may discard messages. Those messages include frames but also messages used for the communication between components since they are sent via the same socket.
Due to the above, if the socket buffer discards the EOS message, the pipeline never ends.

We have two options right now:
1.- Allow to configure the size of the socket buffers so the user can update it depending on how long the user application takes to return each frame.
2.- Use different sockets for components communication. This may lead to complex synchronization since, for example, we won't know which is the last frame and may close the pipeline too early.

Right now when running using all the logs are a bit messy. It would be more clear to add the component name to the logger.

Describe the bug
Following the Container guide.
Running the init command:
docker run --rm -it -v /home/nvidia/pipeless:/app miguelaeh/pipeless init my_project
yields:
Creating Python virtual env... thread 'main' panicked at src/cli/init.rs:10:34: calledResult::unwrap()on anErrvalue: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } note: run withRUST_BACKTRACE=1environment variable to display a backtrace
The /home/nvidia/pipeless folder has been created and had the permissions set with:
chgrp -R root /home/nvidia/pipeless

To Reproduce
Follow the guide at: https://www.pipeless.ai/docs/docs/v1/container
Every command up until the init ran smoothly.
Interestingly, I'm able to start a pipeless instance successfully:
docker run --rm -v /your/project/dir:/app miguelaeh/pipeless start --stages-dir /app

`
Creating Python virtual env...

(pipeless:1): GStreamer-WARNING **: 09:22:28.059: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
[2024-04-09T09:22:31Z INFO pipeless_ai::stages::parser] ⚙️ Loading stages from /app
[2024-04-09T09:22:31Z INFO pipeless_ai::config::adapters::rest] REST adapter running
[2024-04-09T09:22:31Z INFO warp::server] Server::run; addr=0.0.0.0:3030
[2024-04-09T09:22:31Z INFO warp::server] listening on http://0.0.0.0:3030
`

Expected behavior
A pipeless project to be initiated.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu Bionic Beaver
• Version: 18.04.3 LTS
• Processor arch: aarch64 (Jetson Nano)

Right now, Pipeless needs to be restarted to load new stages or load changes on a stage code.
The purpose of this issue is to implement a dynamic approach for both cases:

1. Pipeless should be able to load and unload stages dynamically once running. This will improve resource consumption and production management experience.
2. Pipeless should be able to automatically re-load a stage when its code has changed. This will improve development experience.

We are loading the user code per hook file. This does not allow the user to split the code of a hook into multiple files.
It is a common and desired practice to separate utils functions from the "main" logic of the hook for readability, so we must support it.

Hi @miguelaeh, wonderfull job. I readed source code and found all stages processed in execute_path，so fps will be very low when stages take a lot time, Maybe one stage have one thread，then the fps will be restricted with the most time-consuming stage, not all stages.

The dispatcher is never removing the managers from its hashmap, maintaining them even when they are no longer required.

This issue is not perceived by users directly, however, it leads to consume more memory than required.

Right now, the output receives a EOS message from the input when it finishes processing the stream. However, this produces a high number of discarded frames at the end of the stream.

The EOS message should be sent to the output by the workers instead to avoid the premature end of stream.

When saving the video as .mp4 the bitrate decreases with respect to the original video. Even though the bitrate property is being updated on the output x264enc when new tags appear, it doesn't seem to take effect.

Describe the bug

1. The hyperlink for the "Multipose Detection with TensorFlow" example in the documentation's "Examples" category is pointing to the wrong page, resulting in a 404 Page Not Found error.
2. Additionally, attempting to edit any page on the documentation website using the "Edit this page" link leads to a Not Found page, making it impossible to contribute changes.

To Reproduce

1. Navigate to the "Examples" category on the left side of the docs website.
2. Click on the "Multipose Detection with TensorFlow" hyperlink.
3. Observe the 404 Page Not Found error.
4. Click on the "Edit this page" link for any sub-page on the documentation website.

Expected behavior

1. The hyperlink for "Multipose Detection with TensorFlow" should lead to https://www.pipeless.ai/docs/docs/v1/examples/tf-pose instead of https://www.pipeless.ai/docs/docs/v1/examples/pose.

2. The "Edit this page" link should direct to the corresponding Markdown file for editing, allowing users to contribute changes to the documentation.

Screenshots

Desktop:

• OS: Windows 11
• Version: 23H2
• Processor arch: x86

Additional context

• This issue is specific to the "Multipose Detection with TensorFlow" example in the "Examples" category. All other examples in the same category have correct hyperlinks.
• The editing issue is prevalent across all sub-pages, making it impossible to contribute changes to the documentation.

Describe the bug
When i try to open multiple streams i run into too many open files error
To Reproduce
Please describe the steps to reproduce the behavior:

1. Run 30 + streams simultaneously on TensorRT runtime with rtsp as input
2. The GPU usage is around ~35% during period

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• Ubuntu 22
• Tesla T4

Pipeless 1.x does not yet include distributed processing as its predecessor (0.x) which had it enabled by default.

Implementing distributed processing should be simple. It just requires to provide an option (--distributed) that replaces the local bus by a message broker like Kafka, Redis or RabbitMQ. Supporting different brokers would be ideal.

Also, the Pipeless node in distributed mode should behave as a worker node, simply taking events from the broker and processing them. Only of frame change events should be moved to the broker and ideally only when the node receiving the stream has not enough capacity.

This includes the implementation of a distributed KV store. See #71

The stage context is read-only by design, mainly because it depends on the language and can contain non-serializable values. This means it cannot be modified from hooks to share data with other frames. Plus, in a distributed setup, the stage context on each node will be independent of the stage context in other nodes.
We could have a global KV instance in the Pipeless node when running on device and use Redis when using distributed mode.

Sharing data between the processing of frames is sometimes useful, thus providing a KV store, that can also be distributed since will contain serializable data only, makes sense.
Also, it should be namespaced by stream id in order to differentiate data from different streams.

Finally, since frames are processed in parallel, it could be a good idea to provide some util functions with this KV store. For example, it could happen that the hook code tries to access data for the previous frame and it is not yet available. The library could provide wait functionality (which would add delay) or error bypass.

Describe the solution you'd like

Be able to provide an alias for a stream, so it can be identified from hooks.

It should be included as part of the stream API, CLI, and also as part of the frame data.

Some people requested to support webcams as input.

Mac OS and Windows child processes do not inherit the memory state of the parent process when forking. this process a configuration error when using pipeless run all.

The easiest solution is to load the configuration in all the compnents instead of before forking.

Describe the bug

When using multiple streams, after adding the second one it enters in a loop of restart.
After some debugging the hashes always differ when using stages that contain - in their name instead of _.

To Reproduce

1. Create a stage that includes a - in the name.
2. Start pipeless
3. Provide a stream
4. Provide a second stream

While building the container images I noticed some differences between Ubuntu and Debian that lead to a segmentation fault in Debian that didn't happen in Ubuntu. It happens when importing NumPy AND uridecodebin uses https:// (note it works with file://). The pipeline will never start and simply exit with a core dumped when trying to set it to the PLAYING state.

So far, the only solution I have found is to downgrade NumPy from 1.25.2 to 1.24.3.

The main difference between Debian and Ubuntu is that the Gstreamer version available in Debian repositories is 1.18, while in Ubuntu it is 1.20.

Describe the bug
I followed tutorials for watermark and cats and it failed to run at the step pipeless start --stages-dir .

To Reproduce
Please describe the steps to reproduce the behavior:

1. I have installed required libraries (pillow, numpy and opencv-python). They are correctly shown as present when run pip list.
2. Run wget -O - https://github.com/pipeless-ai/pipeless/archive/main.tar.gz | tar -xz --strip=2 "pipeless-main/examples/cats"
3. Running pipeless start --stages-dir . will then give the following output:

[2023-11-18T20:24:00Z INFO  pipeless_ai::stages::parser] ⚙️  Loading stages from .
[2023-11-18T20:24:00Z INFO  pipeless_ai::stages::parser] ⏳ Loading stage 'cats' from ./cats
[2023-11-18T20:24:00Z INFO  pipeless_ai::stages::parser]        Loading hook from ./cats/pre-process.py
thread 'main' panicked at src/stages/languages/python.rs:174:15:
Unable to create Python module from hook: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'cv2'"), traceback: Some(<traceback object at 0x7f885cf6f280>) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace         

Expected behavior
It will work without any failures.

Desktop (please complete the following information):

• OS: WSL, Ubuntu 22.04
• Processor arch: amd64
• Python: 3.11

Thanks for any suggestions.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
This is equivalent to ask yourself why you want this feature.

When you have several cameras connected, v4l2 will select the one with fd 0 by default.
We should allow an option to specify the device to use.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Add a discriminator when providing the input URI like v4l2:/dev/device1

Describe the bug
When providing screen as output URI to directly visualize the output, if you close the video screen Pipeless crashes.

To Reproduce

1. Start Pipeless
2. Provide a stream with --output-uri screen
3. Close the video window before the stream ends.

Expected behavior
It should continue running without crashing and simply ignore the output video

According to Document in https://www.pipeless.ai/docs/docs/v0/getting-started

When I use command pipeless run it showsERROR - OUTPUT - Unable to set the pipeline to the playing state.

Please suggest. How to solve this error. Thank you.

Detail is i the Text below

Operating System: Ubuntu 22.04.3 LTS

Python 3.10.12
pip 23.3.1 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)

pip install pipeless-ai pipeless-ai-cli

pipeless-ai            0.1.19
pipeless-ai-cli        0.1.16

Install Dependencies (GStreamer)

apt-get install -y pkg-config libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
    libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base \
    gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools \
    gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 \
    gstreamer1.0-qt5 gstreamer1.0-pulseaudio libcairo2-dev libgirepository1.0-dev

My config.yaml

input:
  address:
    host: localhost
    port: 1234
  video:
    enable: true
    uri: http://192.168.10.31:800/Gate1-Camera4-2023-11-24_0855_0930_REC.mp4
log_level: INFO
output:
  address:
    host: localhost
    port: 1237
  video:
    enable: true
    uri: file:///home/devteam/my_first_project/output/output.mp4
plugins:
  order: ''
worker:
  n_workers: 1
model:
    uri: ""

Output when pipeless run

running project...
Loading config.yaml...
Config file config.yaml detected
17:15:29 INFO     - INPUT - Reading video from http://192.168.10.31:800/Gate1-Camera4-2023-11-24_0855_0930_REC.mp4                                input.py:290
         INFO     - INPUT - Waiting first worker to be available                                                                                  input.py:305
17:15:30 INFO     - WORKER - Worker ready! Notifying input                                                                                       worker.py:179
17:15:30 INFO     - INPUT - First worker ready                                                                                                    input.py:307
         INFO     - INPUT - Starting input pipeline                                                                                               input.py:216
         INFO     - INPUT - Linking uridecodebin pad to videoconvert pad                                                                          input.py:149
17:15:31 INFO     - INPUT - Tags parsed: taglist, video-codec=(string)"H.264\ \(High\ Profile\)", language-code=(string)en,                       input.py:276
                  bitrate=(uint)23889190;
         INFO     - INPUT - Tags parsed: taglist, datetime=(datetime)2023-11-24T02:33:41Z, encoder=(string)"vlc\ 3.0.20\ stream\ output",         input.py:276
                  container-format=(string)"ISO\ MP4/M4A";
17:15:31 INFO     - OUTPUT - Output tags updated to: taglist, video-codec=(string)"H.264\ \(High\ Profile\)", language-code=(string)en,          output.py:324
                  bitrate=(uint)23889190;
         INFO     - OUTPUT - Output tags updated to: taglist, datetime=(datetime)2023-11-24T02:33:41Z, encoder=(string)"vlc\ 3.0.20\ stream\     output.py:324
                  output", container-format=(string)"ISO\ MP4/M4A", video-codec=(string)"H.264\ \(High\ Profile\)", language-code=(string)en,
                  bitrate=(uint)23889190;
         INFO     - INPUT - dynamic source pad "video_0" with caps: video/x-raw, format=(string)I420, width=(int)3840, height=(int)2160,           input.py:76
                  interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)1:3:5:1,
                  framerate=(fraction)25/1
         INFO     - OUTPUT - Creating new pipeline for caps: video/x-raw, format=(string)I420, width=(int)3840, height=(int)2160,                output.py:370
                  interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)1:3:5:1,
                  framerate=(fraction)25/1
         ERROR    - OUTPUT - Unable to set the pipeline to the playing state.      

When I change log_level to DEBUG

Loading config.yaml...
Config file config.yaml detected
17:16:34 DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read                                                            connection.py:60
         DEBUG    - OUTPUT - No data to read, try again on: InputOutputSocket-Read
..................................

My Nginx Web Server Access Log

192.168.11.236 - - [03/Dec/2023:00:16:37 +0700] "GET /Gate1-Camera4-2023-11-24_0855_0930_REC.mp4 HTTP/1.1" 200 21669061 "-" "GStreamer souphttpsrc 1.20.3 libsoup/2.74.2

The frame path executor is becoming a bottleneck, limiting the parallelization of frames.
This happens because there is single instance of the executor and we are locking it for the entire duration of the path execution.

Add support for writing hooks in Rust.

Unlike for Python hooks that the code interpreted, Rust hooks would require a build when loading stages.

Moving the frames into Python and getting them back is adding a few milliseconds of latency that we can optimize.More specifically, it adds between 1 and 3 ms on each operation.