In some cases, the input video can contain invalid PTS timestamps, e.g. that go backwards. This has been only been seen once in a short section of video that appeared to be corrupt.

Nevertheless, for robustness it is better to deal with these cases rather than be unable to process the video.

Please edit github project info - currently showing "No description, website, or topics provided." at the top. Should have a short description and link to http://track4k.co.za

Add functionality to the MainDriver class to accept parameters from the command line.
Parameters include:

• Input filename
• Output filename for cropped video and board video

The y-value for the crop co-ordinates doesn't seem to be set correctly until the first movement.

# track4k presenter.mkv 89877 frames (frame top-left-x top-left-y) output frame size 1920 1080
0 0 0
24 0 600
79 1 600
80 2 600

As track4k is currently pan-only, write out the co-ords with fixed y value.

The function to check aspect ratio is not robust enough and fails when two boards overlap. This is because the function only classifies a board if a certain aspect is met. Might need to check colour and shape as well to make a better classification of boards

Light correction is used to detect boards under varying lighting conditions. Currently the lighting correction only corrects the light for the frame needed. The parameters need to be experimented with more and light correction might only be applied if lighting is bad. So add a way to check light quality first (Assuming this would be more efficient)

In this example, the tracking is distracted by the white beanie on the right of the frame and moves away from the lecturer on the left.

On some cameras, privacy masks can be set to black out the top and/or bottom of an image.

If these have been set, then track4k should not select a crop frame that includes any portion of the blacked-out part, i.e. the crop y co-ordinates should snap to the top and/or bottom of the real image, or if the real image height is less than the crop height, the centre the real image so there's an equal height of black above and below the real image.

Need to remove redundant rectangle detection methods and combine into one function that takes parameters to determine which rectangle detection to use. This is used to detect boards

Output the co-ordinates of blackboards instead of a full video.

out_w:out_h:x:y

to make it easy to run a follow-up ffmpeg process

http://video.stackexchange.com/questions/4563/how-can-i-crop-a-video-with-ffmpeg

Test that the system works on Ubuntu and that no code was accidentally removed in the cleaning process.

If you give a 1280x720 video to track4k with an output resolution of 1280x720, it fails with an assertion error.

There's nothing meaningful to do in this instance (may as well use the input video unchanged), so track4k should check at startup and exit with an error message if input resolution is equal to or smaller than output resolution.

OpenCV: FFMPEG: tag 0x34363258/'X264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'
Input frame resolution: Width=1280  Height=720 of nr#: 7499
Input codec type:
Video Duration (Seconds): 299
FPS: 25
OpenCV Error: Assertion failed (0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows) in Mat, file /usr/local/src/track4k/opencv/modules/core/src/matrix.cpp, line 526
terminate called after throwing an instance of 'cv::Exception'
  what():  /usr/local/src/track4k/opencv/modules/core/src/matrix.cpp:526: error: (-215) 0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows in function Mat

Building cropvid as described in the docs on a clean ubuntu 16.04 fails:

Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
cropvid.c:36:34: fatal error: libavfilter/avfilter.h: Datei oder Verzeichnis nicht gefunden

The repo is about 89M to check out, when it should be tiny:

~ $ git clone https://github.com/LectureTracking/trackhd.git
Cloning into 'trackhd'...
remote: Counting objects: 604, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 604 (delta 9), reused 0 (delta 0), pack-reused 578
Receiving objects: 100% (604/604), 89.01 MiB | 169.00 KiB/s, done.
Resolving deltas: 100% (333/333), done.

There are some techniques for cleaning up git history that you could look into, otherwise you may want to consider deleting and re-creating the repo with a clean commit of source only.

Copy into all files and into the github repository and update the readme once all is completed. Rename folders to single lowercase names and ensure the .ignore file prevents git from adding build/executable files.

Test case video
https://github.com/LectureTracking/trackhd/files/1464667/vid.zip

clean debug and unnecessary code from the project and add comments where necessary.

• Rename Variables

• Remove unnecessary and redundant code

Stage [3 of 3] - Virtual Cinematographer

baqwa-720p.mp4
BoardSegment-baqwa-720pmp4
875967064
OpenCV: FFMPEG: tag 0x34363258/'X264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'
OpenCV: FFMPEG: tag 0x34363258/'X264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'
[flv @ 0x18e4b40] AMF_DATA_TYPE_STRING parsing failed
Input frame resolution: Width=3840 Height=2160 of nr#: 1948
Input codec type:
Video Duration (Seconds): 67
FPS: 29
End of video file
OpenCV Error: Assertion failed (0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows) in Mat, file /srv/dev/opencv/modules/core/src/matrix.cpp, line 522
terminate called after throwing an instance of 'cv::Exception'
what(): /srv/dev/opencv/modules/core/src/matrix.cpp:522: error: (-215) 0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows in function Mat

Currently the board is detected based on shape, aspect ratio, size as well as its intensity. It is assumed that the boards are green and when using a threshold, this area will be black, and for each rectangle assumed to be a board, the colour is evaluated and if it is black, this adds confidence to the classification. However under varying light, such as very bright light, the boards turn out to be white after applying the threshold, so the threshold value needs to be variable based on current lighting conditions. Need to make a method to detect light levels and then based on that select a threshold value.

Remove unused methods, variables and classes. Rename variables that are ambiguous and provide comments explaining methods and classes in more detail.

Currently each frame is read into a C++ vector. This can be changed to instead to initialize a pointer array that would store the frames and this pointer can be passed around instead of having to clone the vector and frames several times.

Track4K always starts panning from the very left of the image.

If there is an identified presenter already, the first frame should be there (rather than pan to the presenter).

if there's no identified presenter in the first frame, then start in the horizontal centre of the image.

• Comment methods:

1. List input variables by type

2. Describe purpose of the method

3. List output variables by type

ffprobe presenter-MAM1000W-20160811.mp4 shows:

Duration: 00:50:00.01, start: 0.033367, bitrate: 5655 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 5652 kb/s, 29.95 fps, 29.97 tbr, 90k tbn, 180k tbc (default)

Output processed to 1920x1080p shows:

Duration: 00:51:38.28, start: 0.000000, bitrate: 1925 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1922 kb/s, 29 fps, 29 tbr, 14848 tbn, 58 tbc (default)

It looks like the framerate has been scaled down so the duration is incorrect, and possibly the frame timestamp information has been dropped (which is important if the source is not a fixed framerate).

Suggestions from @slampunk

On the GitHub README.md:

1. Where applicable, list the minimum required versions for prerequisite packages [e.g. CMake (>= 3.5)]

2. "Firstly change into the OpenCV directory (download destination)..."

I would assume I need to cd into "<your_chosen_working_directory>/opencv", where in fact, I'd simply need to be in "<your_chosen_working_directory>".
Have it read as

"Your chosen working directory now contains two folders, opencv and opencv_contrib. The opencv folder contains the main OpenCV libraries, and the opencv_contrib...."

3. Consider the use of backticks for inline nouns (commands, packages, directories). Makes it clear that the word refers to a terminal/console item.

4. "Inside the main OpenCV folder, change directory into the build folder and remove all files, since it will require rebuilding. "

build folder did not exist for me.
Append with something along the lines of....
"Create a build folder if it does not exist".

5. make -j8 assumes a cpu with at least 8 cores. Maybe add "cat /proc/cpuinfo | grep processor" to get number of cpus.

Write an installation README explaining how to install program's dependencies and run the program itself.
Automate the installation process of the program (excluding dependencies) by writing a shell script.

• Paste the Open Source license into every header and source file

Track4K currently only does panning. It would be useful to be able to set the vertical cropping height manually (as a command-line parameter).

Estimate memory requirements and describe in README.

Hello there,
can you please let me know how to apply track4k on a live video stream on ubuntu server that is hosting an open source video conference software?

thank you.
Jay