tomhumphries / simple-nvr Goto Github PK

A simple Network Video Recorder written in Node.js and using ffmpeg

JavaScript 80.95% Shell 1.37% EJS 17.68%

nodejs ffmpeg nvr network-video-recorder raspberry-pi ip-camera ipcamera raspberrypi rtsp javascript

simple-nvr's Introduction

Simple Network Video Recorder in Node.js

This is a simple Network Video Recorder (NVR) that is designed to run on cheap hardware, such as a Raspberry Pi with a hard drive. 24/7 video streams from network cameras are saved, and the recorded files are browsable from a basic web interface.

The project is deliberately bare-bones, and configuration is done through .json files.

The camera video streams are saved in 5 minute files (to prevent long periods of video loss should a file become corrupted). At 01:00 UTC, the video files for the previous day are concatinated into a single 24 hour file, and the 5 minute video files are deleted.

ffmpeg is used to connect to the camera streams and save the video feeds.

Set up & configuration

To get started, the following steps must be taken:

Install ffmpeg.
Choose where you want video files to be saved, and update the rootpath directory in the /storage.json configuration file.
Add camera names and RTSP addresses to the /cameras.json configuation file.
Run the nvr.js server. e.g. using PM2 with:

pm2 start nvr.js --name nvr

The nvr.js server will record the videos in 5 minute clips, and combine them at 01:00 UTC every day into a 24 hour video file. Running nvr-browser.js will start a webserver at http://localhost:3000 that will enable you to browser the folder structure and view video files (see example image below)

If you just want to record video without the browser, you can choose to only run nvr.js.

Notes about the code and methods used

Extra details about the implementation and ffmpeg configuration

MP4 vs MKV

mkv files seem to be more resistent to corruption. When unplugging the camera while an mp4 file is being written to, the file is un-openable. When recording to an mkv file and the camera is unplugged, the files can be played and data is available until nearly the point of unplugging. mkv files can be played in the browser in the latest version of Chrome (as of October 2021).

Connecting to camera streams

Using a wireless connection for the cameras appears to work well, and the video feeds very rarely drop connections (usually <60 seconds a day). However using a wireless connection for the Raspberry Pi 3b+ causes many video connection drops, often several minutes a day. For this reason it is recommended to use a wired network connection for the Raspberry Pi / base station.

TCP vs UDP

UDP was tested for the ffmpeg streams, and although it resulted in fewer warning errors from ffmpeg, the video files were often corrupted with the video frames being incorrectly ordered when played back, and some files not opening at all. TCP connections do not seem to suffer from this problem. Many ffmpeg settings variations (e.g. the buffer size) were used to try to mitigate the UPD corruption problem, but none worked reliably.

Detecting stream errors

Several methods of detecting when a video feed fails have been tried. Attempting to detect dropped streams by the error events raised by ffmpeg gave inconsistent results, and occasionally resulted in either:

The feed not restarting
Multiple streams from the same camera

Multiple streams causes further problems, as one or more of the streams creates corrupted files that are difficult to detect programatically.

The best results are achieved with a filewatcher script. The filewatcher looks for constant changes to the raw file that is being streamed to, and when the file is not changed for a set period of time it is assumed that the stream connection has failed. The ffmpeg stream is then killed (if it still exists), and the stream is recreated.

Saving the streams

The streams are saved in 5 minute segments at "regular" 5 minute intervals (i.e. at 00:00:00, 00:05:00, 00:10:00, etc.). The naming configuration offered by ffmpeg allows for some customisation of the filenames, but we change the filenames to a "friendlier" UTC-like format of:

yyyy-mm-ddThh mm ss.mkv

This allows easy identification of the file time as a human, and the filename is also easily parsable back to a UTC time.

Saving location

The file that the stream is currently being written to is located in a raw folder. The ffmpeg configured datetime pattern does not seem to parse correctly according to ISO8601 on Windows, with the lower case z parsing to a descriptor like GMT Summer Time instead of +0100. This causes problems with the default Date() parsing which the code automatically accounts for on Windows machines.

/camera-name/raw/%Y-%m-%dT%H %M %S%z.mkv

A file watcher looks for when multiple files exist in the raw directory, and moves all but the newest file to the camera's day directory (below), renaming it at the same time.

/camera-name/year/month/day/yyyy-mm-ddThh mm ss.mkv

Detecting corrupted video files

Very occasionally a video file becomes corrupted, and causes the concatination script to crash. To avoid this, each video file is scanned before the concatination script runs with ffprobe. Corrupted files are not deleted in case they contain important (but corrupted) footage, and fixing the files may be possible.

Hardware & Cameras

Each camera on a Raspberry Pi 3b+ writing to an external HDD seems to use ~9% CPU.

Two ieGeek cameras bought on Amazon run well when paired with a Raspberry Pi 3+. I suspect the Pi could easily handle more than 2 cameras given the CPU consumption.

simple-nvr's People

Contributors

Stargazers

Watchers

simple-nvr's Issues

about h265 mkv

Whether to support h265 encoded mkv to play with chrome?
thanks

Error when trying to start app

Getting following error - dunno if it's NPM/PM2 or simple-nvr?


0|nvr      | Require stack:
0|nvr      | - /home/benjamin/simple-nvr-master/nvr.js
0|nvr      |     at Module._resolveFilename (node:internal/modules/cjs/loader:1070:15)
0|nvr      |     at Hook._require.Module.require (/usr/local/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:81:25)
0|nvr      |     at require (node:internal/modules/helpers:121:18)
0|nvr      |     at Object.<anonymous> (/home/benjamin/simple-nvr-master/nvr.js:1:17)
0|nvr      |     at Module._compile (node:internal/modules/cjs/loader:1255:14)
0|nvr      |     at Module._extensions..js (node:internal/modules/cjs/loader:1309:10)
0|nvr      |     at Module.load (node:internal/modules/cjs/loader:1113:32)
0|nvr      |     at Module._load (node:internal/modules/cjs/loader:960:12)
0|nvr      |     at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainerFork.js:33:23)
0|nvr      |     at Module._compile (node:internal/modules/cjs/loader:1255:14) {
0|nvr      |   code: 'MODULE_NOT_FOUND',
0|nvr      |   requireStack: [ '/home/benjamin/simple-nvr-master/nvr.js' ]
0|nvr      | }

Web streaming

Hi!
Thanks for this simple NVR.
I faced an issue with streaming video on my iPhone with Safari.
But it works well on Windows with Chrome.

Time zone issue

I see the correct time and time zone (+3) at RAW folder.
But at 2023 folders all files named with 0 UTC time zone.
Is it correct ?

Handling disk full

Hey,
Nice work. Can we implement a handler to delete older recordings if the disk is full? I can work on a PR if you agree.

NVR app crashes when launched

I am encountering an issue while trying to use the simple-nvr application. The moment I launch the application, it crashes with a 'MODULE_NOT_FOUND' error.

I've tried a few suggested fixes but none have worked:
npn link
npm i require-hook

Here are some logs:
rubicon@hufflepuff:$
rubicon@hufflepuff:$ pm2 start /home/rubicon/simple-nvr/nvr.js
[PM2] Applying action restartProcessId on app [nvr](ids: [ 0 ])
[PM2] nvr ✓
[PM2] Process successfully started
┌────┬────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
├────┼────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0 │ nvr │ default │ 1.0.0 │ fork │ 12906 │ 0s │ 15 │ online │ 0% │ 19.3mb │ rubicon │ disabled │
└────┴────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
rubicon@hufflepuff:$
rubicon@hufflepuff:$
rubicon@hufflepuff:$
rubicon@hufflepuff:$ pm2 status
┌────┬────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
├────┼────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0 │ nvr │ default │ 1.0.0 │ fork │ 0 │ 0 │ 30 │ errored │ 0% │ 0b │ rubicon │ disabled │
└────┴────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
rubicon@hufflepuff:$
rubicon@hufflepuff:$
rubicon@hufflepuff:$
rubicon@hufflepuff:$ pm2 logs
[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/home/rubicon/.pm2/pm2.log last 15 lines:
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] online
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] starting in -fork mode-
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] online
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] starting in -fork mode-
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] online
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] starting in -fork mode-
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] online
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] starting in -fork mode-
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] online
PM2 | 2024-05-29T21:14:37: PM2 log: App [nvr:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-05-29T21:14:37: PM2 log: Script /home/rubicon/simple-nvr/nvr.js had too many unstable restarts (16). Stopped. "errored"

/home/rubicon/.pm2/logs/nvr-out.log last 15 lines:
/home/rubicon/.pm2/logs/nvr-error.log last 15 lines:
0|nvr | Require stack:
0|nvr | - /home/rubicon/simple-nvr/nvr.js
0|nvr | at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
0|nvr | at Hook._require.Module.require (/home/rubicon/.nvm/versions/node/v20.14.0/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:81:25)
0|nvr | at require (node:internal/modules/helpers:179:18)
0|nvr | at Object. (/home/rubicon/simple-nvr/nvr.js:1:17)
0|nvr | at Module._compile (node:internal/modules/cjs/loader:1358:14)
0|nvr | at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
0|nvr | at Module.load (node:internal/modules/cjs/loader:1208:32)
0|nvr | at Module._load (node:internal/modules/cjs/loader:1024:12)
0|nvr | at Object. (/home/rubicon/.nvm/versions/node/v20.14.0/lib/node_modules/pm2/lib/ProcessContainerFork.js:33:23)
0|nvr | at Module._compile (node:internal/modules/cjs/loader:1358:14) {
0|nvr | code: 'MODULE_NOT_FOUND',
0|nvr | requireStack: [ '/home/rubicon/simple-nvr/nvr.js' ]
0|nvr | }

rubicon@hufflepuff:$ nvm -v
0.39.1
rubicon@hufflepuff:$
rubicon@hufflepuff:~$ npm -v
10.8.0

rubicon@hufflepuff:~$ node -v
v20.14.0

rubicon@hufflepuff:~$ npm i require-hook
added 2 packages in 1s

about mkv name

hi,it's nice on centos8.
but, in folder,like this:/XXX/test/test/2022/06/06

[root@sf 06]# ll -ah
total 53M
drwxr-xr-x 2 root root 68 Jun 6 13:30 .
drwxr-xr-x 3 root root 16 Jun 6 12:45 ..
-rw-r--r-- 1 root root 23M Jun 6 13:24 '2022-06-06T05 22 15.mkv'
-rw-r--r-- 1 root root 31M Jun 6 13:30 '2022-06-06T05 27 14.mkv'
[root@sf 06]# date
Mon Jun 6 13:33:01 CST 2022

The videos name are use UTC, i want to change others timezone. i tried, it's no effects.

can you help me?
thanks.

Password protection for nvr-browser.js ?

Something like:

NVR_USER=Username NVR_PASS=Secret  node nvr-browser.js

nvr-browser.js

//...
const _USERNAME = process.env.NVR_USER
const _PASSWORD = process.env.NVR_PASS

const auth = (req, res) => {

    if(!_USERNAME) return

    const authorization = req.headers.authorization

    const reject = () => {
       res.setHeader('www-authenticate', 'Basic');
       res.sendStatus(401)
       return false
    }

    if(!authorization) return reject()

    const [username, password] = Buffer.from(authorization.replace('Basic ', ''), 'base64').toString().split(':')

    if(! (username === _USERNAME && password === _PASSWORD)) return reject()

    return true
}

// ...

app.get('*.*', async (req, res, next) => {

    if(!auth(req,res)) return

//...

app.get('*', async (req, res, next) => {

    if(!auth(req,res)) return
//...

Support Forum? Motion Detection?

Not a bug per se, just curious where a good forum is to ask questions? I tried googling but because the name of the software is so generic, it's about impossible to find anything.

I realize this is bare-bones, and it works great! Just curious if there's a configuration for motion-detection-only recording?

Thanks!

not able start the application

getting the below are while running the application .
| Require stack:
0|nvr | - /home/pshewale/simple-nvr/nvr.js
0|nvr | at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
0|nvr | at Module.Hook._require.Module.require (/usr/local/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:81:25)
0|nvr | at require (internal/modules/cjs/helpers.js:74:18)
0|nvr | at Object. (/home/pshewale/simple-nvr/nvr.js:1:17)
0|nvr | at Module._compile (internal/modules/cjs/loader.js:999:30)
0|nvr | at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
0|nvr | at Module.load (internal/modules/cjs/loader.js:863:32)
0|nvr | at Function.Module._load (internal/modules/cjs/loader.js:708:14)
0|nvr | at Object. (/usr/local/lib/node_modules/pm2/lib/ProcessContainerFork.js:33:23)
0|nvr | at Module._compile (internal/modules/cjs/loader.js:999:30) {
0|nvr | code: 'MODULE_NOT_FOUND',
0|nvr | requireStack: [ '/home/pshewale/simple-nvr/nvr.js'

Retry mechanism

Hi. Thank you for sharing your code and experience. I'm looking at https://medium.com/@tom.humph/saving-rtsp-camera-streams-with-ffmpeg-baab7e80d767 and was wondering if you know how to implement and handle retry / relaunch on (stream) error?