Giter VIP home page Giter VIP logo

mmt's Introduction

MMT

Media Management Tool

Or, what to do if your desk looks like this:

Backstory:

I've been using an assortment of scripts over the years to manage media from my different action cameras and drones, it's clear a centralized and unified solution is needed.

This tool draws inspiration from my dji-utils/offload.sh script as well as the popular gopro-linux tool and @deviantollam's dohpro

Right now the script supports these cameras:

  • GoPro:
    • HERO2 - HERO5
    • MAX
    • Fusion
    • HERO6 - HERO12
  • Insta360: X2, GO2, X3
  • DJI: Osmo Pocket 1/2, DJI Osmo Action 1/2/3, Mavics, Minis
  • Android: All, but with Pixel 6 (Google Camera) specific fixes

Feel free to PR!

I plan have the tool read a directory, use a config file and act accordingly to offload media from any type of drive

To-do:

Refer to Issues

Features:

  • Import videos and photos from the most popular action cameras (GoPro, Insta360, DJI)
  • Fix nonsensical filenames and file structures:
    • GH011273.MP4 and GH021273.MP4 will become GH1273-01.MP4 and GH1273-02.MP4 respectively
    • VID_20221012_102725_10_586.insv and VID_20221012_102725_00_586.insv will become 102725/VID_20221012_102725_10_586.insv and 102725/VID_20221012_102725_00_586.insv therefore making organizing Insta360 footage easier
  • Group multi shots/related files together, such as GoPro bursts, timelapses and Insta360 timelapse photos
  • Update camera firmware
  • Merge GoPro chaptered videos together
  • Sort files into folders depending on:
    • Camera Name (eg: HERO9 Black, Mavic Air 2)
    • Location (eg: El Escorial, España)
  • Apply LUT profiles to photos

Installing:

Download binary from the releaser Github Action

Running:

Different commands are supported, refer to the wiki

How to configure mmt

How it looks:

mmt's People

Contributors

daktak avatar inode64 avatar konradit avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

mmt's Issues

Test `order` from config file with different orders

Test location.order from config file with different orders

Move location.order to order.

Default:

order:
- date
- location
- camera

Orders:

order:
- location
- date
- camera

Tested:

  • GoPro

    • works
    Tree log
       C:\USERS\KONRA\VIDEOS\PROJECTS\MADRIDTEST
       ├───Madrid, Comunidad de Madrid, España
       │   └───22-12-2022
       │       └───HERO11 Black
       │           ├───photos
       │           └───videos
       │               └───3840x2160 25
       └───NoLocation
           ├───22-12-2022
           │   └───HERO11 Black
           │       ├───photos
           │       └───videos
           │           ├───3840x2160 25
           │           ├───3840x2160 50
           │           ├───4000x3000 25
           │           └───5312x2988 25
           └───23-12-2022
               └───HERO11 Black
                   ├───photos
                   └───videos
                       ├───3840x2160 25
                       ├───4000x3000 25
                       └───5312x4648 25
  • DJI

    • works
    Tree log
       C:\USERS\KONRA\VIDEOS\PROJECTS\ELESCORIALUAV
       ├───El Escorial, España
       │   └───21-12-2022
       │       └───Air 2S
       │           ├───photos
       │           └───videos
       ├───NoLocation
       │   └───22-12-2022
       │       └───Air 2S
       │           └───videos
       └───San Lorenzo de El Escorial, España
           ├───21-12-2022
           │   └───Air 2S
           │       ├───photos
           │       └───videos
           ├───22-12-2022
           │   └───Air 2S
           │       └───videos
           └───24-12-2022
               └───Air 2S
                   └───photos
                       └───raw
order:
- date
- camera
- location

Tested:

  • GoPro

    • works
    Tree log
       C:\USERS\KONRA\VIDEOS\PROJECTS\MADRIDTEST
       ├───22-12-2022
       │   └───HERO11 Black
       │       ├───Madrid, Comunidad de Madrid, España
       │       │   ├───photos
       │       │   └───videos
       │       │       └───3840x2160 25
       │       └───NoLocation
       │           ├───photos
       │           └───videos
       │               ├───3840x2160 25
       │               ├───3840x2160 50
       │               ├───4000x3000 25
       │               └───5312x2988 25
       └───23-12-2022
           └───HERO11 Black
               └───NoLocation
                   ├───photos
                   └───videos
                       ├───3840x2160 25
                       ├───4000x3000 25
                       └───5312x4648 25
    
  • DJI

    • works
order:
- location
- camera
- date

Tested:

  • GoPro

    • works
    Tree log
       C:\USERS\KONRA\VIDEOS\PROJECTS\MADRIDTEST
       ├───Madrid, Comunidad de Madrid, España
       │   └───HERO11 Black
       │       └───22-12-2022
       │           ├───photos
       │           └───videos
       │               └───3840x2160 25
       └───NoLocation
           └───HERO11 Black
               ├───22-12-2022
               │   ├───photos
               │   └───videos
               │       ├───3840x2160 25
               │       ├───3840x2160 50
               │       ├───4000x3000 25
               │       └───5312x2988 25
               └───23-12-2022
                   ├───photos
                   └───videos
                       ├───3840x2160 25
                       ├───4000x3000 25
                       └───5312x4648 25
  • DJI

    • works

Todo:

Last two orderings get the deviceName wrong for DJI:

└───San Lorenzo de El Escorial, España
    ├───Air 2S
    │   ├───21-12-2022
    │   │   ├───photos
    │   │   └───videos
    │   ├───22-12-2022
    │   │   └───videos
    │   └───24-12-2022
    │       └───photos
    │           └───raw
    └───DJI Device
        └───21-12-2022
            ├───photos
            └───videos

Clarification on config usage

from help string, mmt states:

      --config string   config file (default is $HOME/.mmt.yaml)

yet it's not entirely clear what the structure of config should be. Are there any guidelines for this?

Tests

Pretty open ended task, figure out how to test this.

H265 to H264 conversion

Some older PCs, and also NLE video editors don't support the hyper compressed CPU intensive H.265 videos off of GoPro cameras and DJI devices.

Go Pro Session (1st gen) supported?

Is the GoPro Session 1st gen supported by the tool?
Tried to use the tool but my GoPro Session did not show up when I ran the list command. It only showed the 2 partitions I have on my hard drive.
Go Pro session is accessible by Windows manager but also does not have an assigned drive.
Address: This PC\HERO4 Session\External Memory

Merging chapters automatically

Merge chapters from separate files losslessly using ffmpeg in the background as a separate task. Investigate how to keep the GPMF metadata intact too.

Done:

  • Merge function
  • Merge subcommand
  • Support GoPro V1 GOPR[id] and GP[id] merging logic

Sub command: generate CSV/JSON/.EDL DavInci files off HiLight tags from video

  • DaVinci .EDL file
  • CSV
  • JSON

Context:

func tagAsDuration(tag int, increase bool) string {
	seconds := tag / int(time.Microsecond)
	if increase {
		seconds++
	}
	return fmt.Sprintf("01:00:%d:%03d", seconds, 0)
}
func main() {

	tags := []int{18920, 20480}

	content := `TITLE: Timeline 1
FCM: NON-DROP FRAME
`
	for index, tag := range tags {

		content = fmt.Sprintf("%s\n%03d  AX       V     C        %s %s %s %s\n* FROM CLIP NAME: %s\n", content, index, "00:00:00:00", "00:00:00:01", tagAsDuration(tag, false), tagAsDuration(tag, true), "GOPRO.mp4")
	}

	fmt.Println(content)
}

Subcommand: print month calendar view of days with media (mayb heatmap?)

Month calendar view of days with media taken

+----+----+----+----+----+----+----+
| S  | M  | T  | W  | T  | F  | S  |
+----+----+----+----+----+----+----+
| 1  | 2  | 3  | 4  | 5  | 6  | 7  |
| 8  | 9  | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | 31 |
+----+----+----+----+----+----+----+

Colors:

  • yellow: no media

  • red: videos

  • cyan: photos

  • green: photos+videos

  • fast way to get tree of SD card, with nodes being: name,creationdate

  • Parse gpMediaList for GoPro Connect

License

Hi,

I'd like to package this repository in GNU Guix. However, I'm missing the license for this project. Could you please specify? I've checked the code but unfortunately haven't find any.

Unable to get connect method working - HERO 5 Session

This is the result, camera has 43 videos. I can see them in GoProDashboard for example.

gopro mmt import -x connect -o . -c gopro -i 10.5.5.9 -v
GoPro Tool for Linux
To see a list of commands and syntax available run: gopro help
Checking dependencies...
+----------------+---------------+--------+
| FILES IMPORTED | FILES SKIPPED | ERRORS |
+----------------+---------------+--------+
|              0 |             0 |      0 |
+----------------+---------------+--------+
##try without specifying ip address
gopro mmt import -x connect -o . -c gopro -v
GoPro Tool for Linux
To see a list of commands and syntax available run: gopro help
Checking dependencies...
X Something went wrong
  Get "http:///gp/gpTurbo?p=1": http: no Host in request URL

curl 'http://10.5.5.9/gp/gpTurbo?p=1'
Error 404: Not Found

Time on media is incorrect

connect.go
tm := time.Unix(i, 0).UTC()
zone_name, _ := end.Zone()
new_time := strings.Replace(tm.Format(time.UnixDate), "UTC", zone_name, -1)
tm, _ = time.Parse(time.UnixDate, new_time)

If I change to make the time UTC it is correct.
Is this the case with other cameras?
If I don't add this, the date is +11 hours in the future (my timezone)

override via cli: location, date

Possibility of overriding location and date.

Useful for Insta360 videos (clock date gets reset after battery out)

and GoPro media (GPS is spotty)

Blank folder name if location detection fails

Folder name blank if location detection fails

Context:

  • Operating system: macOS Ventura 13.1
  • Camera: Insta360, DJI
  • Version v1.6
    When files are imported manually (not tested it with config file), if the location detection fails, the folder will either created with a blank name or it will be skipped entirely.

Expected Behavior:

Just skip folder with location name.

Current Behavior:

Check out the tree output. For the date "20221010" location was detected correctly (I replaced city name with xxx for privacy reasons). But for the date "20230108" there is no location folder. That would be the expected behaviour if there where a flag to disable location detection.
Folder name problem - for the date "20230112" there is an invisible folder (at least on the command line, in the macOS finder it shows a folder with an empty name) which contains the dji and insta360 videos.

Logs / Screenshots:

├── 20221010
│   └── xxx Deutschland
│       └── Mini 3 Pro
│           └── photos
│               ├── DJI_0006.JPG
│               └── raw
│                   └── DJI_0006.DNG
├── 20230108
│   └── Insta360 Camera
│       └── videos
│           └── 360
│               └── 161519
│                   ├── VID_20230108_161519_00_001.insv
│                   └── VID_20230108_161519_10_001.insv
├── 20230109
│   └── Insta360 Camera
│       └── videos
│           └── 360
│               └── 144742
│                   ├── VID_20230109_144742_00_009.insv
│                   └── VID_20230109_144742_10_009.insv
└── 20230112
    ├──  
    │   └── DJI Device
    │       └── photos
    │           ├── DJI_0012.JPG
    │           └── raw
    │               └── DJI_0012.DNG
    └── Insta360 Camera
        ├── photos
        │   ├── 151433
        │   │   ├── IMG_20230112_151433_00_011.dng
        │   │   └── IMG_20230112_151433_00_011.insp
        └── videos
            └── 360
                └── 151354
                    ├── VID_20230112_151354_00_010.insv
                    └── VID_20230112_151354_10_010.insv

Steps to Reproduce:

GoPro Plus integration

GoPro Plus integration

  • Import media from GoPro Plus
  • Upload entire project / one file / one folder to GoPro Plus

some flags are not working for me on linux with hero 9

Hi,

thank you for your tools, they are really helpful!

I have a problem with the linux version and my GoPro Hero 9: I tried to use the -d (--date) and the -s (--skip_aux) flags, but I get the default behavior. (I tried -d yyyy-mm-dd -s false and -d "yyyy-mm-dd" -s THM.)
I want to import the .LRV files, too.

Maybe I did not use them correctly?

I also tried to use -v but that did not change anything as well.

cli/config flag: Move .SRT from DJI to its own folder

cli/config flag: Move .SRT from DJI to its own folder

--srt_dir true/false (default false) should copy the .SRTs to .../videos/srt/ instead of .../videos/, due to videos stuttering when playing back.

Also maybe have --skip_aux skip the SRTs entirely.

release page empty

the installation readme says "download from the release page" sadly there are no releases (yet?).

Fix wrong Insta360 import

Fix wrong Insta360 import

│   │       └───flat
│   │           ├───104_19
│   │           │       PRO_VID_20230104_191202_10_141.mp4
│   │           │
│   │           ├───191233
│   │           │       VID_20230104_191233_10_142.mp4
│   │           │
│   │           └───pro_mode
│   │               └───191202
│   │                       PRO_VID_20230104_191202_10_141.mp4

flat/104_19/PRO_VID_20230104_191202_10_141.mp4 shouldn't exist as its already imported in pro mode. Make sure the regex looks from the start of the filename.

Add filter support to CLI

Filter by: some suggestions:

  • video duration
  • media type (Videos/Photos)
  • resolution
  • frame rate
  • bit rate
  • is vertical or horizontal

Ideally doesn't require copying the file to memory/disk to pass it thru the filter. All of these can be gathered via FFmpeg/FFprobe or just the filename.

Subcommand: fix erroneous data on imported file

Subcommand: fix erroneous data on imported file

Idea:

mmt fix --location XXXX /path/to/location_folder (just rename the folder)

mmt fix --date XXXX /path/to/date_folder (rename the folder and fix EXIF date/file date)

HiLight parsing and organizing

I've found that the best way to see which clip I will use later on is to put some tags at the end of it (press mode button on GoPro, or shout "Oh Sh*t", or use the app/pebble app).

Then when I run a script that prints the number of hilight tags during the last 30 seconds of each video. That lets me know the clips are important.

This tool should let you label each tag count (eg: --tag-labels="good,great,important") for each hilight count.

HERO6 and newer: GPMF

HERO5 and older: HMMT in UDTA box

Resources:

https://github.com/icegoogles/GoPro-Highlight-Parser

https://gopro.github.io/OpenGoPro/http_2_0#get-hilights

https://github.com/iNViTiON/GoPro-Tagging

Sort by geographic location

Ideal end result:

$ ls /mnt/hdd2/projects/EndOfYearVideo
├───Mexico City, Mexico
│   └───2020-01-02
├───New York, NY, United States
│   └───2020-05-01
├───Madrid, Spain
│   └───2020-10-01

To get location info:

  • GoPro (GPMF)
  • DJI (SRT file)
  • Insta360 (???)

Concurrently import files

Import X videos at the same time in parallel instead of sequentially, use goroutines, mutex lock, synwaitgroups, etc...

Done in #36:

  • Import in parallel via GoPro Connect / HTTP USB / HTTP WiFi
  • Add merge ffmpeg function which detects the GoPro chapters
  • Merge sub command

Todo:

  • Limit number of active jobs via max-active param
  • Add concurrency to GoPro filesystem import
  • Add concurrency to DJI/Insta360 import
  • Construct a tree where each leaf is one part of the path to a file. Once import is all done, iterate over and detect G****-[chapter] and group them into a new leaf with n children, then merge (and keep the files)
  • Make a Go version of udtacopy and put the video 1 chapter's UDTA box into the merged file, so it plays back on gyroflow

Before:

graph TD;
    A[15-03-2021]-->B[HERO9 Black];
    B-->C[videos];
    C-->D[3840x2160 50];
    D-->E[GH1273-01.MP4];
    D-->F[GH1274-01.MP4];
    D-->G[GH1274-02.MP4];
    D-->H[GH1274-03.MP4];

After:

graph TD;
    A[15-03-2021]-->B[HERO9 Black];
    B-->C[videos];
    C-->D[3840x2160 50];
    D-->E[GH1273-01.MP4];
    D-->F[GH1274]
    F-->G[GH1274-01.MP4];
    F-->H[GH1274-02.MP4];
    F-->I[GH1274-03.MP4];
    F-->J[GH1274-merged.MP4];

Sort by other GPMF data

Such as:

  • Scene/Activity detection (SCEN GPMF tag)
  • Number of max faces detected per MP4 file

Handle interruptions

On the event of:

  • camera disconnect:
    • allow for either keep/remove interrupted file
  • SD card unplugged
    • allow for either keep/remove interrupted file
  • CTRL-C
    • cleanup interrupted file

Should also use filesystem seek/HTTP header seek to finish getting the file when called with --resume flag (--resume will look for *.tmp file and resume download)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.