As nice as the custom format is, it does not allow for easy readability or parsing by other programs. A standard format would be preferable, and be much more extensible.

As of 5dda5e0, VODs now includes all videos on a channel. This means that "ARCHIVE" (previous full streams), "HIGHLIGHT" (segments from a previous stream), "UPLOAD" (channel owner uploaded content), and "PAST_PREMIERE" (channel owner uploaded content as a live stream) broadcast types are now saved into the vod directory. These should be separated out into their own directories and settings, separate from just "ARCHIVE" types which is what VOD originally referred to in VodBot's vocabulary.

Proposal:

• New directories for each type of video. vods will remain as the "ARCHIVE" type. highlights and uploads match their respective types. premieres goes with "PAST_PREMIERE", which is specifically a type of upload that is showcased as a live stream; not a rerun stream.
• Each new type of video should be treated like a VOD when downloading, but things like print statements and logic should be able to distinguish them.
• QUESTION: should queries to twitch pull all the videos at once and sort them locally, or do individual queries about different videos.
• QUESTION: When querying, should videos stop being queried for if the first previously cached video is found? I believe videos are paged from newest to oldest, but can we be certain about that?
  • ANSWER: after looking at twitch.py, yes returned videos are sorted by time. Yes, videos should stop being queried for when the first locally available video is found.

Not a majorly important change, but pulling chat logs along with stream vods will help with archiving.

In addition, the ability to upload VODs with the proper sections of chat ripped and uploaded to YouTube using their advanced closed captioning system would be perfect. More research will need to be done with this before it can even be ready to implement, and perhaps could be split into its own issue.

Technically, Vodbot's current behavior is fine, but for long term use there should be a change. Vodbot has both Twitch and Google generate new tokens every time. This is not ideal and makes managing tokens nigh impossible. Tokens should be saved to a cache file on the local machine and read every startup, and on every command a check must be performed to make sure the token is still valid and update it if necessary. This will be a bit slow, but it's necessary with these APIs.

A dedicated cache file that is loaded once on every execution with useful info like channel ID's, saved video IDs, etc would make certain things like bash completion and checking for existing vods way faster than needing to do string manipulation. An additional argument (-u, --update-cache) would be useful for if a user put the cache out of sync by manually removing or editing files.

To improve readability, having colored text and some better descriptive prints would be great.

• No "dim" text, Windows doesn't support it (technically). Use dim text sparingly.
• There should be an option to toggle color easily, and functions supporting that.
• "VodBot" name has magenta background and black foreground.
• Errors should be bright (bold) red, with error codes in bright yellow.
• First printed line should be something along the lines of "VodBot, (c) 2021" or some little blurb.
• "Loading config..." "Logging in to Twitch.tv" and "Getting User ID's" should be consolidated to a single line.
• VOD/Clip checks should be reordered internally to check after each individual list is grabbed, and to display the name and count of each, along with a number total.
• VOD/Clip title should be green, and speeds/stats should be cyan.
• The final print should reset the colors just before exit.

A big annoyance in staging videos (for me at least) is typing out many names for big collaborations. An easy solution would be to use numbered indices to reference a specific channel in the config for the pull command. It would require it to be printed out, but it would work. Regular channel names should still work in place of numbers in case a collaborator is not in the config file.

A clear issue would be differentiating between small character-count names and the number index. A patch solution to that would be to limit the indexing to the first 100 channels in the pull list, starting at 0 and going to 99. Beyond that you'll need to type the name, because you're crazy enough to be using VodBot to manage videos for over 100 channels.

The readme is a bit cluttered because of the massive list of commands and info. I'd like to move it to somewhere else, and the GitHub wiki is a good start, but it's also very important to have an offline copy. I can write markdown easily for the wiki, but man pages are new, and I'd like to explore them a bit. This, along with #24 isn't essential for a first release (in my eyes) and can be worked on separately.

Having an entire database keep track of what timezone is where is the closest thing to bloat a non-animal can get. UTC offset strings are pretty recognizable and understood.

Untested, but currently VodBot does not restrict what characters are allowed in the title. This is fine for everything except for certain characters like those reserved by filesystems for specific functions (like directory separators). Underscores would work as a simple solution for characters that shouldn't be allowed.

An easily addable new function. In this case, it'd match the upload command with the --ss and --to arguments, or it would prompt the user for them, and redirect them to ffmpeg for the actual processing. once finished, the file will get moved out of temp storage into the location provided by the user from the last argument.

Currently each video is spliced then uploaded one at a time. A major amount of time is wasted waiting on the video to upload (especially on slower connections), so a queue of video splices would be very good while they're waiting to upload.

Appropriate config settings would include a max number of concurrent stages processed at once.

Pickling is a horrible way of going about storing important information, but because Google did not have good documentation for how to store important information I was lead to believe it was the only option. Turns out I did not research enough, and that it is rather easy to store this information in a JSON format, see Credentials.from_authorized_user_file.

Switching to this method can prevent any sort of issues arising from ever using pickling, that being potential security bugs mainly.

Had a nice discussion in the FTI Discord about this issue, and I think it's time I just get over it. Distributing/handling of Google API OAuth credentials is an absolute disaster and is a shame Google needs to confront. Until that point in time, however, credentials will need to be distributed in a manner that can't be easily gained and abused, so I'm going to lay out exactly how they can be gotten and used (by VodBot).

The credentials should be hosted in base64 by some HTTPS accessible site (preferably one in our control) with a static, canonical URL. VodBot will attempt to use this URL to pull the credentials and decode them from base64 if they do not exist locally. If these credentials are found without pulling first, VodBot will attempt to use them. If that fails, VodBot will the perform the credential pulling process. If those credentials fail, VodBot will error. If they work, then VodBot will save the credentials locally for use later (to prevent unnecessary calls out to the internet). The OAuth procedure will then continue as normal (sign in to Google, save the session, etc).

The assumption was that YouTube would accept HTML entities for special reserved characters such as <, >, &, ", ', etc. This is unfortunately not the case. Some special testing will need to be done to see how to print out these special characters.

Currently VodBot's init function does nothing, which isn't good! I think there should be some prompts when first setting up VodBot. Mainly it should go through all the required config options such as Twitch channels, timezone, and paths (with defaults for all except the first), as well as a printout of location of the config, and a notice that you can edit the stage_format section on your own for ease of use.

This format is not documented anywhere by Google. Thankfully, the VTuber's have figured it out: https://github.com/arcusmaximus/YTSubConverter/blob/master/ytt.ytt

Using existing systems this will be no problem at all to implement. Biggest issue will be creating pens for each user. Should font be configurable, maybe? Config string should be "YouTubeTimedText" or "YTT", probably the latter. Positioning should go the left by default, YouTube has options to override this.

Currently the only indication of upload is a simple print saying that a video upload has begun, nothing on progress or network connection. It would be nice to have some sort of graphic that can be referred to while waiting, since not even YouTube's website details progress.

C:\Users\Apex\Desktop\Projects-Software\vodbot>vodbot export all D:\exportvodbot\
* vodbot 1.0.5 (c) 2020-21 Logan "NotQuiteApex" Hickok-Dickson *
Loading and slicing stages...
Slicing stage part (0/1) `1545657956` (1:16:0 - 1:17:0)
frame= 3720 fps=0.0 q=-1.0 Lsize=   28255kB time=00:01:00.00 bitrate=3857.5kbits/s speed=1.42e+03x
Loading all chat messages for `xhgz`. Exporting as format `YTT`.
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Apex\AppData\Roaming\Python\Python310\Scripts\vodbot.exe\__main__.py", line 7, in <module>
  File "C:\Users\Apex\AppData\Roaming\Python\Python310\site-packages\vodbot\__main__.py", line 17, in deffered_main
    main()
  File "C:\Users\Apex\AppData\Roaming\Python\Python310\site-packages\vodbot\__main__.py", line 137, in main
    export.run(args)
  File "C:\Users\Apex\AppData\Roaming\Python\Python310\site-packages\vodbot\commands\export.py", line 81, in run
    os_replace(tmpfile, args.path / (title+tmpfile.suffix))
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\\Users\\Apex\\.vodbot\\temp\\1545657956=0.mp4' -> 'D:\\exportvodbot\\test.mp4'

A quick check for Windows and using a copy-and-delete-original method instead may work better.

Some old videos from Sky is having issues being downloaded, because of the weird naming and exporting Twitch used to use for highlights 5+ years ago.

A notable culprit is this: https://www.twitch.tv/videos/39737400

The playlist gets written to disk as:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-DISCONTINUITY
#EXTINF:10,
transmux-0000000159-g3R9.ts?start_offset=0&end_offset=2854967
#EXTINF:10,
transmux-0000000160-lv2p.ts?start_offset=0&end_offset=2678811
#EXTINF:10,
transmux-0000000161-lBiF.ts?start_offset=0&end_offset=2999351
#EXTINF:10,
transmux-0000000162-CVKI.ts?start_offset=0&end_offset=2914751
#EXTINF:10,
transmux-0000000163-efwG.ts?start_offset=0&end_offset=2816239
#EXTINF:10,
transmux-0000000164-gQFH.ts?start_offset=0&end_offset=2776007
#EXT-X-ENDLIST

All the files are named appropriately, yet FFMPEG seems to struggle with these. Reading the stderr logs, it appears FFMPEG is ignoring these segments as a security measure.

[hls @ 0x5649438efd40] Filename extension of '/mnt/scratch/vodbot/temp/46779644/transmux-0000000103-TsNL.ts?start_offset=0&end_offset=2881663' is not a common multimedia extension, blocked for security reasons.
If you wish to override this adjust allowed_extensions, you can set it to 'ALL' to allow all

Seems like all that needs to be done is to add -allowed_extensions ALL before the input file in the FFMPEG command.

Currently the staging process only accepts one video and set of timestamps. This isn't ideal, as sometimes videos can get chopped up and maybe you need to cut out specific parts. The new stages should have basically all the info they currently have, except for the slices (video path, timestamps, etc). This should be put into a list of objects, each with a path, set of timestamps, and each will be in the order that they should appear in. This can be initiated by calling vodbot stage new [each video id in order]. VodBot will ask the initial questions of title and description, then ask about slices of each and what timestamps you'd like to use. If you want a part cut out of a specific Vod, you can just enter in the Vod ID twice, and select the timestamps around where the cut out part should be. Once done, VodBot will use the concat demuxer method with ffmpeg by generating a list of all the files, and running the command and outputting to the temp directory. Clean up round one begins and cleans up all the slices, then VodBot exports or uploads the video, then cleans up the list and everything else.

In addition, the stage ID system is not great, its long and difficult and requires cryptography because it was initially based off of the way git calculates hashes. A similar thing can be achieved by simply randomly generating a 4 character string with the alphabet of it being 0-9 and a-z. The string should be random enough and the lifetime of stage files short enough that collisions should be rare but the implementations can just run a while loop of generation until a non-colliding name is found. Otherwise there should probably be an error with your filesystem long before VodBot can generate that many files in one directory.

Chat parsing expects both fields of user and message to exist, which may not exist if chat is just an empty file. There should be a check for if the line is blank or missing the proper separator.

* vodbot 1.0.5 (c) 2020-21 Logan "NotQuiteApex" Hickok-Dickson *
Traceback (most recent call last):
  File "/home/apex/.local/bin/vodbot", line 8, in <module>
    sys.exit(deffered_main())
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 17, in deffered_main
    main()
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 131, in main
    stage.run(args)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/stage.py", line 464, in run
    util.make_dir(stagedir)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/util.py", line 50, in make_dir
    os.makedirs(str(directory), exist_ok=True)
  File "/usr/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.vodbot'

If a directory cannot be made, a clean exit should probably be made. This specifically occurred when trying to check the directory for the stage folder as defined in the config file.

So because I'm an idiot, I haven't been keeping track of this and it seems the original method for downloading clips has been broken by Twitch, and now requires more GraphQL stuff. This is fine, as most of the methods that are necessary to work with the system are in place and reuseable. Just a matter of getting it done.

Title. Turning a Vod to a string attempts to print an incorrect attribute.

Twitch video and clip data has fields for times, which are simply unix timestamps and can easily be compared against.

Issues to consider include:

• Are dates inclusive? Just before and/or after dates?
• Relative dates? How do they work?
• Timezones? If so, how do they get accounted for in the command? How are they accounted for in relative dates?

For future versions, a reformatting of the config file is clearly necessary to cover all intended features. An example is below.

{
	"channels": [
		{
			"username": "notquiteapex",
			"save_chat": true,
			"save_vods": true,
			"save_clips": true,
			"thumbnail_icon": {
				"filename": "heads/apex.png",
				"ox": 92, "oy": 130
			},
		},
		{
			"username": "percy_creates",
			"save_chat": false,
			"save_vods": false,
			"save_clips": true,
			"thumbnail_icon": {
				"filename": "heads/percy.png",
				"ox": 146, "oy": 95
			},
		},
	],
	
	"thumbnail": {
		"enable": false,
		
		"thumb_filename": "thumb_bg.png",
		"thumb_x": 0, "thumb_y": 0,
		"thumb_size": "1280x720",
		
		"screenshot_x": 300, "screenshot_y": 0,
		"screenshot_w,h": "1280,720",
		
		"font": "Ubuntu-Bold-Italic",
		"pointsize": 160,
		
		"font_gravity": "NorthWest",
		"font_x": 420, "font_y": 50,
		
		"head_order": [5,6,7,8,4,3,2,1],
		"head_positions": {
			[250, 250], [465, 280],
			[100, 388], [502, 508], [250, 250, 2],
			[130, 616], [544, 650], [342, 654],
		},
		
		"game_gravity": "center",
		"game_x": -430, "game_y": -240, 
		
		"games": [
			{
				"name": "minecraft", "filename": "games/minecraft.png",
				"ox": 0, "oy": 0, "w,h": "220,220",
			},
		]
	},
	
	"webhook": {
		"enable": false,
		"username": "VodBot",
		"url": "discord_webhook_url",
		"avatar_url": "image_url",
		
		"pull_vod": {
			"enable": false,
			"username": "VodBot Pulled VOD",
			"message": "VOD Pulled! @NotQuiteApex",
			"url": "discord_webhook_url",
			"avatar_url": "image_url"
		},
		"pull_clip": {
			"enable": false,
			"username": "VodBot Pulled Clip",
			"message": "Clip Pulled! @NotQuiteApex",
		},
		"pull_job_done": { "enable": true },
		
		"export_video": { "enable": true },
		"export_job_done": { "enable": true },
		
		"upload_video": { "enable": true },
		"upload_job_done": { "enable": true }
	}
	
	"pull": {
		"save_chat": true,
		
		"gql_client": "kimne78kx3ncx6brgo4mv6wki5h1ko",
		
		"api_use_alt": false,
		"api_client": "",
		"api_secret": "",
	},
	
	"stage": {
		"timezone": "-0400",
		"desc_macros": {
			"captions": "Want to replay chat with video streams? No!",
			"watch": "Watch streams live at {links}",
			"discord": "Don't Join the Friend Team Inc.",
			"vodbot": "Stream archived by VodBot, maybe",
			"e": "on {date}\n\n{captions}\n{watch}\n{discord}\n{vodbot}"
		}
	}
	
	"export": {
		"ffmpeg_loglevel": "warning",
		
		"hardware_accel": {
			"enable": true,
			"hw_args": "",
		},
		
		"chat": {
			"enable": true,
			"msg_time": 10,
			"format": "RealText",
			"random_uncolored_names": true
		}
	},
	
	"upload": {
		"client_path": "/root/.vodbot/yt-client.json",
		"pickle_path": "/root/.vodbot/yt-session.pkl",
		
		"chat": {
			"enable": true,
			"msg_time": 10,
			"random_uncolored_names": true,
			
			"format": {
				"align": "left",
				"pos_x": 0,
				"pos_y": 100,
				"pos_weight": 6,
			}
		}
	},
	
	"directories": {
		"vods": "/root/.vodbot/vods",
		"clips": "/root/.vodbot/clips",
		
		"temp": "/root/.vodbot/temp",
		"stage": "/root/.vodbot/stage",
		
		"thumbnail": "/root/.vodbot/thumbnail"
		
		"games": "/root/.vodbot/thumbnail/games",
		"heads": "/root/.vodbot/thumbnail/heads"
	}
}

• channels - channel info for every user to be tracked by VodBot, has toggles for pull commands which are on by default, and a table for describing how to handle thumbnail generation for the user.
• thumbnail - describes what resources can be used for generating a thumbnail, such as what background image and positioning of game logos.
• webhook - options for sending webhooks to discord on certain action completions. this should be eventually extensible to other services as necessary.
• pull - options for the pull command. save_chat is self explanatory. gql_client controls the client id for using Twitch's GraphQL interface, which should not be changed unless youd like to run the risk of a ban. api section allows for alternative methods of pulling info using the official api in place of GQL calls.
• stage - identical to old section of the same name
• export - handles all export operations for both export and upload command. loglevel is the same, hardware_accel is settings relating to external hardware encoding, and chat is for exporting chat log files.
• upload - handles all operations relating to the upload command. paths are the same as for youtube, chat upload uploads exclusively in the youtube timed text format, with the format section allowing custom formatting and positioning.
• directories - same as previous config, describes what directories exist.

It appears Twitch is now beginning to enforce its new integrity policy on certain GQL data. Before data can be retrieved for most things, the requesting client must provide a "Client-Integrity" header token. This token can be received by making an empty body POST request to "https://gql.twitch.tv/integrity", with the appropriate Client-ID header.

However, it appears that this integrity check alone does not provide enough clearance to actually start making requests. There are other headers that must be populated: "X-Device-Id", "x-kpsdk-cd", and "x-kpsdk-ct". The former appears to just be a unique identifier, where as the latter two headers appear to be related to Kasada, a tool to combat botting. It is not entirely clear how these headers are generated, likely kept hidden on purpose for the purpose of bot-prevention. Copying these headers into Postman ended up flagging the new token as a botting token, but it still allowed my request to go through.

Google has removed out-of-band authentication for OAuth via API. Basically, VodBot can no longer authenticate itself when need be via the terminal by copy pasting important tokens. A solution would be adding a keyword or new command to authenticate in-bound (aka, opening a web browser and authenticating that way) and then pointing out the related session json being generated and moveable to other systems as necessary. Not ideal, but doable. This will need to be fixed before the next release.

Currently, VodBot takes each segment from Twitch as it exists without question. When VOD mutes happen, the muted segments get a -muted infix in the filename. When this happens, VodBot should check that the unmuted segment still exists in Twitch's CDN. If it does, it should download it and update the m3u8 playlist so that FFmpeg concats the segment correctly.

VodBot should also let the user know of any segments that were found unmuted, as well as if any segments were muted.

Currently, VodBot cannot automatically upload VODs or Clips to YouTube channels. This is a planned feature. The arguments that can be supplied also need some cleaning up and overhauling.

• Rework argparse to accept a command, then sub commands/additional arguments.
• VOD/Clip pulling should be able to be executed simultaneously, or individually
• Videos are pulled, syntax should be vodbot pull <vods/clips/both>
• Reorganize classes and API calls to a single file.
• Main should call functions from other files, and have less code overall.
• Add command for staging a VOD or Clip (time stamps start and stop, title, description). example: vodbot stage add <vod/clip-id> with some prompts asking for info with input(), or straight from the command line if provided. Description should be .format()'d with some info like date streamed, duration, streamer link, etc.
• stages are to be made with a Stage class, which will store the info described above. hash() will need to be implemented, it should create a tuple of all the class fields and return hash(tuple).
• Save stages to stage folder in .vodbot as JSON files.
• Stage names are made from the first eight characters of hexidecimal representation of the hash() function called on a Stage object. ie "1ef52531.stage"
• vodbot stage list [id] (or something similar) should read out an enumerated list of the current stages. If an ID is passed then a detailed output should be made.
• vodbot upload <id/all> will upload the named stage to YouTube, or all of them if --all is passed.
• Add dependencies of Google's API for Python
• Load them optionally if the command is dictating an upload
• When uploading, ffmpeg should be invoked to cut out the section of the video to be uploaded, and store it in .vodbot/temp temporarily
• a final confirmation should be invoked before uploading
• printouts of progress on upload should be made (so that progress can be tracked on uploads)

Currently the project relies on streamlink to download VODs and Clips, which is way too hefty of a dependency for my liking. I'd like for the downloader to be self contained, and rely on only ffmpeg to concat the video chunks. There's a few projects including streamlink that document how the video data is pulled.

• Make use of the available data with existing code, using the user ID and video ID
• Find where and how to pull stream chunks from Twitch's CDN
• Fork threads to pull the data concurrently and save it to a temp directory (specified in the config)
• Work on system to rewrite current terminal line
• Use terminal line system to add progress numbers and data rates
• Merge all forks back to main process safely (and start new forks until all data is downloaded)
• Use ffmpeg to concat the data into an mkv in the proper location

Sometimes a description might contain something like angle brackets (<>) which are disallowed by YouTube. The error sent by the API does not say this is the case, but after some testing I discovered that the YouTube API disallows certain characters from its description. These include the previously stated angle brackets, as well as ASCII control characters.

There should be a simple search and error to catch things like this.

Currently VodBot simply shells out commands to ffmpeg to handle all processing. It'd be good to have some configuration flags to help enable external encoders to be used, or to just use the CPU. I currently don't know if ffmpeg does this automatically, or if it can be controlled with the functions VodBot currently uses.

When an entire video is staged as the only part, it's still run through ffmpeg. This isn't ideal, a simple copy paste of the file from the archive to the temp directory would suffice. Best place to add this check might be in video.py:concat_video

This will require a reverse lookup through Twitch's API, but downloading specific VODs or clips with just the VOD ID or Clip Slug would be good

VodBot is skirting a fine line right now and with how Twitch currently handles the specific calls for info on videos and clips, it might just be smarter to swap to using only the Twitch Client ID from the website.

apex@serv ~ $ vodbot upload all
Loading stages... Authenticating with Google... done.
About to upload 1 stage(s).
Slicing stage part (1/1) `1863427077` (0:0:0 - EOF)
Traceback (most recent call last):% (6.4 GB/16.3 GB)...
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/upload.py", line 63, in _upload_artifact
    status, resp = response_upload.next_chunk()
  File "/home/apex/.local/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/apex/.local/lib/python3.10/site-packages/googleapiclient/http.py", line 1084, in next_chunk
    resp, content = http.request(
  File "/home/apex/.local/lib/python3.10/site-packages/google_auth_httplib2.py", line 218, in request
    response, content = self.http.request(
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1725, in request
    (response, content) = self._request(
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1441, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1393, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/apex/.local/bin/vodbot", line 8, in <module>
    sys.exit(deffered_main())
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 82, in deffered_main
    main()
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 184, in main
    import_module(".commands.upload", "vodbot").run(args)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/upload.py", line 357, in run
    video_id = upload_video(conf, service, stage)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/upload.py", line 145, in upload_video
    uploaded = _upload_artifact(f"stage video #r`#fM{stagedata.id}#r`", response_upload, getting_video=True, filesize=filesize)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/upload.py", line 92, in _upload_artifact
    print_error([err])
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/upload.py", line 57, in print_error
    cprint(f"#fY#dWARN: An HTTP error has occurred ({errn}/{errn_max}), retyring in {secs} seconds... ({', '.join(f)})#r")
TypeError: sequence item 0: expected str instance, TimeoutError found

Seems that the main error is the ', '.join(f), where f is a list of errors that need to be converted to a string from TimeoutError. This mishandling of exceptions makes the upload fail when it could be recoverable.

The video in question is from Alchana's stream of Ocarina of Time Master Quest. Video may not be available after writing due to Twitch's VOD lifetime policy. https://www.twitch.tv/videos/1545657956

An early section of the video got claimed by a supposed copyright holder, marked in the timeline in red, and thus was muted to prevent issue by Twitch automatically. Since VodBot pulls stream data from Twitch's public web player, the muted audio persists into the downloaded VOD.

This caused a very odd issue where VodBot uploaded an incomplete VOD with the sections in red being completely absent in the YouTube upload.

This would be the last big feature to add before a "1.0" release. Bash has the ability to have scripts complete command arguments by running scripts in /etc/bash_completion.d/, so the script would have to be included and managed by the make process. Doing something similar for Windows would be nice but not required. I also need to research how zsh does autocompletion compared to bash, or if it's easily backwards compatible.

This will require a new program dependency, ImageMagick, to be in the PATH environment variable, like FFmpeg.

This feature will allow VodBot to generate thumbnails based on inputs from the configuration (see #40) and staging process. For staging, there will be an additional prompt for a timestamp to take a screenshot of to use as the background, another prompt for what game is being played (for the logo), and lastly what text should go beside the game logo (like a part number, #69). The process will also use the info from what streamers are a part in the stream to generate the heads.

Webhooks would be great for letting a user know a task has been completed, like pulls and uploads.

The hardest bit about it would be config and formatting, and possibly needing a new library to send the payload?

Traceback (most recent call last):
  File "/usr/local/bin/vodbot", line 8, in <module>
    sys.exit(deffered_main())
  File "/usr/local/lib/python3.8/dist-packages/vodbot/__main__.py", line 17, in deffered_main
    main()
  File "/usr/local/lib/python3.8/dist-packages/vodbot/__main__.py", line 128, in main
    pull.run(args)
  File "/usr/local/lib/python3.8/dist-packages/vodbot/commands/pull.py", line 16, in run
    download_twitch_video(args)
  File "/usr/local/lib/python3.8/dist-packages/vodbot/commands/pull.py", line 130, in download_twitch_video
    itd_dl.dl_video_chat(vod, chatname)
  File "/usr/local/lib/python3.8/dist-packages/vodbot/itd/download.py", line 103, in dl_video_chat
    msgs = get_video_comments(video_id)
  File "/usr/local/lib/python3.8/dist-packages/vodbot/twitch.py", line 365, in get_video_comments
    usr = c["commenter"]["displayName"]
TypeError: 'NoneType' object is not subscriptable

There should be a check on the commenter being none, and if so display some kind of temporary name.

Twitch does not have a proper api endpoint for this, and whenever a request is made for a stream it always returns the first game played. However, there is a section of the video query called "moments" (VideoMomentConnection) that can be pulled. It's paginated, which shouldn't be a massive deal. It shows all the games switched to, how long each section lasted and when the section began in the stream. It should be noted that this is measured with milliseconds, not seconds.

# VOD chapters query
GET_VIDEO_CHAPTERS = """{{
video(id: {id}) {{
    moments(first=100) {{
        edges {{ cursor node {{
            createdAt description subDescription id type
            positionMilliseconds durationMilliseconds
}}  }}  }}  }}  }}
"""

Pulling this info for metadata purposes as well as displaying it when staging videos would be uber helpful.

This would be relatively simple to implement. A simple info command followed by a login, id, or slug can do the trick, the hardest part is determining what is what, and then using the right query to get the info. Allowing pastes of twitch links might also be good, but then that runs the issue of urls with extra bits at the end to denote timestamps, etc.

When resuming a previously cancelled download, the calculation for time remaining and speed of the download is entirely messed up. I haven't looked into it much, but I assume it has to do with the already downloaded bits of data messing with the calculation, so there should probably be a check to not count already downloaded bits from previous runs of the command.

Where is the upload to YouTube function?

If VodBot attempts to query about a channel that does not exist, the program crashes due to some hasty subscripting (bad assumptions about the return type being a dictionary and not-null/not-None).

Quick fix would be adding checks at each stage of the subscript, though "data" field can probably be assumed to be not-null due to this always being filled on a successful query.

apex@serv ~ $ vodbot pull
Loading config... Loading channel data... Checking directories... Pulling video lists...
notquiteapex: 1 VODs & 1 Clips
percy_creates: 0 VODs & 0 Clips
juicibit: 0 VODs & 0 Clips
alkana: Traceback (most recent call last):
  File "/home/apex/.local/bin/vodbot", line 8, in <module>
    sys.exit(deffered_main())
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 82, in deffered_main
    main()
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/__main__.py", line 180, in main
    import_module(".commands.pull", "vodbot").run(args)
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/commands/pull.py", line 75, in run
    channelvods = [vod for vod in twitch.get_channel_vods(channel) if vod.id not in cache.channels[channel.login].vods]
  File "/home/apex/.local/lib/python3.10/site-packages/vodbot/twitch.py", line 208, in get_channel_vods
    resp = gql.gql_query(query=query).json()["data"]["user"]["videos"]
TypeError: 'NoneType' object is not subscriptable

Found this because Iris now goes by "iris_prismatica" on Twitch, rather than "alkana".

Vodbot has two commands, pull and upload. Pull gathers info on clips and VODs and downloads them to the local machine, where as upload takes videos from the local machine, slices them up, and pushes them to a YouTube channel.

This feels like a bit of a misnomer, and a proper name scheme like upload and download or push and pull would be better, but it'd be preferable to keep existing names as aliases and allow for user choice.