mk-fg / fgtk Goto Github PK

View Code? Open in Web Editor NEW

151.0 12.0 33.0 3.16 MB

A set of a misc tools to work with files and processes

Shell 6.29% Python 63.89% HTML 23.44% C 3.66% OCaml 0.83% Nim 1.83% Awk 0.06%

python bash tools scraps collection dev linux sysadmin distro metrics

fgtk's People

Contributors

Stargazers

Watchers

fgtk's Issues

twitch_vod_fetch --ytdl-output-format option can't move output file to another drive on Windows

When the --ytdl-output-format option is used to specify a location on a different drive to the temporary files, the file_dst_path.rename call fails with this error:

Traceback (most recent call last):
  File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 900, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 896, in main
    loop, conf, vod_queue, list_formats_only=opts.ytdl_list_formats ))
  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 466, in run_until_complete
    return future.result()
  File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 273, in vod_fetch
    try: await fetch(url, prefix, info_suffix=info_suffix)
  File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 189, in _wrapper
    return await func(self, ctx, *args, **kws)
  File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 766, in get
    file_dst_path.rename(file_dst_done)
  File "C:\Program Files\Python36\lib\pathlib.py", line 1292, in rename
    self._accessor.rename(self, target)
  File "C:\Program Files\Python36\lib\pathlib.py", line 396, in wrapped
    return strfunc(str(pathobjA), str(pathobjB), *args)
OSError: [WinError 17] The system cannot move the file to a different disk drive: '..\\download_temp\\Dave.mp4' -> 
'C:\\Users\\USER\\Downloads\\LivestreamDownloads\\KirinDave\\20170529 - X of 2 - GAME - Chill with Minecraft & Dave 
For A Bit - v148136907.mp4'

shutil.move can be used to move a file to another location, including across drives.

Connect to PAN from Smartphone

Hi,

If I run the script, and issue a "show" command in bluetoothctl, I can see the PAN uuid.

From the android phone, I have trouble keeping a connection, but when the devices are paired I see the option to use for internet access, but it fails. Any ideas how I can make this work?

I want to run a PAN server on my Ubuntu box. I want to connect to the Ubuntu box via bluetooth from iPhone and Android, and then issue REST commands. Can I use this script to accomplish that?

vfat_shuffler: Cannot re-mount into /media/$user/$autoname

Debian stable (and Ubuntu 18.x and 20.x) automatically mount USB flash drives under /media/$username/$driveLabel.

vfat_shuffler unmounts it, and the system automatically removes the folder. Then the vfat_shuffler wants to re-mount it, and that fails because the folder is gone:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./bin/vfat_shuffler.py", line 667, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "./bin/vfat_shuffler.py", line 665, in main
    if res.returncode: res.check_returncode()
  File "/usr/lib/python3.8/subprocess.py", line 444, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['mount', '/dev/sdb1', '/media/cweiske/music']' returned non-zero exit status 32.

vfat_shuffler: Listing device without path does not work

Using vfat_shuffler to list files of a mounted directory works fine now (#20):

$ sudo ./bin/vfat_shuffler --umount /media/cweiske/CWEISKEOUYA/
[      0] {name}.{ext}
[  43075] 01 - Dont Tell Me That Its Over.ogg
[  43419] 01 - Ich bin hier.ogg

But passing a device file does not work:

$ sudo ./bin/vfat_shuffler --umount /dev/sdb1 
usage: vfat_shuffler [-h] [-l] [-s] [-o] [--rename template] [-r] [-c] [-g type] [--rename-inside]
                     [-x] [-u] [-n] [-v] [-d]
                     dev_or_vfs_path [path]
vfat_shuffler: error: Failed to resolve path to non-root mountpoint: dev/sdb1

When I add a path after the device path, it works:

$ sudo ./bin/vfat_shuffler --umount /dev/sdb1 /
[      0] {name}.{ext}
[  43075] 01 - Dont Tell Me That Its Over.ogg
[  43419] 01 - Ich bin hier.ogg

The help says that path is optional, but it does not seem to be optional when a device is given.

ssh-keyparse base64

When i base_64 decode the seed to store it as binary in the database it often defaults to nothing.

            base64_decode($base64string),
            $base64string,

results in
["","5_DbkBn6IhczM4r-rlIIzrkoLaiNYUchVviGnNqNuf0="]

About 30% of the time the first half stays empty, otherwise it works just as intended.

And when you paste the base64 string into other converters (e.g. https://cryptii.com/base64/binary) it does not interpret it as well.

What am i missing here?

vfat_shuffler: cannot import name 'Embedded' from 'construct'

When running vfat_shuffler.py with the latest construct installed, I get an error:

Traceback (most recent call last):
File "./bin/vfat_shuffler.py", line 3, in
from construct import (
ImportError: cannot import name 'Embedded' from 'construct' (/usr/local/lib/python3.8/dist-packages/construct/init.py)

The reason is that "Embedded" is not part of construct anymore since v2.10.

Workaround:

pip3 install construct==2.9.52

(Info taken from rytilahti/python-yeelightbt#36)

twitch_vod_fetch does not finish appending batched chunks

The script often does not append the last batch of chunks to the destination file. This causes the VoD to be truncated up to 100 seconds early with the default config. Consider the following example.
For ease of demonstration, I changed line 27 in my local copy to file_append_batch = 2. twitch_vod_fetch output truncated for legibility

* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 10 -l 10 -k
  ------ Started 1 downloads, last gid: 000001 ------  
* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 40 -l 40 -k
  ------ Started 4 downloads, last gid: 000004 ------  
* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 5chunks-by2 -l 50 -k
  ------ Started 5 downloads, last gid: 000005 ------

40.mp4 comes out fine (and for some reason, when I tried 20- and 30-second videos, they worked, too), but 5chunks-by2.mp4's last chunk is left in the tmp folder, and poor 10.mp4 never receives its single chunk!

* $ ls -l 5chunks-by2.mp4 40.mp4 10.mp4 
-rw-r--r-- 1 dan dan        0 Jan 14 07:01 10.mp4
-rw-r--r-- 1 dan dan 15039812 Jan 14 07:04 40.mp4
-rw-r--r-- 1 dan dan 15039812 Jan 14 07:10 5chunks-by2.mp4
* $ ls 5chunks-by2.tmp/
5chunks-by2.0000050000000000.mp4.chunk	5chunks-by2.aria2c_conf  5chunks-by2.m3u8  5chunks-by2.m3u8.ua	5chunks-by2.m3u8.url  5chunks-by2.pos
* $ ls 40.tmp/
40.aria2c_conf	40.m3u8  40.m3u8.ua  40.m3u8.url  40.pos
* $ ls 10.tmp/
10.0000010000000000.mp4.chunk  10.aria2c_conf  10.m3u8	10.m3u8.ua  10.m3u8.url  10.pos

I've seen this issue on many (longer) videos since the batched chunk append was implemented; I just chose this short highlight as an example.
Thanks again for your work :)

[Duplicate Issue] twitch_vod_fetch --ytdl-output-format can't move output file to another drive on Windows

Duplicate of #7, please ignore.

base64 encoding discrepancy

Something is not right… I get different base64-encoded result depending on whether I use the default option or I use hex option then transform to base64 (notice the different symbols around 'D', i.e., +D+ vs -D-).

Why is that? Is it because the base64 transform you're using is "urlsafe"? Could we not have a switch to use ordinary base64 instead?

Thanks.

ssh-keyparse.py -s AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy684= AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy685cD9bgfGqnSchnrugs-D-Ths6A0rVCAg7wAUX8R0Mnpw==

ssh-keyparse.py -s AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy684= -x | xxd -r -p | base64 -w 0 AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy685cD9bgfGqnSchnrugs+D+Ths6A0rVCAg7wAUX8R0Mnpw==

Reverting ssh-keyparse

After searching the web for hours, i was not able to find a way to convert ssh-keyphrase back into the 'BEGIN OPENSSH PRIVATE KEY' format, could someone point me in the right direction?

vfat_shuffler: Randomization crashes: get_chain() missing 1 required positional argument: 'clidx'

Randomizing a directory crashes here:

$ sudo ./bin/vfat_shuffler --umount /dev/sdb1 / -s
Traceback (most recent call last):
  File "./bin/vfat_shuffler", line 667, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "./bin/vfat_shuffler", line 659, in main
    op_reorder(path, src, dry_run=opts.dry_run, **sort_opts)
  File "./bin/vfat_shuffler", line 489, in op_reorder
    slice_map = path.get_slice_map()
  File "./bin/vfat_shuffler", line 205, in get_slice_map
    return list(map(self.fs.get_slice, self.get_chain()))
TypeError: get_chain() missing 1 required positional argument: 'clidx'

Full debug log: vfat_shuffler_posarg.log

vfat sort

I am trying to find or write a command line program for Windows that will sort
FAT32 drives

http://softwarerecs.stackexchange.com/q/19575

Is this possible using vfat_shuffler as a starting point? I could use a push in the
right direction.

Interoperability with OpenSSL

This is a request for an enhancement.

I wonder if you could add an option similar to --public, perhaps call it --private, to produce the OpenSSL private key (whether in base64 or hex form) for convenience.

Notice that I'm making a distinction here between the private key for OpenSSH, which OpenSSL would call 'seed', and OpenSSL's own key they call 'private', derived from seed.

The reason is that right now it is convenient to isolate a private key to inject later into other OpenSSH key files, but injecting the said key into a private key file generated by OpenSSL requires deriving the private key from the hash, and it is not convenient to do by hand, or I don't know a ready-made tool to do it.

The closest I know is NewKeyFromSeed() in this Go implementation.

The RFC 8032 has an Appendix A with a detailed Python 3.2 implementation but that only implements sign, not NewKeyFromSeed (even though within sign it might get calculated).

If we could output the private key with --private, the next step could be that -d --private inserts the key into a OpenSSL-generated private key file instead of OpenSSH.

That'd help bridging the gap between OpenSSH and OpenSSL which is not always easy to cross.

twitch_vod_fetch: final file move fails when --ytdl-output-format specifies non-existing directory

In the same vain as #7, the script fails when attempting to move the output file to its final destination when the destination directory does not yet exist. Understandably, you may not expect to be given a non-existing directory before you implemented --ytdl-output-format, but now it may be desirable, seeing as said option can be used to create directories dynamically, or at least youtube-dl can be used in that way. For example, here is what I just tried:

$ twitch_vod_fetch --aria2c-opts "max-concurrent-downloads=2 lowest-speed-limit=10K rpc-listen-all" -o '%(uploader)s/2018-10-10 %(title)s.ts' https://www.twitch.tv/videos/321031441 -s 9:57 -l 19:10
2018-12-08 04:01:14 :: INFO :: --- Downloading VoD vod-001.mp4 (final-name='Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts', url=https://www.twitch.tv/videos/321031441)
2018-12-08 04:01:27 :: INFO :: 

  ------ Started 115 downloads, last gid: 000115 ------  

                                                                                                                                   
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
Traceback (most recent call last):
  File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: 'vod-001.mp4' -> 'Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dan/bin/vendor/twitch_vod_fetch", line 930, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "/home/dan/bin/vendor/twitch_vod_fetch", line 926, in main
    loop, conf, vod_queue, list_formats_only=opts.ytdl_list_formats ))
  File "/home/dan/.local/apps/miniconda3/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "/home/dan/bin/vendor/twitch_vod_fetch", line 271, in vod_fetch
    try: await fetch(url, prefix, info_suffix=info_suffix)
  File "/home/dan/bin/vendor/twitch_vod_fetch", line 189, in _wrapper
    return await func(self, ctx, *args, **kws)
  File "/home/dan/bin/vendor/twitch_vod_fetch", line 783, in get
    shutil.move(file_dst_path, file_dst_done, copy_function=shutil.copy)
  File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 571, in move
    copy_function(src, real_dst)
  File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts'
Command exited with non-zero status 1

After this, I created the directory manually and re-ran the command, and it finished normally.

Granted, this is a minor inconvenience for my use case, as I am only creating a directory for each channel, and I understand if this feature would be out of scope for this script. Regardless, thank you for the script; it does work very nicely. 👍

twitch_vod_fetch not using ytdl/aria2c options

The --ytdl-opts/--aria2c-opts arguments aren't being passed to the youtube-dl/aria2c commands.

I believe the issue is here, the arguments are being stored in local variables rather than the conf object.

To reproduce, run this command:

py twitch_vod_fetch --ytdl-opts="--verbose" --aria2c-opts="log=aria2c.log" https://www.twitch.tv/videos/148136907 "../download_temp/Dave"

When a youtube-dl command is logged, the --verbose option isn't present and youtube-dl doesn't print any debugging output. The same applies to aria2c and the log=aria2c.log option, it's not present in the .aria2c_conf file and no aria2c.log file is created.

ssh-keyparse --patch-key timings

time python ssh-keyparse.py id_25519 HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=

real	0m0.111s
user	0m0.047s
sys	0m0.046s

time python ssh-keyparse.py  --expand-seed HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=

real	0m1.385s
user	0m1.317s
sys	0m0.048s

time python ssh-keyparse.py storage/app/default -d HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=

real	0m1.438s
user	0m1.317s
sys	0m0.080s

time ssh-keygen -t ed25519 -N '' -f ./id_ed25519
Generating public/private ed25519 key pair.
Your identification has been saved in ./id_ed25519.
Your public key has been saved in ./id_ed25519.pub.
The key fingerprint is:
SHA256:PuypdWkGucbhnnIY8eaiphD/CCPeLZNqAZFWVpJhPP0 root@localhost
The key's randomart image is:
+--[ED25519 256]--+
| .o*=.           |
|o.o+..           |
|..  . .          |
|.     .E .       |
|o      oS        |
| +    .=o+ .     |
|= o .  =X =      |
|o=.*o +=oB       |
|.o++=o.==        |
+----[SHA256]-----+

real	0m0.011s
user	0m0.005s
sys	0m0.005s

I have compared some of the timings regarding this script, and i am curious as to why the 'seed > seed + public' takes such a long time, given that ssh-keygen 'does both' in a fraction of the time taken by ssh-keyparse?

twitch_vod_fetch download fails (with with rpc error?)

./twitch-dl --debug http://www.twitch.tv/oculus/v/17538854 oculus3
2015-09-25 16:19:46 :: main INFO :: --- Downloading VoD oculus3 (url: http://www.twitch.tv/oculus/v/17538854)
2015-09-25 16:19:46 :: main DEBUG :: Starting aria2c daemon: aria2c --summary-interval=0 --console-log-level=notice --stop-with-process=2258 --enable-rpc=true --rpc-listen-port=57329 --rpc-secret=iD079kYMUur5sogvFzFB5IIP --no-netrc --no-proxy --always-resume=false --max-concurrent-downloads=5 --max-connection-per-server=5 --max-file-not-found=5 --max-tries=8 --timeout=30 --connect-timeout=20 --lowest-speed-limit=100K --user-agent=Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome) --on-download-complete=/tmp/.fetch_twitch_vod.2258.done.hook
2015-09-25 16:19:46 :: main DEBUG :: Starting downloads (rpc port: 57329)...

09/25 16:19:46 [NOTICE] IPv4 RPC: listening on TCP port 57329

09/25 16:19:46 [NOTICE] IPv6 RPC: listening on TCP port 57329
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_S_create
2015-09-25 16:19:47 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=4, connect=None, read=None, redirect=None)
2015-09-25 16:19:47 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=3, connect=None, read=None, redirect=None)
2015-09-25 16:19:49 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=2, connect=None, read=None, redirect=None)
2015-09-25 16:19:53 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=1, connect=None, read=None, redirect=None)
2015-09-25 16:20:01 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=0, connect=None, read=None, redirect=None)
Traceback (most recent call last):
  File "./twitch-dl", line 445, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "./twitch-dl", line 443, in main
    vod_fetch(url, prefix, dl_info_suffix=info_suffix, **dl_kws)
  File "./twitch-dl", line 257, in vod_fetch
    if len(line_buff) >= line_buff_max: line_buff_flush()
  File "./twitch-dl", line 232, in line_buff_flush
    if gid_files: queue_gid_downloads(gid_files)
  File "./twitch-dl", line 216, in queue_gid_downloads
    for gid_str, path in gid_files ))
  File "./twitch-dl", line 97, in req_jrpc
    with req('post', url, json=data_req) as r: data_res = r.json()
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "./twitch-dl", line 80, in req
    with closing(req.s.request(method, *args, **kws)) as r:
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 424, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=57329): Max retries exceeded with url: /jsonrpc (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))

vfat_shuffler: Listing crashes with KeyError: '.'

vfat_shuffler crashes when listing files at entry ".", which is the current folder:

cweiske:~> sudo ./bin/vfat_shuffler.py /media/cweiske --list
[sudo] Passwort für cweiske: 
Traceback (most recent call last):
  File "./bin/vfat_shuffler.py", line 667, in <module>
    if __name__ == '__main__': sys.exit(main())
  File "./bin/vfat_shuffler.py", line 647, in main
    paths = [op_navigate(fs, path)]
  File "./bin/vfat_shuffler.py", line 428, in op_navigate
    for k in filter(None, slugs): path = path[k]
  File "./bin/vfat_shuffler.py", line 263, in __getitem__
    raise KeyError(name)
KeyError: '.'

mk-fg / fgtk Goto Github PK

fgtk's People

Contributors

Stargazers

Watchers

Forkers

fgtk's Issues

Recommend Projects

Recommend Topics

Recommend Org