Giter VIP home page Giter VIP logo

9001 / copyparty Goto Github PK

View Code? Open in Web Editor NEW
529.0 6.0 29.0 5.09 MB

Portable file server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps

License: MIT License

Python 63.83% Batchfile 0.10% Shell 4.26% CSS 4.00% HTML 1.62% JavaScript 24.86% Dockerfile 0.32% Makefile 0.16% LiveScript 0.14% PowerShell 0.15% Nix 0.55%
nas-frontend file-sharing file-server file-upload-server webdav-server ftp-server copyparty tftp-server

copyparty's Introduction

πŸ’ΎπŸŽ‰ copyparty

turn almost any device into a file server with resumable uploads/downloads using any web browser

πŸ‘‰ Get started! or visit the read-only demo server πŸ‘€ running from a basement in finland

πŸ“· screenshots: browser // upload // unpost // thumbnails // search // fsearch // zip-DL // md-viewer

🎬 videos: upload // cli-upload // race-the-beam

readme toc

quickstart

just run copyparty-sfx.py -- that's it! πŸŽ‰

enable thumbnails (images/audio/video), media indexing, and audio transcoding by installing some recommended deps:

  • Alpine: apk add py3-pillow ffmpeg
  • Debian: apt install --no-install-recommends python3-pil ffmpeg
  • Fedora: rpmfusion + dnf install python3-pillow ffmpeg --allowerasing
  • FreeBSD: pkg install py39-sqlite3 py39-pillow ffmpeg
  • MacOS: port install py-Pillow ffmpeg
  • MacOS (alternative): brew install pillow ffmpeg
  • Windows: python -m pip install --user -U Pillow
    • install python and ffmpeg manually; do not use winget or Microsoft Store (it breaks $PATH)
    • copyparty.exe comes with Pillow and only needs ffmpeg
  • see optional dependencies to enable even more features

running copyparty without arguments (for example doubleclicking it on Windows) will give everyone read/write access to the current folder; you may want accounts and volumes

or see some usage examples for inspiration, or the complete windows example

some recommended options:

  • -e2dsa enables general file indexing
  • -e2ts enables audio metadata indexing (needs either FFprobe or Mutagen)
  • -v /mnt/music:/music:r:rw,foo -a foo:bar shares /mnt/music as /music, readable by anyone, and read-write for user foo, password bar
    • replace :r:rw,foo with :r,foo to only make the folder readable by foo and nobody else
    • see accounts and volumes (or --help-accounts) for the syntax and other permissions

at home

make it accessible over the internet by starting a cloudflare quicktunnel like so:

first download cloudflared and then start the tunnel with cloudflared tunnel --url http://127.0.0.1:3923

as the tunnel starts, it will show a URL which you can share to let anyone browse your stash or upload files to you

since people will be connecting through cloudflare, run copyparty with --xff-hdr cf-connecting-ip to detect client IPs correctly

on servers

you may also want these, especially on servers:

and remember to open the ports you want; here's a complete example including every feature copyparty has to offer:

firewall-cmd --permanent --add-port={80,443,3921,3923,3945,3990}/tcp  # --zone=libvirt
firewall-cmd --permanent --add-port=12000-12099/tcp  # --zone=libvirt
firewall-cmd --permanent --add-port={69,1900,3969,5353}/udp  # --zone=libvirt
firewall-cmd --reload

(69:tftp, 1900:ssdp, 3921:ftp, 3923:http/https, 3945:smb, 3969:tftp, 3990:ftps, 5353:mdns, 12000:passive-ftp)

features

also see comparison to similar software

PS: something missing? post any crazy ideas you've got as a feature request or discussion πŸ€™

testimonials

small collection of user feedback

good enough, surprisingly correct, certified good software, just works, why, wow this is better than nextcloud

motivations

project goals / philosophy

  • inverse linux philosophy -- do all the things, and do an okay job
    • quick drop-in service to get a lot of features in a pinch
    • some of the alternatives might be a better fit for you
  • run anywhere, support everything
    • as many web-browsers and python versions as possible
      • every browser should at least be able to browse, download, upload files
      • be a good emergency solution for transferring stuff between ancient boxes
    • minimal dependencies
      • but optional dependencies adding bonus-features are ok
      • everything being plaintext makes it possible to proofread for malicious code
    • no preparations / setup necessary, just run the sfx (which is also plaintext)
  • adaptable, malleable, hackable
    • no build steps; modify the js/python without needing node.js or anything like that

notes

general notes:

  • paper-printing is affected by dark/light-mode! use lightmode for color, darkmode for grayscale
    • because no browsers currently implement the media-query to do this properly orz

browser-specific:

  • iPhone/iPad: use Firefox to download files
  • Android-Chrome: increase "parallel uploads" for higher speed (android bug)
  • Android-Firefox: takes a while to select files (their fix for ☝️)
  • Desktop-Firefox: may use gigabytes of RAM if your files are massive seems to be OK now
  • Desktop-Firefox: may stop you from unplugging USB flashdrives until you visit about:memory and click Minimize memory usage

server-os-specific:

  • RHEL8 / Rocky8: you can run copyparty using /usr/libexec/platform-python

server notes:

  • pypy is supported but regular cpython is faster if you enable the database

bugs

roughly sorted by chance of encounter

  • general:

    • --th-ff-jpg may fix video thumbnails on some FFmpeg versions (macos, some linux)
    • --th-ff-swr may fix audio thumbnails on some FFmpeg versions
    • if the up2k.db (filesystem index) is on a samba-share or network disk, you'll get unpredictable behavior if the share is disconnected for a bit
      • use --hist or the hist volflag (-v [...]:c,hist=/tmp/foo) to place the db on a local disk instead
    • all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
    • probably more, pls let me know

  • python 3.4 and older (including 2.7):

  • python 2.7 on Windows:

    • cannot index non-ascii filenames with -e2d
    • cannot handle filenames with mojibake

if you have a new exciting bug to share, see reporting bugs

not my bugs

same order here too

  • Chrome issue 1317069 -- if you try to upload a folder which contains symlinks by dragging it into the browser, the symlinked files will not get uploaded

  • Chrome issue 1352210 -- plaintext http may be faster at filehashing than https (but also extremely CPU-intensive)

  • Firefox issue 1790500 -- entire browser can crash after uploading ~4000 small files

  • Android: music playback randomly stops due to battery usage settings

  • iPhones: the volume control doesn't work because apple doesn't want it to

    • AudioContext will probably never be a viable workaround as apple introduces new issues faster than they fix current ones

  • iPhones: the preload feature (in the media-player-options tab) can cause a tiny audio glitch 20sec before the end of each song, but disabling it may cause worse iOS bugs to appear instead

    • just a hunch, but disabling preloading may cause playback to stop entirely, or possibly mess with bluetooth speakers
    • tried to add a tooltip regarding this but looks like apple broke my tooltips

  • Windows: folders cannot be accessed if the name ends with .

    • python or windows bug

  • Windows: msys2-python 3.8.6 occasionally throws RuntimeError: release unlocked lock when leaving a scoped mutex in up2k

    • this is an msys2 bug, the regular windows edition of python is fine

  • VirtualBox: sqlite throws Disk I/O Error when running in a VM and the up2k database is in a vboxsf

    • use --hist or the hist volflag (-v [...]:c,hist=/tmp/foo) to place the db inside the vm instead
    • also happens on mergerfs, so put the db elsewhere

  • Ubuntu: dragging files from certain folders into firefox or chrome is impossible

    • due to snap security policies -- see snap connections firefox for the allowlist, removable-media permits all of /mnt and /media apparently

breaking changes

upgrade notes

  • 1.9.16 (2023-11-04):
    • --stats/prometheus: cpp_bans renamed to cpp_active_bans, and that + cpp_uptime are gauges
  • 1.6.0 (2023-01-29):
    • http-api: delete/move is now POST instead of GET
    • everything other than GET and HEAD must pass cors validation
  • 1.5.0 (2022-12-03): new chunksize formula for files larger than 128 GiB
    • users: upgrade to the latest cli uploader if you use that
    • devs: update third-party up2k clients (if those even exist)

FAQ

"frequently" asked questions

  • is it possible to block read-access to folders unless you know the exact URL for a particular file inside?

    • yes, using the g permission, see the examples there
    • you can also do this with linux filesystem permissions; chmod 111 music will make it possible to access files and folders inside the music folder but not list the immediate contents -- also works with other software, not just copyparty

  • can I link someone to a password-protected volume/file by including the password in the URL?

    • yes, by adding ?pw=hunter2 to the end; replace ? with & if there are parameters in the URL already, meaning it contains a ? near the end

  • how do I stop .hist folders from appearing everywhere on my HDD?

    • by default, a .hist folder is created inside each volume for the filesystem index, thumbnails, audio transcodes, and markdown document history. Use the --hist global-option or the hist volflag to move it somewhere else; see database location

  • can I make copyparty download a file to my server if I give it a URL?

  • firefox refuses to connect over https, saying "Secure Connection Failed" or "SEC_ERROR_BAD_SIGNATURE", but the usual button to "Accept the Risk and Continue" is not shown

    • firefox has corrupted its certstore; fix this by exiting firefox, then find and delete the file named cert9.db somewhere in your firefox profile folder

  • the server keeps saying thank you for playing when I try to access the website

    • you've gotten banned for malicious traffic! if this happens by mistake, and you're running a reverse-proxy and/or something like cloudflare, see real-ip on how to fix this

  • copyparty seems to think I am using http, even though the URL is https

    • your reverse-proxy is not sending the X-Forwarded-Proto: https header; this could be because your reverse-proxy itself is confused. Ensure that none of the intermediates (such as cloudflare) are terminating https before the traffic hits your entrypoint

  • i want to learn python and/or programming and am considering looking at the copyparty source code in that occasion

    •  _|  _      __   _  _|_
(_| (_)     | | (_)  |_

accounts and volumes

per-folder, per-user permissions - if your setup is getting complex, consider making a config file instead of using arguments

  • much easier to manage, and you can modify the config at runtime with systemctl reload copyparty or more conveniently using the [reload cfg] button in the control-panel (if the user has a/admin in any volume)
    • changes to the [global] config section requires a restart to take effect

a quick summary can be seen using --help-accounts

configuring accounts/volumes with arguments:

  • -a usr:pwd adds account usr with password pwd
  • -v .::r adds current-folder . as the webroot, readable by anyone
    • the syntax is -v src:dst:perm:perm:... so local-path, url-path, and one or more permissions to set
    • granting the same permissions to multiple accounts:
      -v .::r,usr1,usr2:rw,usr3,usr4 = usr1/2 read-only, 3/4 read-write

permissions:

  • r (read): browse folder contents, download files, download as zip/tar, see filekeys/dirkeys
  • w (write): upload files, move files into this folder
  • m (move): move files/folders from this folder
  • d (delete): delete files/folders
  • . (dots): user can ask to show dotfiles in directory listings
  • g (get): only download files, cannot see folder contents or zip/tar
  • G (upget): same as g except uploaders get to see their own filekeys (see fk in examples below)
  • h (html): same as g except folders return their index.html, and filekeys are not necessary for index.html
  • a (admin): can see upload time, uploader IPs, config-reload
  • A ("all"): same as rwmda. (read/write/move/delete/admin/dotfiles)

examples:

  • add accounts named u1, u2, u3 with passwords p1, p2, p3: -a u1:p1 -a u2:p2 -a u3:p3
  • make folder /srv the root of the filesystem, read-only by anyone: -v /srv::r
  • make folder /mnt/music available at /music, read-only for u1 and u2, read-write for u3: -v /mnt/music:music:r,u1,u2:rw,u3
    • unauthorized users accessing the webroot can see that the music folder exists, but cannot open it
  • make folder /mnt/incoming available at /inc, write-only for u1, read-move for u2: -v /mnt/incoming:inc:w,u1:rm,u2
    • unauthorized users accessing the webroot can see that the inc folder exists, but cannot open it
    • u1 can open the inc folder, but cannot see the contents, only upload new files to it
    • u2 can browse it and move files from /inc into any folder where u2 has write-access
  • make folder /mnt/ss available at /i, read-write for u1, get-only for everyone else, and enable filekeys: -v /mnt/ss:i:rw,u1:g:c,fk=4
    • c,fk=4 sets the fk (filekey) volflag to 4, meaning each file gets a 4-character accesskey
    • u1 can upload files, browse the folder, and see the generated filekeys
    • other users cannot browse the folder, but can access the files if they have the full file URL with the filekey
    • replacing the g permission with wg would let anonymous users upload files, but not see the required filekey to access it
    • replacing the g permission with wG would let anonymous users upload files, receiving a working direct link in return

anyone trying to bruteforce a password gets banned according to --ban-pw; default is 24h ban for 9 failed attempts in 1 hour

shadowing

hiding specific subfolders by mounting another volume on top of them

for example -v /mnt::r -v /var/empty:web/certs:r mounts the server folder /mnt as the webroot, but another volume is mounted at /web/certs -- so visitors can only see the contents of /mnt and /mnt/web (at URLs / and /web), but not /mnt/web/certs because URL /web/certs is mapped to /var/empty

dotfiles

unix-style hidden files/folders by starting the name with a dot

anyone can access these if they know the name, but they normally don't appear in directory listings

a client can request to see dotfiles in directory listings if global option -ed is specified, or the volume has volflag dots, or the user has permission .

dotfiles do not appear in search results unless one of the above is true, and the global option / volflag dotsrch is set

the browser

accessing a copyparty server using a web-browser

copyparty-browser-fs8

tabs

the main tabs in the ui

  • [πŸ”Ž] search by size, date, path/name, mp3-tags ...
  • [🧯] unpost: undo/delete accidental uploads
  • [πŸš€] and [🎈] are the uploaders
  • [πŸ“‚] mkdir: create directories
  • [πŸ“] new-md: create a new markdown document
  • [πŸ“Ÿ] send-msg: either to server-log or into textfiles if --urlform save
  • [🎺] audio-player config options
  • [βš™οΈ] general client config options

hotkeys

the browser has the following hotkeys (always qwerty)

  • ? show hotkeys help
  • B toggle breadcrumbs / navpane
  • I/K prev/next folder
  • M parent folder (or unexpand current)
  • V toggle folders / textfiles in the navpane
  • G toggle list / grid view -- same as η”° bottom-right
  • T toggle thumbnails / icons
  • ESC close various things
  • ctrl-K delete selected files/folders
  • ctrl-X cut selected files/folders
  • ctrl-V paste
  • Y download selected files
  • F2 rename selected file/folder
  • when a file/folder is selected (in not-grid-view):
    • Up/Down move cursor
    • shift+Up/Down select and move cursor
    • ctrl+Up/Down move cursor and scroll viewport
    • Space toggle file selection
    • Ctrl-A toggle select all
  • when a textfile is open:
    • I/K prev/next textfile
    • S toggle selection of open file
    • M close textfile
  • when playing audio:
    • J/L prev/next song
    • U/O skip 10sec back/forward
    • 0..9 jump to 0%..90%
    • P play/pause (also starts playing the folder)
    • Y download file
  • when viewing images / playing videos:
    • J/L, Left/Right prev/next file
    • Home/End first/last file
    • F toggle fullscreen
    • S toggle selection
    • R rotate clockwise (shift=ccw)
    • Y download file
    • Esc close viewer
    • videos:
      • U/O skip 10sec back/forward
      • 0..9 jump to 0%..90%
      • P/K/Space play/pause
      • M mute
      • C continue playing next video
      • V loop entire file
      • [ loop range (start)
      • ] loop range (end)
  • when the navpane is open:
    • A/D adjust tree width
  • in the grid view:
    • S toggle multiselect
    • shift+A/D zoom
  • in the markdown editor:
    • ^s save
    • ^h header
    • ^k autoformat table
    • ^u jump to next unicode character
    • ^e toggle editor / preview
    • ^up, ^down jump paragraphs

navpane

switching between breadcrumbs or navpane

click the 🌲 or pressing the B hotkey to toggle between breadcrumbs path (default), or a navpane (tree-browser sidebar thing)

  • [+] and [-] (or hotkeys A/D) adjust the size
  • [🎯] jumps to the currently open folder
  • [πŸ“ƒ] toggles between showing folders and textfiles
  • [πŸ“Œ] shows the name of all parent folders in a docked panel
  • [a] toggles automatic widening as you go deeper
  • [↡] toggles wordwrap
  • [πŸ‘€] show full name on hover (if wordwrap is off)

thumbnails

press g or η”° to toggle grid-view instead of the file listing and t toggles icons / thumbnails

  • can be made default globally with --grid or per-volume with volflag grid

copyparty-thumbs-fs8

it does static images with Pillow / pyvips / FFmpeg, and uses FFmpeg for video files, so you may want to --no-thumb or maybe just --no-vthumb depending on how dangerous your users are

  • pyvips is 3x faster than Pillow, Pillow is 3x faster than FFmpeg
  • disable thumbnails for specific volumes with volflag dthumb for all, or dvthumb / dathumb / dithumb for video/audio/images only

audio files are covnerted into spectrograms using FFmpeg unless you --no-athumb (and some FFmpeg builds may need --th-ff-swr)

images with the following names (see --th-covers) become the thumbnail of the folder they're in: folder.png, folder.jpg, cover.png, cover.jpg

  • the order is significant, so if both cover.png and folder.jpg exist in a folder, it will pick the first matching --th-covers entry (folder.jpg)
  • and, if you enable file indexing, it will also try those names as dotfiles (.folder.jpg and so), and then fallback on the first picture in the folder (if it has any pictures at all)

enabling multiselect lets you click files to select them, and then shift-click another file for range-select

  • multiselect is mostly intended for phones/tablets, but the sel option in the [βš™οΈ] settings tab is better suited for desktop use, allowing selection by CTRL-clicking and range-selection with SHIFT-click, all without affecting regular clicking
    • the sel option can be made default globally with --gsel or per-volume with volflag gsel

zip downloads

download folders (or file selections) as zip or tar files

select which type of archive you want in the [βš™οΈ] config tab:

name url-suffix description
tar ?tar plain gnutar, works great with curl | tar -xv
pax ?tar=pax pax-format tar, futureproof, not as fast
tgz ?tar=gz gzip compressed gnu-tar (slow), for curl | tar -xvz
txz ?tar=xz gnu-tar with xz / lzma compression (v.slow)
zip ?zip=utf8 works everywhere, glitchy filenames on win7 and older
zip_dos ?zip traditional cp437 (no unicode) to fix glitchy filenames
zip_crc ?zip=crc cp437 with crc32 computed early for truly ancient software
  • gzip default level is 3 (0=fast, 9=best), change with ?tar=gz:9
  • xz default level is 1 (0=fast, 9=best), change with ?tar=xz:9
  • bz2 default level is 2 (1=fast, 9=best), change with ?tar=bz2:9
  • hidden files (dotfiles) are excluded unless account is allowed to list them
    • up2k.db and dir.txt is always excluded
  • bsdtar supports streaming unzipping: curl foo?zip=utf8 | bsdtar -xv
    • good, because copyparty's zip is faster than tar on small files
  • zip_crc will take longer to download since the server has to read each file twice
    • this is only to support MS-DOS PKZIP v2.04g (october 1993) and older
      • how are you accessing copyparty actually

you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right

copyparty-zipsel-fs8

cool trick: download a folder by appending url-params ?tar&opus or ?tar&mp3 to transcode all audio files (except aac|m4a|mp3|ogg|opus|wma) to opus/mp3 before they're added to the archive

  • super useful if you're 5 minutes away from takeoff and realize you don't have any music on your phone but your server only has flac files and downloading those will burn through all your data + there wouldn't be enough time anyways
  • and url-params &j / &w produce jpeg/webm thumbnails/spectrograms instead of the original audio/video/images (&p for audio waveforms)
    • can also be used to pregenerate thumbnails; combine with --th-maxage=9999999 or --th-clean=0

uploading

drag files/folders into the web-browser to upload

dragdrop is the recommended way, but you may also:

when uploading files through dragdrop or CTRL-V, this initiates an upload using up2k; there are two browser-based uploaders available:

  • [🎈] bup, the basic uploader, supports almost every browser since netscape 4.0
  • [πŸš€] up2k, the good / fancy one

NB: you can undo/delete your own uploads with [🧯] unpost (and this is also where you abort unfinished uploads, but you have to refresh the page first)

up2k has several advantages:

  • you can drop folders into the browser (files are added recursively)
  • files are processed in chunks, and each chunk is checksummed
    • uploads autoresume if they are interrupted by network issues
    • uploads resume if you reboot your browser or pc, just upload the same files again
    • server detects any corruption; the client reuploads affected chunks
    • the client doesn't upload anything that already exists on the server
    • no filesize limit unless imposed by a proxy, for example Cloudflare, which blocks uploads over 383.9 GiB
  • much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
  • the last-modified timestamp of the file is preserved

it is perfectly safe to restart / upgrade copyparty while someone is uploading to it!
all known up2k clients will resume just fine πŸ’ͺ

see up2k for details on how it works, or watch a demo video

copyparty-upload-fs8

protip: you can avoid scaring away users with contrib/plugins/minimal-up2k.js which makes it look much simpler

protip: if you enable favicon in the [βš™οΈ] settings tab (by typing something into the textbox), the icon in the browser tab will indicate upload progress -- also, the [πŸ””] and/or [πŸ”Š] switches enable visible and/or audible notifications on upload completion

the up2k UI is the epitome of polished inutitive experiences:

  • "parallel uploads" specifies how many chunks to upload at the same time
  • [πŸƒ] analysis of other files should continue while one is uploading
  • [πŸ₯”] shows a simpler UI for faster uploads from slow devices
  • [🎲] generate random filenames during upload
  • [πŸ“…] preserve last-modified timestamps; server times will match yours
  • [πŸ”Ž] switch between upload and file-search mode
    • ignore [πŸ”Ž] if you add files by dragging them into the browser

and then theres the tabs below it,

  • [ok] is the files which completed successfully
  • [ng] is the ones that failed / got rejected (already exists, ...)
  • [done] shows a combined list of [ok] and [ng], chronological order
  • [busy] files which are currently hashing, pending-upload, or uploading
    • plus up to 3 entries each from [done] and [que] for context
  • [que] is all the files that are still queued

note that since up2k has to read each file twice, [🎈] bup can theoretically be up to 2x faster in some extreme cases (files bigger than your ram, combined with an internet connection faster than the read-speed of your HDD, or if you're uploading from a cuo2duo)

if you are resuming a massive upload and want to skip hashing the files which already finished, you can enable turbo in the [βš™οΈ] config tab, but please read the tooltip on that button

file-search

dropping files into the browser also lets you see if they exist on the server

copyparty-fsearch-fs8

when you drag/drop files into the browser, you will see two dropzones: Upload and Search

on a phone? toggle the [πŸ”Ž] switch green before tapping the big yellow Search button to select your files

the files will be hashed on the client-side, and each hash is sent to the server, which checks if that file exists somewhere

files go into [ok] if they exist (and you get a link to where it is), otherwise they land in [ng]

  • the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much

unpost

undo/delete accidental uploads

copyparty-unpost-fs8

you can unpost even if you don't have regular move/delete access, however only for files uploaded within the past --unpost seconds (default 12 hours) and the server must be running with -e2d

self-destruct

uploads can be given a lifetime, afer which they expire / self-destruct

the feature must be enabled per-volume with the lifetime upload rule which sets the upper limit for how long a file gets to stay on the server

clients can specify a shorter expiration time using the up2k ui -- the relevant options become visible upon navigating into a folder with lifetimes enabled -- or by using the life upload modifier

specifying a custom expiration time client-side will affect the timespan in which unposts are permitted, so keep an eye on the estimates in the up2k ui

race the beam

download files while they're still uploading (demo video) -- it's almost like peer-to-peer

requires the file to be uploaded using up2k (which is the default drag-and-drop uploader), alternatively the command-line program

file manager

cut/paste, rename, and delete files/folders (if you have permission)

file selection: click somewhere on the line (not the link itsef), then:

  • space to toggle

  • up/down to move

  • shift-up/down to move-and-select

  • ctrl-shift-up/down to also scroll

  • shift-click another line for range-select

  • cut: select some files and ctrl-x

  • paste: ctrl-v in another folder

  • rename: F2

you can move files across browser tabs (cut in one tab, paste in another)

batch rename

select some files and press F2 to bring up the rename UI

batch-rename-fs8

quick explanation of the buttons,

  • [βœ… apply rename] confirms and begins renaming
  • [❌ cancel] aborts and closes the rename window
  • [β†Ί reset] reverts any filename changes back to the original name
  • [decode] does a URL-decode on the filename, fixing stuff like & and %20
  • [advanced] toggles advanced mode

advanced mode: rename files based on rules to decide the new names, based on the original name (regex), or based on the tags collected from the file (artist/title/...), or a mix of both

in advanced mode,

  • [case] toggles case-sensitive regex
  • regex is the regex pattern to apply to the original filename; any files which don't match will be skipped
  • format is the new filename, taking values from regex capturing groups and/or from file tags
    • very loosely based on foobar2000 syntax
  • presets lets you save rename rules for later

available functions:

  • $lpad(text, length, pad_char)
  • $rpad(text, length, pad_char)

so,

say you have a file named meganeko - Eclipse - 07 Sirius A.mp3 (absolutely fantastic album btw) and the tags are: Album:Eclipse, Artist:meganeko, Title:Sirius A, tn:7

you could use just regex to rename it:

  • regex = (.*) - (.*) - ([0-9]{2}) (.*)
  • format = (3). (1) - (4)
  • output = 07. meganeko - Sirius A.mp3

or you could use just tags:

  • format = $lpad((tn),2,0). (artist) - (title).(ext)
  • output = 7. meganeko - Sirius A.mp3

or a mix of both:

  • regex = - ([0-9]{2})
  • format = (1). (artist) - (title).(ext)
  • output = 07. meganeko - Sirius A.mp3

the metadata keys you can use in the format field are the ones in the file-browser table header (whatever is collected with -mte and -mtp)

media player

plays almost every audio format there is (if the server has FFmpeg installed for on-demand transcoding)

the following audio formats are usually always playable, even without FFmpeg: aac|flac|m4a|mp3|ogg|opus|wav

some hilights:

  • OS integration; control playback from your phone's lockscreen (windows // iOS // android)
  • shows the audio waveform in the seekbar
  • not perfectly gapless but can get really close (see settings + eq below); good enough to enjoy gapless albums as intended
  • videos can be played as audio, without wasting bandwidth on the video

click the play link next to an audio file, or copy the link target to share it (optionally with a timestamp to start playing from, like that example does)

open the [🎺] media-player-settings tab to configure it,

  • "switches":
    • [πŸ”€] shuffles the files inside each folder
    • [preload] starts loading the next track when it's about to end, reduces the silence between songs
    • [full] does a full preload by downloading the entire next file; good for unreliable connections, bad for slow connections
    • [~s] toggles the seekbar waveform display
    • [/np] enables buttons to copy the now-playing info as an irc message
    • [os-ctl] makes it possible to control audio playback from the lockscreen of your device (enables mediasession)
    • [seek] allows seeking with lockscreen controls (buggy on some devices)
    • [art] shows album art on the lockscreen
    • [🎯] keeps the playing song scrolled into view (good when using the player as a taskbar dock)
    • [⟎] shrinks the playback controls
  • "buttons":
    • [uncache] may fix songs that won't play correctly due to bad files in browser cache
  • "at end of folder":
    • [loop] keeps looping the folder
    • [next] plays into the next folder
  • "transcode":
    • [flac] converts flac and wav files into opus (if supported by browser) or mp3
    • [aac] converts aac and m4a files into opus (if supported by browser) or mp3
    • [oth] converts all other known formats into opus (if supported by browser) or mp3
      • aac|ac3|aif|aiff|alac|alaw|amr|ape|au|dfpwm|dts|flac|gsm|it|m4a|mo3|mod|mp2|mp3|mpc|mptm|mt2|mulaw|ogg|okt|opus|ra|s3m|tak|tta|ulaw|wav|wma|wv|xm|xpk
  • "tint" reduces the contrast of the playback bar

audio equalizer

and dynamic range compressor

can also boost the volume in general, or increase/decrease stereo width (like crossfeed just worse)

has the convenient side-effect of reducing the pause between songs, so gapless albums play better with the eq enabled (just make it flat)

not available on iPhones / iPads because AudioContext currently breaks background audio playback on iOS (15.7.8)

fix unreliable playback on android

due to phone / app settings, android phones may randomly stop playing music when the power saver kicks in, especially at the end of an album -- you can fix it by disabling power saving in the app settings of the browser you use for music streaming (preferably a dedicated one)

markdown viewer

and there are two editors

copyparty-md-read-fs8

there is a built-in extension for inline clickable thumbnails;

  • enable it by adding <!-- th --> somewhere in the doc
  • add thumbnails with !th[l](your.jpg) where l means left-align (r = right-align)
  • a single line with --- clears the float / inlining
  • in the case of README.md being displayed below a file listing, thumbnails will open in the gallery viewer

other notes,

  • the document preview has a max-width which is the same as an A4 paper when printed

markdown vars

dynamic docs with serverside variable expansion to replace stuff like {{self.ip}} with the client's IP, or {{srv.htime}} with the current time on the server

see ./srv/expand/ for usage and examples

other tricks

  • you can link a particular timestamp in an audio file by adding it to the URL, such as &20 / &20s / &1m20 / &t=1:20 after the .../#af-c8960dab

  • enabling the audio equalizer can help make gapless albums fully gapless in some browsers (chrome), so consider leaving it on with all the values at zero

  • get a plaintext file listing by adding ?ls=t to a URL, or a compact colored one with ?ls=v (for unix terminals)

  • if you are using media hotkeys to switch songs and are getting tired of seeing the OSD popup which Windows doesn't let you disable, consider ./contrib/media-osd-bgone.ps1

  • click the bottom-left Ο€ to open a javascript prompt for debugging

  • files named .prologue.html / .epilogue.html will be rendered before/after directory listings unless --no-logues

  • files named descript.ion / DESCRIPT.ION are parsed and displayed in the file listing, or as the epilogue if nonstandard

  • files named README.md / readme.md will be rendered after directory listings unless --no-readme (but .epilogue.html takes precedence)

  • README.md and *logue.html can contain placeholder values which are replaced server-side before embedding into directory listings; see --help-exp

searching

search by size, date, path/name, mp3-tags, ...

copyparty-search-fs8

when started with -e2dsa copyparty will scan/index all your files. This avoids duplicates on upload, and also makes the volumes searchable through the web-ui:

  • make search queries by size/date/directory-path/filename, or...
  • drag/drop a local file to see if the same contents exist somewhere on the server, see file-search

path/name queries are space-separated, AND'ed together, and words are negated with a - prefix, so for example:

  • path: shibayan -bossa finds all files where one of the folders contain shibayan but filters out any results where bossa exists somewhere in the path
  • name: demetori styx gives you good stuff

the raw field allows for more complex stuff such as ( tags like *nhato* or tags like *taishi* ) and ( not tags like *nhato* or not tags like *taishi* ) which finds all songs by either nhato or taishi, excluding collabs (terrible example, why would you do that)

for the above example to work, add the commandline argument -e2ts to also scan/index tags from music files, which brings us over to:

server config

using arguments or config files, or a mix of both:

  • config files (-c some.conf) can set additional commandline arguments; see ./docs/example.conf and ./docs/example2.conf
  • kill -s USR1 (same as systemctl reload copyparty) to reload accounts and volumes from config files without restarting
    • or click the [reload cfg] button in the control-panel if the user has a/admin in any volume
    • changes to the [global] config section requires a restart to take effect

NB: as humongous as this readme is, there is also a lot of undocumented features. Run copyparty with --help to see all available global options; all of those can be used in the [global] section of config files, and everything listed in --help-flags can be used in volumes as volflags.

zeroconf

announce enabled services on the LAN (pic) -- -z enables both mdns and ssdp

  • --z-on / --z-off' limits the feature to certain networks

mdns

LAN domain-name and feature announcer

uses multicast dns to give copyparty a domain which any machine on the LAN can use to access it

all enabled services (webdav, ftp, smb) will appear in mDNS-aware file managers (KDE, gnome, macOS, ...)

the domain will be http://partybox.local if the machine's hostname is partybox unless --name specifies soemthing else

ssdp

windows-explorer announcer

uses ssdp to make copyparty appear in the windows file explorer on all machines on the LAN

doubleclicking the icon opens the "connect" page which explains how to mount copyparty as a local filesystem

if copyparty does not appear in windows explorer, use --zsv to see why:

  • maybe the discovery multicast was sent from an IP which does not intersect with the server subnets

qr-code

print a qr-code (screenshot) for quick access, great between phones on android hotspots which keep changing the subnet

  • --qr enables it
  • --qrs does https instead of http
  • --qrl lootbox/?pw=hunter2 appends to the url, linking to the lootbox folder with password hunter2
  • --qrz 1 forces 1x zoom instead of autoscaling to fit the terminal size
    • 1x may render incorrectly on some terminals/fonts, but 2x should always work

it uses the server hostname if mdns is enbled, otherwise it'll use your external ip (default route) unless --qri specifies a specific ip-prefix or domain

ftp server

an FTP server can be started using --ftp 3921, and/or --ftps for explicit TLS (ftpes)

  • based on pyftpdlib
  • needs a dedicated port (cannot share with the HTTP/HTTPS API)
  • uploads are not resumable -- delete and restart if necessary
  • runs in active mode by default, you probably want --ftp-pr 12000-13000
    • if you enable both ftp and ftps, the port-range will be divided in half
    • some older software (filezilla on debian-stable) cannot passive-mode with TLS
  • login with any username + your password, or put your password in the username field

some recommended FTP / FTPS clients; wark = example password:

webdav server

with read-write support, supports winXP and later, macos, nautilus/gvfs ... a greay way to access copyparty straight from the file explorer in your OS

click the connect button in the control-panel to see connection instructions for windows, linux, macos

general usage:

  • login with any username + your password, or put your password in the username field (password field can be empty/whatever)

on macos, connect from finder:

in order to grant full write-access to webdav clients, the volflag daw must be set and the account must also have delete-access (otherwise the client won't be allowed to replace the contents of existing files, which is how webdav works)

connecting to webdav from windows

using the GUI (winXP or later):

  • rightclick [my computer] -> [map network drive] -> Folder: http://192.168.123.1:3923/
    • on winXP only, click the Sign up for online storage hyperlink instead and put the URL there
    • providing your password as the username is recommended; the password field can be anything or empty

known client bugs:

  • win7+ doesn't actually send the password to the server when reauthenticating after a reboot unless you first try to login with an incorrect password and then switch to the correct password
    • or just type your password into the username field instead to get around it entirely
  • connecting to a folder which allows anonymous read will make writing impossible, as windows has decided it doesn't need to login
    • workaround: connect twice; first to a folder which requires auth, then to the folder you actually want, and leave both of those mounted
  • win7+ may open a new tcp connection for every file and sometimes forgets to close them, eventually needing a reboot
    • maybe NIC-related (??), happens with win10-ltsc on e1000e but not virtio
  • windows cannot access folders which contain filenames with invalid unicode or forbidden characters (<>:"/\|?*), or names ending with .
  • winxp cannot show unicode characters outside of some range
    • latin-1 is fine, hiragana is not (not even as shift-jis on japanese xp)

tftp server

a TFTP server (read/write) can be started using --tftp 3969 (you probably want ftp instead unless you are actually communicating with hardware from the 90s (in which case we should definitely hang some time))

that makes this the first RTX DECT Base that has been updated using copyparty πŸŽ‰

  • based on partftpy
  • no accounts; read from world-readable folders, write to world-writable, overwrite in world-deletable
  • needs a dedicated port (cannot share with the HTTP/HTTPS API)
    • run as root (or see below) to use the spec-recommended port 69 (nice)
  • can reply from a predefined portrange (good for firewalls)
  • only supports the binary/octet/image transfer mode (no netascii)
  • RFC 7440 is not supported, so will be extremely slow over WAN
    • assuming default blksize (512), expect 1100 KiB/s over 100BASE-T, 400-500 KiB/s over wifi, 200 on bad wifi

most clients expect to find TFTP on port 69, but on linux and macos you need to be root to listen on that. Alternatively, listen on 3969 and use NAT on the server to forward 69 to that port;

  • on linux: iptables -t nat -A PREROUTING -i eth0 -p udp --dport 69 -j REDIRECT --to-port 3969

some recommended TFTP clients:

  • curl (cross-platform, read/write)
    • get: curl --tftp-blksize 1428 tftp://127.0.0.1:3969/firmware.bin
    • put: curl --tftp-blksize 1428 -T firmware.bin tftp://127.0.0.1:3969/
  • windows: tftp.exe (you probably already have it)
    • tftp -i 127.0.0.1 put firmware.bin
  • linux: tftp-hpa, atftp
    • atftp --option "blksize 1428" 127.0.0.1 3969 -p -l firmware.bin -r firmware.bin
    • tftp -v -m binary 127.0.0.1 3969 -c put firmware.bin

smb server

unsafe, slow, not recommended for wan, enable with --smb for read-only or --smbw for read-write

click the connect button in the control-panel to see connection instructions for windows, linux, macos

dependencies: python3 -m pip install --user -U impacket==0.11.0

  • newer versions of impacket will hopefully work just fine but there is monkeypatching so maybe not

some BIG WARNINGS specific to SMB/CIFS, in decreasing importance:

  • not entirely confident that read-only is read-only
  • the smb backend is not fully integrated with vfs, meaning there could be security issues (path traversal). Please use --smb-port (see below) and prisonparty
    • account passwords work per-volume as expected, and so does account permissions (read/write/move/delete), but --smbw must be given to allow write-access from smb
    • shadowing probably works as expected but no guarantees

and some minor issues,

  • clients only see the first ~400 files in big folders; impacket#1433
  • hot-reload of server config (/?reload=cfg) does not include the [global] section (commandline args)
  • listens on the first IPv4 -i interface only (default = :: = 0.0.0.0 = all)
  • login doesn't work on winxp, but anonymous access is ok -- remove all accounts from copyparty config for that to work
    • win10 onwards does not allow connecting anonymously / without accounts
  • on windows, creating a new file through rightclick --> new --> textfile throws an error due to impacket limitations -- hit OK and F5 to get your file
  • python3 only
  • slow (the builtin webdav support in windows is 5x faster, and rclone-webdav is 30x faster)

known client bugs:

  • on win7 only, --smb1 is much faster than smb2 (default) because it keeps rescanning folders on smb2
    • however smb1 is buggy and is not enabled by default on win10 onwards
  • windows cannot access folders which contain filenames with invalid unicode or forbidden characters (<>:"/\|?*), or names ending with .

the smb protocol listens on TCP port 445, which is a privileged port on linux and macos, which would require running copyparty as root. However, this can be avoided by listening on another port using --smb-port 3945 and then using NAT on the server to forward the traffic from 445 to there;

  • on linux: iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 445 -j REDIRECT --to-port 3945

authenticate with one of the following:

  • username $username, password $password
  • username $password, password k

browser ux

tweaking the ui

  • set default sort order globally with --sort or per-volume with the sort volflag; specify one or more comma-separated columns to sort by, and prefix the column name with - for reverse sort
    • the column names you can use are visible as tooltips when hovering over the column headers in the directory listing, for example href ext sz ts tags/.up_at tags/Cirle tags/.tn tags/Artist tags/Title
    • to sort in music order (album, track, artist, title) with filename as fallback, you could --sort tags/Cirle,tags/.tn,tags/Artist,tags/Title,href
    • to sort by upload date, first enable showing the upload date in the listing with -e2d -mte +.up_at and then --sort tags/.up_at

see ./docs/rice for more, including how to add stuff (css/<meta>/...) to the html <head> tag, or to add your own translation

opengraph

discord and social-media embeds

can be enabled globally with --og or per-volume with volflag og

note that this disables hotlinking because the opengraph spec demands it; to sneak past this intentional limitation, you can enable opengraph selectively by user-agent, for example --og-ua '(Discord|Twitter|Slack)bot' (or volflag og_ua)

you can also hotlink files regardless by appending ?raw to the url

NOTE: because discord (and maybe others) strip query args such as ?raw in opengraph tags, any links which require a filekey or dirkey will not work

if you want to entirely replace the copyparty response with your own jinja2 template, give the template filepath to --og-tpl or volflag og_tpl (all members of HttpCli are available through the this object)

file indexing

enables dedup and music search ++

file indexing relies on two database tables, the up2k filetree (-e2d) and the metadata tags (-e2t), stored in .hist/up2k.db. Configuration can be done through arguments, volflags, or a mix of both.

through arguments:

  • -e2d enables file indexing on upload
  • -e2ds also scans writable folders for new files on startup
  • -e2dsa also scans all mounted volumes (including readonly ones)
  • -e2t enables metadata indexing on upload
  • -e2ts also scans for tags in all files that don't have tags yet
  • -e2tsr also deletes all existing tags, doing a full reindex
  • -e2v verfies file integrity at startup, comparing hashes from the db
  • -e2vu patches the database with the new hashes from the filesystem
  • -e2vp panics and kills copyparty instead
  • --xlink enables deduplication across volumes

the same arguments can be set as volflags, in addition to d2d, d2ds, d2t, d2ts, d2v for disabling:

  • -v ~/music::r:c,e2ds,e2tsr does a full reindex of everything on startup
  • -v ~/music::r:c,d2d disables all indexing, even if any -e2* are on
  • -v ~/music::r:c,d2t disables all -e2t* (tags), does not affect -e2d*
  • -v ~/music::r:c,d2ds disables on-boot scans; only index new uploads
  • -v ~/music::r:c,d2ts same except only affecting tags

note:

  • upload-times can be displayed in the file listing by enabling the .up_at metadata key, either globally with -e2d -mte +.up_at or per-volume with volflags e2d,mte=+.up_at (will have a ~17% performance impact on directory listings)
  • e2tsr is probably always overkill, since e2ds/e2dsa would pick up any file modifications and e2ts would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise
  • the rescan button in the admin panel has no effect unless the volume has -e2ds or higher
  • deduplication is possible on windows if you run copyparty as administrator (not saying you should!)

exclude-patterns

to save some time, you can provide a regex pattern for filepaths to only index by filename/path/size/last-modified (and not the hash of the file contents) by setting --no-hash \.iso$ or the volflag :c,nohash=\.iso$, this has the following consequences:

  • initial indexing is way faster, especially when the volume is on a network disk
  • makes it impossible to file-search
  • if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected

similarly, you can fully ignore files/folders using --no-idx [...] and :c,noidx=\.iso$

  • when running on macos, all the usual apple metadata files are excluded by default

if you set --no-hash [...] globally, you can enable hashing for specific volumes using flag :c,nohash=

filesystem guards

avoid traversing into other filesystems using --xdev / volflag :c,xdev, skipping any symlinks or bind-mounts to another HDD for example

and/or you can --xvol / :c,xvol to ignore all symlinks leaving the volume's top directory, but still allow bind-mounts pointing elsewhere

  • symlinks are permitted with xvol if they point into another volume where the user has the same level of access

these options will reduce performance; unlikely worst-case estimates are 14% reduction for directory listings, 35% for download-as-tar

as of copyparty v1.7.0 these options also prevent file access at runtime -- in previous versions it was just hints for the indexer

periodic rescan

filesystem monitoring; if copyparty is not the only software doing stuff on your filesystem, you may want to enable periodic rescans to keep the index up to date

argument --re-maxage 60 will rescan all volumes every 60 sec, same as volflag :c,scan=60 to specify it per-volume

uploads are disabled while a rescan is happening, so rescans will be delayed by --db-act (default 10 sec) when there is write-activity going on (uploads, renames, ...)

upload rules

set upload rules using volflags, some examples:

  • :c,sz=1k-3m sets allowed filesize between 1 KiB and 3 MiB inclusive (suffixes: b, k, m, g)
  • :c,df=4g block uploads if there would be less than 4 GiB free disk space afterwards
  • :c,vmaxb=1g block uploads if total volume size would exceed 1 GiB afterwards
  • :c,vmaxn=4k block uploads if volume would contain more than 4096 files afterwards
  • :c,nosub disallow uploading into subdirectories; goes well with rotn and rotf:
  • :c,rotn=1000,2 moves uploads into subfolders, up to 1000 files in each folder before making a new one, two levels deep (must be at least 1)
  • :c,rotf=%Y/%m/%d/%H enforces files to be uploaded into a structure of subfolders according to that date format
    • if someone uploads to /foo/bar the path would be rewritten to /foo/bar/2021/08/06/23 for example
    • but the actual value is not verified, just the structure, so the uploader can choose any values which conform to the format string
      • just to avoid additional complexity in up2k which is enough of a mess already
  • :c,lifetime=300 delete uploaded files when they become 5 minutes old

you can also set transaction limits which apply per-IP and per-volume, but these assume -j 1 (default) otherwise the limits will be off, for example -j 4 would allow anywhere between 1x and 4x the limits you set depending on which processing node the client gets routed to

  • :c,maxn=250,3600 allows 250 files over 1 hour from each IP (tracked per-volume)
  • :c,maxb=1g,300 allows 1 GiB total over 5 minutes from each IP (tracked per-volume)

notes:

  • vmaxb and vmaxn requires either the e2ds volflag or -e2dsa global-option

compress uploads

files can be autocompressed on upload, either on user-request (if config allows) or forced by server-config

  • volflag gz allows gz compression
  • volflag xz allows lzma compression
  • volflag pk forces compression on all files
  • url parameter pk requests compression with server-default algorithm
  • url parameter gz or xz requests compression with a specific algorithm
  • url parameter xz requests xz compression

things to note,

  • the gz and xz arguments take a single optional argument, the compression level (range 0 to 9)
  • the pk volflag takes the optional argument ALGORITHM,LEVEL which will then be forced for all uploads, for example gz,9 or xz,0
  • default compression is gzip level 9
  • all upload methods except up2k are supported
  • the files will be indexed after compression, so dupe-detection and file-search will not work as expected

some examples,

  • -v inc:inc:w:c,pk=xz,0
    folder named inc, shared at inc, write-only for everyone, forces xz compression at level 0
  • -v inc:inc:w:c,pk
    same write-only inc, but forces gz compression (default) instead of xz
  • -v inc:inc:w:c,gz
    allows (but does not force) gz compression if client uploads to /inc?pk or /inc?gz or /inc?gz=4

other flags

  • :c,magic enables filetype detection for nameless uploads, same as --magic

database location

in-volume (.hist/up2k.db, default) or somewhere else

copyparty creates a subfolder named .hist inside each volume where it stores the database, thumbnails, and some other stuff

this can instead be kept in a single place using the --hist argument, or the hist= volflag, or a mix of both:

  • --hist ~/.cache/copyparty -v ~/music::r:c,hist=- sets ~/.cache/copyparty as the default place to put volume info, but ~/music gets the regular .hist subfolder (- restores default behavior)

note:

  • markdown edits are always stored in a local .hist subdirectory
  • on windows the volflag path is cyglike, so /c/temp means C:\temp but use regular paths for --hist
    • you can use cygpaths for volumes too, -v C:\Users::r and -v /c/users::r both work

metadata from audio files

set -e2t to index tags on upload

-mte decides which tags to index and display in the browser (and also the display order), this can be changed per-volume:

  • -v ~/music::r:c,mte=title,artist indexes and displays title followed by artist

if you add/remove a tag from mte you will need to run with -e2tsr once to rebuild the database, otherwise only new files will be affected

but instead of using -mte, -mth is a better way to hide tags in the browser: these tags will not be displayed by default, but they still get indexed and become searchable, and users can choose to unhide them in the [βš™οΈ] config pane

-mtm can be used to add or redefine a metadata mapping, say you have media files with foo and bar tags and you want them to display as qux in the browser (preferring foo if both are present), then do -mtm qux=foo,bar and now you can -mte artist,title,qux

tags that start with a . such as .bpm and .dur(ation) indicate numeric value

see the beautiful mess of a dictionary in mtag.py for the default mappings (should cover mp3,opus,flac,m4a,wav,aif,)

--no-mutagen disables Mutagen and uses FFprobe instead, which...

  • is about 20x slower than Mutagen
  • catches a few tags that Mutagen doesn't
    • melodic key, video resolution, framerate, pixfmt
  • avoids pulling any GPL code into copyparty
  • more importantly runs FFprobe on incoming files which is bad if your FFmpeg has a cve

--mtag-to sets the tag-scan timeout; very high default (60 sec) to cater for zfs and other randomly-freezing filesystems. Lower values like 10 are usually safe, allowing for faster processing of tricky files

file parser plugins

provide custom parsers to index additional tags, also see ./bin/mtag/README.md

copyparty can invoke external programs to collect additional metadata for files using mtp (either as argument or volflag), there is a default timeout of 60sec, and only files which contain audio get analyzed by default (see ay/an/ad below)

  • -mtp .bpm=~/bin/audio-bpm.py will execute ~/bin/audio-bpm.py with the audio file as argument 1 to provide the .bpm tag, if that does not exist in the audio metadata
  • -mtp key=f,t5,~/bin/audio-key.py uses ~/bin/audio-key.py to get the key tag, replacing any existing metadata tag (f,), aborting if it takes longer than 5sec (t5,)
  • -v ~/music::r:c,mtp=.bpm=~/bin/audio-bpm.py:c,mtp=key=f,t5,~/bin/audio-key.py both as a per-volume config wow this is getting ugly

but wait, there's more! -mtp can be used for non-audio files as well using the a flag: ay only do audio files (default), an only do non-audio files, or ad do all files (d as in dontcare)

  • "audio file" also means videos btw, as long as there is an audio stream
  • -mtp ext=an,~/bin/file-ext.py runs ~/bin/file-ext.py to get the ext tag only if file is not audio (an)
  • -mtp arch,built,ver,orig=an,eexe,edll,~/bin/exe.py runs ~/bin/exe.py to get properties about windows-binaries only if file is not audio (an) and file extension is exe or dll
  • if you want to daisychain parsers, use the p flag to set processing order
    • -mtp foo=p1,~/a.py runs before -mtp foo=p2,~/b.py and will forward all the tags detected so far as json to the stdin of b.py
  • option c0 disables capturing of stdout/stderr, so copyparty will not receive any tags from the process at all -- instead the invoked program is free to print whatever to the console, just using copyparty as a launcher
    • c1 captures stdout only, c2 only stderr, and c3 (default) captures both
  • you can control how the parser is killed if it times out with option kt killing the entire process tree (default), km just the main process, or kn let it continue running until copyparty is terminated

if something doesn't work, try --mtag-v for verbose error messages

event hooks

trigger a program on uploads, renames etc (examples)

you can set hooks before and/or after an event happens, and currently you can hook uploads, moves/renames, and deletes

there's a bunch of flags and stuff, see --help-hooks

if you want to write your own hooks, see devnotes

upload events

the older, more powerful approach (examples):

-v /mnt/inc:inc:w:c,mte=+x1:c,mtp=x1=ad,kn,/usr/bin/notify-send

so filesystem location /mnt/inc shared at /inc, write-only for everyone, appending x1 to the list of tags to index (mte), and using /usr/bin/notify-send to "provide" tag x1 for any filetype (ad) with kill-on-timeout disabled (kn)

that'll run the command notify-send with the path to the uploaded file as the first and only argument (so on linux it'll show a notification on-screen)

note that this is way more complicated than the new event hooks but this approach has the following advantages:

  • non-blocking and multithreaded; doesn't hold other uploads back
  • you get access to tags from FFmpeg and other mtp parsers
  • only trigger on new unique files, not dupes

note that it will occupy the parsing threads, so fork anything expensive (or set kn to have copyparty fork it for you) -- otoh if you want to intentionally queue/singlethread you can combine it with --mtag-mt 1

handlers

redefine behavior with plugins (examples)

replace 404 and 403 errors with something completely different (that's it for now)

identity providers

replace copyparty passwords with oauth and such

you can disable the built-in password-based login sysem, and instead replace it with a separate piece of software (an identity provider) which will then handle authenticating / authorizing of users; this makes it possible to login with passkeys / fido2 / webauthn / yubikey / ldap / active directory / oauth / many other single-sign-on contraptions

a popular choice is Authelia (config-file based), another one is authentik (GUI-based, more complex)

there is a docker-compose example which is hopefully a good starting point (alternatively see ./docs/idp.md if you're the DIY type)

a more complete example of the copyparty configuration options look like this

using the cloud as storage

connecting to an aws s3 bucket and similar

there is no built-in support for this, but you can use FUSE-software such as rclone / geesefs / JuiceFS to first mount your cloud storage as a local disk, and then let copyparty use (a folder in) that disk as a volume

you may experience poor upload performance this way, but that can sometimes be fixed by specifying the volflag sparse to force the use of sparse files; this has improved the upload speeds from 1.5 MiB/s to over 80 MiB/s in one case, but note that you are also more likely to discover funny bugs in your FUSE software this way, so buckle up

someone has also tested geesefs in combination with gocryptfs with surprisingly good results, getting 60 MiB/s upload speeds on a gbit line, but JuiceFS won with 80 MiB/s using its built-in encryption

you may improve performance by specifying larger values for --iobuf / --s-rd-sz / --s-wr-sz

hiding from google

tell search engines you dont wanna be indexed, either using the good old robots.txt or through copyparty settings:

  • --no-robots adds HTTP (X-Robots-Tag) and HTML (<meta>) headers with noindex, nofollow globally
  • volflag [...]:c,norobots does the same thing for that single volume
  • volflag [...]:c,robots ALLOWS search-engine crawling for that volume, even if --no-robots is set globally

also, --force-js disables the plain HTML folder listing, making things harder to parse for search engines

themes

you can change the default theme with --theme 2, and add your own themes by modifying browser.css or providing your own css to --css-browser, then telling copyparty they exist by increasing --themes

0. classic dark 2. flat pm-monokai 4. vice
1. classic light 3. flat light 5. hotdog stand

the classname of the HTML tag is set according to the selected theme, which is used to set colors as css variables ++

  • each theme generally has a dark theme (even numbers) and a light theme (odd numbers), showing in pairs
  • the first theme (theme 0 and 1) is html.a, second theme (2 and 3) is html.b
  • if a light theme is selected, html.y is set, otherwise html.z is
  • so if the dark edition of the 2nd theme is selected, you use any of html.b, html.z, html.bz to specify rules

see the top of ./copyparty/web/browser.css where the color variables are set, and there's layout-specific stuff near the bottom

if you want to change the fonts, see ./docs/rice/

complete examples

  • see running on windows for a fancy windows setup

    • or use any of the examples below, just replace python copyparty-sfx.py with copyparty.exe if you're using the exe edition

  • allow anyone to download or upload files into the current folder:
    python copyparty-sfx.py

    • enable searching and music indexing with -e2dsa -e2ts

    • start an FTP server on port 3921 with --ftp 3921

    • announce it on your LAN with -z so it appears in windows/Linux file managers

  • anyone can upload, but nobody can see any files (even the uploader):
    python copyparty-sfx.py -e2dsa -v .::w

    • block uploads if there's less than 4 GiB free disk space with --df 4

    • show a popup on new uploads with --xau bin/hooks/notify.py

  • anyone can upload, and receive "secret" links for each upload they do:
    python copyparty-sfx.py -e2dsa -v .::wG:c,fk=8

  • anyone can browse (r), only kevin (password okgo) can upload/move/delete (A) files:
    python copyparty-sfx.py -e2dsa -a kevin:okgo -v .::r:A,kevin

  • read-only music server:
    python copyparty-sfx.py -v /mnt/nas/music:/music:r -e2dsa -e2ts --no-robots --force-js --theme 2

    • ...with bpm and key scanning
      -mtp .bpm=f,audio-bpm.py -mtp key=f,audio-key.py

    • ...with a read-write folder for kevin whose password is okgo
      -a kevin:okgo -v /mnt/nas/inc:/inc:rw,kevin

    • ...with logging to disk
      -lo log/cpp-%Y-%m%d-%H%M%S.txt.xz

reverse-proxy

running copyparty next to other websites hosted on an existing webserver such as nginx, caddy, or apache

you can either:

  • give copyparty its own domain or subdomain (recommended)
  • or do location-based proxying, using --rp-loc=/stuff to tell copyparty where it is mounted -- has a slight performance cost and higher chance of bugs
    • if copyparty says incorrect --rp-loc or webserver config; expected vpath starting with [...] it's likely because the webserver is stripping away the proxy location from the request URLs -- see the ProxyPass in the apache example below

some reverse proxies (such as Caddy) can automatically obtain a valid https/tls certificate for you, and some support HTTP/2 and QUIC which could be a nice speed boost, depending on a lot of factors

  • warning: nginx-QUIC (HTTP/3) is still experimental and can make uploads much slower, so HTTP/1.1 is recommended for now
  • depending on server/client, HTTP/1.1 can also be 5x faster than HTTP/2

for improved security (and a tiny performance boost) consider listening on a unix-socket with -i /tmp/party.sock instead of -i 127.0.0.1

example webserver configs:

real-ip

teaching copyparty how to see client IPs when running behind a reverse-proxy, or a WAF, or another protection service such as cloudflare

if you (and maybe everybody else) keep getting a message that says thank you for playing, then you've gotten banned for malicious traffic. This ban applies to the IP address that copyparty thinks identifies the shady client -- so, depending on your setup, you might have to tell copyparty where to find the correct IP

for most common setups, there should be a helpful message in the server-log explaining what to do, but see docs/xff.md if you want to learn more, including a quick hack to just make it work (which is not recommended, but hey...)

prometheus

metrics/stats can be enabled at URL /.cpr/metrics for grafana / prometheus / etc (openmetrics 1.0.0)

must be enabled with --stats since it reduces startup time a tiny bit, and you probably want -e2dsa too

the endpoint is only accessible by admin accounts, meaning the a in rwmda in the following example commandline: python3 -m copyparty -a ed:wark -v /mnt/nas::rwmda,ed --stats -e2dsa

follow a guide for setting up node_exporter except have it read from copyparty instead; example /etc/prometheus/prometheus.yml below

scrape_configs:
  - job_name: copyparty
    metrics_path: /.cpr/metrics
    basic_auth:
      password: wark
    static_configs:
      - targets: ['192.168.123.1:3923']

currently the following metrics are available,

  • cpp_uptime_seconds time since last copyparty restart
  • cpp_boot_unixtime_seconds same but as an absolute timestamp
  • cpp_http_conns number of open http(s) connections
  • cpp_http_reqs number of http(s) requests handled
  • cpp_sus_reqs number of 403/422/malicious requests
  • cpp_active_bans number of currently banned IPs
  • cpp_total_bans number of IPs banned since last restart

these are available unless --nos-vst is specified:

  • cpp_db_idle_seconds time since last database activity (upload/rename/delete)
  • cpp_db_act_seconds same but as an absolute timestamp
  • cpp_idle_vols number of volumes which are idle / ready
  • cpp_busy_vols number of volumes which are busy / indexing
  • cpp_offline_vols number of volumes which are offline / unavailable
  • cpp_hashing_files number of files queued for hashing / indexing
  • cpp_tagq_files number of files queued for metadata scanning
  • cpp_mtpq_files number of files queued for plugin-based analysis

and these are available per-volume only:

  • cpp_disk_size_bytes total HDD size
  • cpp_disk_free_bytes free HDD space

and these are per-volume and total:

  • cpp_vol_bytes size of all files in volume
  • cpp_vol_files number of files
  • cpp_dupe_bytes disk space presumably saved by deduplication
  • cpp_dupe_files number of dupe files
  • cpp_unf_bytes currently unfinished / incoming uploads

some of the metrics have additional requirements to function correctly,

  • cpp_vol_* requires either the e2ds volflag or -e2dsa global-option

the following options are available to disable some of the metrics:

  • --nos-hdd disables cpp_disk_* which can prevent spinning up HDDs
  • --nos-vol disables cpp_vol_* which reduces server startup time
  • --nos-vst disables volume state, reducing the worst-case prometheus query time by 0.5 sec
  • --nos-dup disables cpp_dupe_* which reduces the server load caused by prometheus queries
  • --nos-unf disables cpp_unf_* for no particular purpose

note: the following metrics are counted incorrectly if multiprocessing is enabled with -j: cpp_http_conns, cpp_http_reqs, cpp_sus_reqs, cpp_active_bans, cpp_total_bans

other extremely specific features

you'll never find a use for these:

custom mimetypes

change the association of a file extension

using commandline args, you can do something like --mime gif=image/jif and --mime ts=text/x.typescript (can be specified multiple times)

in a config-file, this is the same as:

[global]
  mime: gif=image/jif
  mime: ts=text/x.typescript

run copyparty with --mimes to list all the default mappings

feature chickenbits

buggy feature? rip it out by setting any of the following environment variables to disable its associated bell or whistle,

env-var what it does
PRTY_NO_IFADDR disable ip/nic discovery by poking into your OS with ctypes
PRTY_NO_IPV6 disable some ipv6 support (should not be necessary since windows 2000)
PRTY_NO_LZMA disable streaming xz compression of incoming uploads
PRTY_NO_MP disable all use of the python multiprocessing module (actual multithreading, cpu-count for parsers/thumbnailers)
PRTY_NO_SQLITE disable all database-related functionality (file indexing, metadata indexing, most file deduplication logic)
PRTY_NO_TLS disable native HTTPS support; if you still want to accept HTTPS connections then TLS must now be terminated by a reverse-proxy
PRTY_NO_TPOKE disable systemd-tmpfilesd avoider

example: PRTY_NO_IFADDR=1 python3 copyparty-sfx.py

packages

the party might be closer than you think

if your distro/OS is not mentioned below, there might be some hints in the Β«on serversΒ» section

arch package

now available on aur maintained by @icxes

it comes with a systemd service and expects to find one or more config files in /etc/copyparty.d/

fedora package

does not exist yet; using the copr-pypi builds is NOT recommended because updates can be delayed by several months

nix package

nix profile install github:9001/copyparty

requires a flake-enabled installation of nix

some recommended dependencies are enabled by default; override the package if you want to add/remove some features/deps

ffmpeg-full was chosen over ffmpeg-headless mainly because we need withWebp (and withOpenmpt is also nice) and being able to use a cached build felt more important than optimizing for size at the time -- PRs welcome if you disagree πŸ‘

nixos module

for this setup, you will need a flake-enabled installation of NixOS.

{
  # add copyparty flake to your inputs
  inputs.copyparty.url = "github:9001/copyparty";

  # ensure that copyparty is an allowed argument to the outputs function
  outputs = { self, nixpkgs, copyparty }: {
    nixosConfigurations.yourHostName = nixpkgs.lib.nixosSystem {
      modules = [
        # load the copyparty NixOS module
        copyparty.nixosModules.default
        ({ pkgs, ... }: {
          # add the copyparty overlay to expose the package to the module
          nixpkgs.overlays = [ copyparty.overlays.default ];
          # (optional) install the package globally
          environment.systemPackages = [ pkgs.copyparty ];
          # configure the copyparty module
          services.copyparty.enable = true;
        })
      ];
    };
  };
}

copyparty on NixOS is configured via services.copyparty options, for example:

services.copyparty = {
  enable = true;
  # directly maps to values in the [global] section of the copyparty config.
  # see `copyparty --help` for available options
  settings = {
    i = "0.0.0.0";
    # use lists to set multiple values
    p = [ 3210 3211 ];
    # use booleans to set binary flags
    no-reload = true;
    # using 'false' will do nothing and omit the value when generating a config
    ignored-flag = false;
  };

  # create users
  accounts = {
    # specify the account name as the key
    ed = {
      # provide the path to a file containing the password, keeping it out of /nix/store
      # must be readable by the copyparty service user
      passwordFile = "/run/keys/copyparty/ed_password";
    };
    # or do both in one go
    k.passwordFile = "/run/keys/copyparty/k_password";
  };

  # create a volume
  volumes = {
    # create a volume at "/" (the webroot), which will
    "/" = {
      # share the contents of "/srv/copyparty"
      path = "/srv/copyparty";
      # see `copyparty --help-accounts` for available options
      access = {
        # everyone gets read-access, but
        r = "*";
        # users "ed" and "k" get read-write
        rw = [ "ed" "k" ];
      };
      # see `copyparty --help-flags` for available options
      flags = {
        # "fk" enables filekeys (necessary for upget permission) (4 chars long)
        fk = 4;
        # scan for new files every 60sec
        scan = 60;
        # volflag "e2d" enables the uploads database
        e2d = true;
        # "d2t" disables multimedia parsers (in case the uploads are malicious)
        d2t = true;
        # skips hashing file contents if path matches *.iso
        nohash = "\.iso$";
      };
    };
  };
  # you may increase the open file limit for the process
  openFilesLimit = 8192;
};

the passwordFile at /run/keys/copyparty/ could for example be generated by agenix, or you could just dump it in the nix store instead if that's acceptable

browser support

TLDR: yes

copyparty-ie4-fs8

ie = internet-explorer, ff = firefox, c = chrome, iOS = iPhone/iPad, Andr = Android

feature ie6 ie9 ie10 ie11 ff 52 c 49 iOS Andr
browse files yep yep yep yep yep yep yep yep
thumbnail view - yep yep yep yep yep yep yep
basic uploader yep yep yep yep yep yep yep yep
up2k - - *1 *1 yep yep yep yep
make directory yep yep yep yep yep yep yep yep
send message yep yep yep yep yep yep yep yep
set sort order - yep yep yep yep yep yep yep
zip selection - yep yep yep yep yep yep yep
file rename - yep yep yep yep yep yep yep
file cut/paste - yep yep yep yep yep yep yep
navpane - yep yep yep yep yep yep yep
image viewer - yep yep yep yep yep yep yep
video player - yep yep yep yep yep yep yep
markdown editor - - *2 *2 yep yep yep yep
markdown viewer - *2 *2 *2 yep yep yep yep
play mp3/m4a - yep yep yep yep yep yep yep
play ogg/opus - - - - yep yep *3 yep
= feature = ie6 ie9 ie10 ie11 ff 52 c 49 iOS Andr
  • internet explorer 6 through 8 behave the same
  • firefox 52 and chrome 49 are the final winxp versions
  • *1 yes, but extremely slow (ie10: 1 MiB/s, ie11: 270 KiB/s)
  • *2 only able to do plaintext documents (no markdown rendering)
  • *3 iOS 11 and newer, opus only, and requires FFmpeg on the server

quick summary of more eccentric web-browsers trying to view a directory index:

browser will it blend
links (2.21/macports) can browse, login, upload/mkdir/msg
lynx (2.8.9/macports) can browse, login, upload/mkdir/msg
w3m (0.5.3/macports) can browse, login, upload at 100kB/s, mkdir/msg
netsurf (3.10/arch) is basically ie6 with much better css (javascript has almost no effect)
opera (11.60/winxp) OK: thumbnails, image-viewer, zip-selection, rename/cut/paste. NG: up2k, navpane, markdown, audio
ie4 and netscape 4.0 can browse, upload with ?b=u, auth with &pw=wark
ncsa mosaic 2.7 does not get a pass, pic1 - pic2
SerenityOS (7e98457) hits a page fault, works with ?b=u, file upload not-impl

client examples

interact with copyparty using non-browser clients

  • javascript: dump some state into a file (two separate examples)

    • await fetch('//127.0.0.1:3923/', {method:"PUT", body: JSON.stringify(foo)});
    • var xhr = new XMLHttpRequest(); xhr.open('POST', '//127.0.0.1:3923/msgs?raw'); xhr.send('foo');

  • curl/wget: upload some files (post=file, chunk=stdin)

    • post(){ curl -F f=@"$1" http://127.0.0.1:3923/?pw=wark;}
      post movie.mkv (gives HTML in return)
    • post(){ curl -F f=@"$1" 'http://127.0.0.1:3923/?want=url&pw=wark';}
      post movie.mkv (gives hotlink in return)
    • post(){ curl -H pw:wark -H rand:8 -T "$1" http://127.0.0.1:3923/;}
      post movie.mkv (randomized filename)
    • post(){ wget --header='pw: wark' --post-file="$1" -O- http://127.0.0.1:3923/?raw;}
      post movie.mkv
    • chunk(){ curl -H pw:wark -T- http://127.0.0.1:3923/;}
      chunk <movie.mkv

  • bash: when curl and wget is not available or too boring

    • (printf 'PUT /junk?pw=wark HTTP/1.1\r\n\r\n'; cat movie.mkv) | nc 127.0.0.1 3923
    • (printf 'PUT / HTTP/1.1\r\n\r\n'; cat movie.mkv) >/dev/tcp/127.0.0.1/3923

  • python: u2c.py is a command-line up2k client (webm)

  • FUSE: mount a copyparty server as a local filesystem

  • sharex (screenshot utility): see ./contrib/sharex.sxcu

  • contextlet (web browser integration); see contrib contextlet

  • igloo irc: Method: post Host: https://you.com/up/?want=url&pw=hunter2 Multipart: yes File parameter: f

copyparty returns a truncated sha512sum of your PUT/POST as base64; you can generate the same checksum locally to verify uplaods:

b512(){ printf "$((sha512sum||shasum -a512)|sed -E 's/ .*//;s/(..)/\\x\1/g')"|base64|tr '+/' '-_'|head -c44;}
b512 <movie.mkv

you can provide passwords using header PW: hunter2, cookie cppwd=hunter2, url-param ?pw=hunter2, or with basic-authentication (either as the username or password)

NOTE: curl will not send the original filename if you use -T combined with url-params! Also, make sure to always leave a trailing slash in URLs unless you want to override the filename

folder sync

sync folders to/from copyparty

the commandline uploader u2c.py with --dr is the best way to sync a folder to copyparty; verifies checksums and does files in parallel, and deletes unexpected files on the server after upload has finished which makes file-renames really cheap (it'll rename serverside and skip uploading)

alternatively there is rclone which allows for bidirectional sync and is way more flexible (stream files straight from sftp/s3/gcs to copyparty, ...), although there is no integrity check and it won't work with files over 100 MiB if copyparty is behind cloudflare

  • starting from rclone v1.63, rclone is faster than u2c.py on low-latency connections

mount as drive

a remote copyparty server as a local filesystem; go to the control-panel and click connect to see a list of commands to do that

alternatively, some alternatives roughly sorted by speed (unreproducible benchmark), best first:

most clients will fail to mount the root of a copyparty server unless there is a root volume (so you get the admin-panel instead of a browser when accessing it) -- in that case, mount a specific volume instead

if you have volumes that are accessible without a password, then some webdav clients (such as davfs2) require the global-option --dav-auth to access any password-protected areas

android app

upload to copyparty with one tap

Get it on F-Droid '' f-droid version info '' github version info

the app is NOT the full copyparty server! just a basic upload client, nothing fancy yet

if you want to run the copyparty server on your android device, see install on android

iOS shortcuts

there is no iPhone app, but the following shortcuts are almost as good:

  • upload to copyparty (offline) (png) based on the original by Daedren (thx!)
    • can strip exif, upload files, pics, vids, links, clipboard
    • can download links and rehost the target file on copyparty (see first comment inside the shortcut)
    • pics become lowres if you share from gallery to shortcut, so better to launch the shortcut and pick stuff from there

performance

defaults are usually fine - expect 8 GiB/s download, 1 GiB/s upload

below are some tweaks roughly ordered by usefulness:

  • disabling HTTP/2 and HTTP/3 can make uploads 5x faster, depending on server/client software

  • -q disables logging and can help a bunch, even when combined with -lo to redirect logs to file

  • --hist pointing to a fast location (ssd) will make directory listings and searches faster when -e2d or -e2t is set

    • and also makes thumbnails load faster, regardless of e2d/e2t

  • --no-hash . when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable

  • if your volumes are on a network-disk such as NFS / SMB / s3, specifying larger values for --iobuf and/or --s-rd-sz and/or --s-wr-sz may help; try setting all of them to 524288 or 1048576 or 4194304

  • --no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1 minimizes the number of threads; can help in some eccentric environments (like the vscode debugger)

  • -j0 enables multiprocessing (actual multithreading), can reduce latency to 20+80/numCores percent and generally improve performance in cpu-intensive workloads, for example:

    • lots of connections (many users or heavy clients)
    • simultaneous downloads and uploads saturating a 20gbps connection
    • if -e2d is enabled, -j2 gives 4x performance for directory listings; -j4 gives 16x

    ...however it also increases the server/filesystem/HDD load during uploads, and adds an overhead to internal communication, so it is usually a better idea to don't

  • using pypy instead of cpython can be 70% faster for some workloads, but slower for many others

    • and pypy can sometimes crash on startup with -j0 (TODO make issue)

client-side

when uploading files,

  • chrome is recommended (unfortunately), at least compared to firefox:

    • up to 90% faster when hashing, especially on SSDs
    • up to 40% faster when uploading over extremely fast internets
    • but u2c.py can be 40% faster than chrome again

  • if you're cpu-bottlenecked, or the browser is maxing a cpu core:

    • up to 30% faster uploads if you hide the upload status list by switching away from the [πŸš€] up2k ui-tab (or closing it)
      • optionally you can switch to the lightweight potato ui by clicking the [πŸ₯”]
      • switching to another browser-tab also works, the favicon will update every 10 seconds in that case
    • unlikely to be a problem, but can happen when uploding many small files, or your internet is too fast, or PC too slow

security

there is a discord server with an @everyone for all important updates (at the lack of better ideas)

some notes on hardening

  • set --rproxy 0 if your copyparty is directly facing the internet (not through a reverse-proxy)
    • cors doesn't work right otherwise
  • if you allow anonymous uploads or otherwise don't trust the contents of a volume, you can prevent XSS with volflag nohtml
    • this returns html documents as plaintext, and also disables markdown rendering
  • when running behind a reverse-proxy, listen on a unix-socket with -i /tmp/party.sock instead of -i 127.0.0.1 for tighter access control (plus you get a tiny performance boost for free)

safety profiles:

  • option -s is a shortcut to set the following options:

    • --no-thumb disables thumbnails and audio transcoding to stop copyparty from running FFmpeg/Pillow/VIPS on uploaded files, which is a good idea if anonymous upload is enabled
    • --no-mtag-ff uses mutagen to grab music tags instead of FFmpeg, which is safer and faster but less accurate
    • --dotpart hides uploads from directory listings while they're still incoming
    • --no-robots and --force-js makes life harder for crawlers, see hiding from google

  • option -ss is a shortcut for the above plus:

    • --unpost 0, --no-del, --no-mv disables all move/delete support
    • --hardlink creates hardlinks instead of symlinks when deduplicating uploads, which is less maintenance
      • however note if you edit one file it will also affect the other copies
    • --vague-403 returns a "404 not found" instead of "401 unauthorized" which is a common enterprise meme
    • --nih removes the server hostname from directory listings

  • option -sss is a shortcut for the above plus:

    • --no-dav disables webdav support
    • --no-logues and --no-readme disables support for readme's and prologues / epilogues in directory listings, which otherwise lets people upload arbitrary (but sandboxed) <script> tags
    • -lo cpp-%Y-%m%d-%H%M%S.txt.xz enables logging to disk
    • -ls **,*,ln,p,r does a scan on startup for any dangerous symlinks

other misc notes:

  • you can disable directory listings by giving permission g instead of r, only accepting direct URLs to files
    • you may want filekeys to prevent filename bruteforcing
    • permission h instead of r makes copyparty behave like a traditional webserver with directory listing/index disabled, returning index.html instead
      • compatibility with filekeys: index.html itself can be retrieved without the correct filekey, but all other files are protected

gotchas

behavior that might be unexpected

  • users without read-access to a folder can still see the .prologue.html / .epilogue.html / README.md contents, for the purpose of showing a description on how to use the uploader for example
  • users can submit <script>s which autorun (in a sandbox) for other visitors in a few ways;
    • uploading a README.md -- avoid with --no-readme
    • renaming some.html to .epilogue.html -- avoid with either --no-logues or --no-dot-ren
    • the directory-listing embed is sandboxed (so any malicious scripts can't do any damage) but the markdown editor is not 100% safe, see below
  • markdown documents can contain html and <script>s; attempts are made to prevent scripts from executing (unless -emp is specified) but this is not 100% bulletproof, so setting the nohtml volflag is still the safest choice
    • or eliminate the problem entirely by only giving write-access to trustworthy people :^)

cors

cross-site request config

by default, except for GET and HEAD operations, all requests must either:

  • not contain an Origin header at all
  • or have an Origin matching the server domain
  • or the header PW with your password as value

cors can be configured with --acao and --acam, or the protections entirely disabled with --allow-csrf

filekeys

prevent filename bruteforcing

volflag fk generates filekeys (per-file accesskeys) for all files; users which have full read-access (permission r) will then see URLs with the correct filekey ?k=... appended to the end, and g users must provide that URL including the correct key to avoid a 404

by default, filekeys are generated based on salt (--fk-salt) + filesystem-path + file-size + inode (if not windows); add volflag fka to generate slightly weaker filekeys which will not be invalidated if the file is edited (only salt + path)

permissions wG (write + upget) lets users upload files and receive their own filekeys, still without being able to see other uploads

dirkeys

share specific folders in a volume without giving away full read-access to the rest -- the visitor only needs the g (get) permission to view the link

volflag dk generates dirkeys (per-directory accesskeys) for all folders, granting read-access to that folder; by default only that folder itself, no subfolders

volflag dky disables the actual key-check, meaning anyone can see the contents of a folder where they have g access, but not its subdirectories

  • dk + dky gives the same behavior as if all users with g access have full read-access, but subfolders are hidden files (as if their names start with a dot), so dky is an alternative to renaming all the folders for that purpose, maybe just for some users

volflag dks lets people enter subfolders as well, and also enables download-as-zip/tar

dirkeys are generated based on another salt (--dk-salt) + filesystem-path and have a few limitations:

  • the key does not change if the contents of the folder is modified
    • if you need a new dirkey, either change the salt or rename the folder
  • linking to a textfile (so it opens in the textfile viewer) is not possible if recipient doesn't have read-access

password hashing

you can hash passwords before putting them into config files / providing them as arguments; see --help-pwhash for all the details

--ah-alg argon2 enables it, and if you have any plaintext passwords then it'll print the hashed versions on startup so you can replace them

optionally also specify --ah-cli to enter an interactive mode where it will hash passwords without ever writing the plaintext ones to disk

the default configs take about 0.4 sec and 256 MiB RAM to process a new password on a decent laptop

https

both HTTP and HTTPS are accepted by default, but letting a reverse proxy handle the https/tls/ssl would be better (probably more secure by default)

copyparty doesn't speak HTTP/2 or QUIC, so using a reverse proxy would solve that as well -- but note that HTTP/1 is usually faster than both HTTP/2 and HTTP/3

if cfssl is installed, copyparty will automatically create a CA and server-cert on startup

  • the certs are written to --crt-dir for distribution, see --help for the other --crt options
  • this will be a self-signed certificate so you must install your ca.pem into all your browsers/devices
  • if you want to avoid the hassle of distributing certs manually, please consider using a reverse proxy

recovering from crashes

client crashes

frefox wsod

firefox 87 can crash during uploads -- the entire browser goes, including all other browser tabs, everything turns white

however you can hit F12 in the up2k tab and use the devtools to see how far you got in the uploads:

  • get a complete list of all uploads, organized by statuts (ok / no-good / busy / queued):
    var tabs = { ok:[], ng:[], bz:[], q:[] }; for (var a of up2k.ui.tab) tabs[a.in].push(a); tabs

  • list of filenames which failed:
    ​var ng = []; for (var a of up2k.ui.tab) if (a.in != 'ok') ng.push(a.hn.split('<a href=\"').slice(-1)[0].split('\">')[0]); ng

  • send the list of filenames to copyparty for safekeeping:
    await fetch('/inc', {method:'PUT', body:JSON.stringify(ng,null,1)})

HTTP API

see devnotes

dependencies

mandatory deps:

  • jinja2 (is built into the SFX)

optional dependencies

install these to enable bonus features

enable hashed passwords in config: argon2-cffi

enable ftp-server:

  • for just plaintext FTP, pyftpdlib (is built into the SFX)
  • with TLS encryption, pyftpdlib pyopenssl

enable music tags:

  • either mutagen (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
  • or ffprobe (20x slower, more accurate, possibly dangerous depending on your distro and users)

enable thumbnails of...

  • images: Pillow and/or pyvips and/or ffmpeg (requires py2.7 or py3.5+)
  • videos/audio: ffmpeg and ffprobe somewhere in $PATH
  • HEIF pictures: pyvips or ffmpeg or pyheif-pillow-opener (requires Linux or a C compiler)
  • AVIF pictures: pyvips or ffmpeg or pillow-avif-plugin
  • JPEG XL pictures: pyvips or ffmpeg

enable smb support (not recommended):

  • impacket==0.11.0

pyvips gives higher quality thumbnails than Pillow and is 320% faster, using 270% more ram: sudo apt install libvips42 && python3 -m pip install --user -U pyvips

dependency chickenbits

prevent loading an optional dependency , for example if:

  • you have an incompatible version installed and it causes problems
  • you just don't want copyparty to use it, maybe to save ram

set any of the following environment variables to disable its associated optional feature,

env-var what it does
PRTY_NO_ARGON2 disable argon2-cffi password hashing
PRTY_NO_CFSSL never attempt to generate self-signed certificates using cfssl
PRTY_NO_FFMPEG audio transcoding goes byebye, thumbnailing must be handled by Pillow/libvips
PRTY_NO_FFPROBE audio transcoding goes byebye, thumbnailing must be handled by Pillow/libvips, metadata-scanning must be handled by mutagen
PRTY_NO_MUTAGEN do not use mutagen for reading metadata from media files; will fallback to ffprobe
PRTY_NO_PIL disable all Pillow-based thumbnail support; will fallback to libvips or ffmpeg
PRTY_NO_PILF disable Pillow ImageFont text rendering, used for folder thumbnails
PRTY_NO_PIL_AVIF disable 3rd-party Pillow plugin for AVIF support
PRTY_NO_PIL_HEIF disable 3rd-party Pillow plugin for HEIF support
PRTY_NO_PIL_WEBP disable use of native webp support in Pillow
PRTY_NO_PSUTIL do not use psutil for reaping stuck hooks and plugins on Windows
PRTY_NO_VIPS disable all libvips-based thumbnail support; will fallback to Pillow or ffmpeg

example: PRTY_NO_PIL=1 python3 copyparty-sfx.py

  • PRTY_NO_PIL saves ram
  • PRTY_NO_VIPS saves ram and startup time
  • python2.7 on windows: PRTY_NO_FFMPEG + PRTY_NO_FFPROBE saves startup time

optional gpl stuff

some bundled tools have copyleft dependencies, see ./bin/#mtag

these are standalone programs and will never be imported / evaluated by copyparty, and must be enabled through -mtp configs

sfx

the self-contained "binary" (recommended!) copyparty-sfx.py will unpack itself and run copyparty, assuming you have python installed of course

you can reduce the sfx size by repacking it; see ./docs/devnotes.md#sfx-repack

copyparty.exe

download copyparty.exe (win8+) or copyparty32.exe (win7+)

copyparty-exe-fs8

can be convenient on machines where installing python is problematic, however is not recommended -- if possible, please use copyparty-sfx.py instead

  • copyparty.exe runs on win8 or newer, was compiled on win10, does thumbnails + media tags, and is currently safe to use, but any future python/expat/pillow CVEs can only be remedied by downloading a newer version of the exe

  • dangerous: copyparty32.exe is compatible with windows7, which means it uses an ancient copy of python (3.7.9) which cannot be upgraded and should never be exposed to the internet (LAN is fine)

  • dangerous and deprecated: copyparty-winpe64.exe lets you run copyparty in WinPE and is otherwise completely useless

meanwhile copyparty-sfx.py instead relies on your system python which gives better performance and will stay safe as long as you keep your python install up-to-date

then again, if you are already into downloading shady binaries from the internet, you may also want my minimal builds of ffmpeg and ffprobe which enables copyparty to extract multimedia-info, do audio-transcoding, and thumbnails/spectrograms/waveforms, however it's much better to instead grab a recent official build every once ina while if you can afford the size

zipapp

another emergency alternative, copyparty.pyz has less features, requires python 3.7 or newer, worse compression, and more importantly is unable to benefit from more recent versions of jinja2 and such (which makes it less secure)... lots of drawbacks with this one really -- but it may just work if the regular sfx fails to start because the computer is messed up in certain funky ways, so it's worth a shot if all else fails

run it by doubleclicking it, or try typing python copyparty.pyz in your terminal/console/commandline/telex if that fails

it is a python zipapp meaning it doesn't have to unpack its own python code anywhere to run, so if the filesystem is busted it has a better chance of getting somewhere

  • but note that it currently still needs to extract the web-resources somewhere (they'll land in the default TEMP-folder of your OS)

install on android

install Termux + its companion app Termux:API (see ocv.me/termux) and then copy-paste this into Termux (long-tap) all at once:

yes | pkg upgrade && termux-setup-storage && yes | pkg install python termux-api && python -m ensurepip && python -m pip install --user -U copyparty && { grep -qE 'PATH=.*\.local/bin' ~/.bashrc 2>/dev/null || { echo 'PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && . ~/.bashrc; }; }
echo $?

after the initial setup, you can launch copyparty at any time by running copyparty anywhere in Termux -- and if you run it with --qr you'll get a neat qr-code pointing to your external ip

if you want thumbnails (photos+videos) and you're okay with spending another 132 MiB of storage, pkg install ffmpeg && python3 -m pip install --user -U pillow

  • or if you want to use vips for photo-thumbs instead, pkg install libvips && python -m pip install --user -U wheel && python -m pip install --user -U pyvips && (cd /data/data/com.termux/files/usr/lib/; ln -s libgobject-2.0.so{,.0}; ln -s libvips.so{,.42})

reporting bugs

ideas for context to include, and where to submit them

please get in touch using any of the following URLs:

in general, commandline arguments (and config file if any)

if something broke during an upload (replacing FILENAME with a part of the filename that broke):

journalctl -aS '48 hour ago' -u copyparty | grep -C10 FILENAME | tee bug.log

if there's a wall of base64 in the log (thread stacks) then please include that, especially if you run into something freezing up or getting stuck, for example OperationalError('database is locked') -- alternatively you can visit /?stack to see the stacks live, so http://127.0.0.1:3923/?stack for example

devnotes

for build instructions etc, see ./docs/devnotes.md

see ./docs/TODO.md for planned features / fixes / changes

copyparty's People

Contributors

9001 avatar chinponya avatar clach04 avatar daedren avatar icxes avatar jeremystartm avatar kipukun avatar lytexx avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

daedren ohsevin securecloud-biz trellixvulnteam dmgolembiowski simhaonline exactly9000 kipukun albertogp chinponya icxes heholek arkanic clach04 ppang0405 mvasuki1 hero ruanwernerkrugerinterversialsystem ghbook pinkpapa69 jadeluo agr96 scantist-ossops-m2 dariussg lewlite mugecevik ashbeats py7hon ehsanre1376

copyparty's Issues

Better configuration

The current way to do configuration using a file is a bit silly and it's not so clear how the various command line arguments (which aren't exactly obvious either) translate to a config file. Comments are pretty much necessary if you don't look at your config too often. Using json/yaml/whatever people use now for configuration would improve readability.

Aside from that, I don't think it currently gives any message if there are config errors (tested by putting gibberish at random points in copyparty.conf, no messages in logs and starts up normally (ideally should probably not start at all))

Versionned docker images

Hello, first of all, thanks for this amazing tool πŸ˜„

As of right now, the only docker images available are tagged with "latest", there is no version tagged images (For example ghcr.io/9001/copyparty-im:v1.9.15 / ghcr.io/9001/copyparty-im:1.9.15).

On Docker alone, it's not much of an issue as you can just pull the image and get the update, but on Kubernetes (What I use), I have to either add or remove the "latest" tag from the deployment to force a rollout of the update.

I know that Kubernetes is not a supported use case, but having version tagged images would probably make some users more comfortable, as some people don't like to use the "latest" tag and prefer to stay on a specific version instead of being on the edge.

requesting ?doc= of large files quickly exhausts memory

Suppose we have a publicly accessible file in /public/2GBfile.mkv. Making a request to http://localhost:3923/public/?doc=2GBfile.mkv will cause the file to be fully loaded into memory, where it likely gets copied a few times, spiking the memory usage by about 5x the file size. This often leads to the OS' OOM-killer acting on the program, even on systems with a ton of available memory.

I'm not exactly sure what's the best mitigation to apply. Maybe the max file size permitted for ?doc= should be limited (and configurable?) or maybe it should never be an option to begin with on binary files.

This issue shouldn't be specific to my setup, but just in case, here's what I have:
copyparty version 1.2.10 from pypi (the same thing happens on copyparty-sfx.py bundle), running under systemd
Python 3.9.6

direct python dependencies:
jinja2 3.0.2
mutagen 1.45.1
Pillow 0.3.2
pyopenssl 20.0.1
pyftpdlib 1.5.6

indirect python dependencies:
Babel 3.9.1
pytz 2021.3
markupsafe 2.0.1
olefile 0.46
defusedxml 0.7.1
packaging 20.9
pyparsing 2.4.7
six 1.16.0
pycparser 2.20
cffi 1.14.6
cryptography 3.4.8
pyasn 0.4.8
idna 3.2
pysendfile 2.0.1

other:
ffmpeg-full 4.4.1

config:

-i 0.0.0.0
-nc 10000
--http-only
--no-reload
--no-stack
--no-rescan
--no-robots
--no-dedup
--th-mt 8
u hotdog:hotdog
/srv/copyparty/public
/public
r
rwmd hotdog
c e2dsa
c e2ts

Ctrl+click in search menu dies

  1. Search for file
  2. Ctrl-click it
  3. Get bug screen because "td.onclick is null"

Full log:

/.cpr/browser.js @4374:4
TypeError: td.onclick is null

UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0

stack:
gclick@/.cpr/browser.js:4374:4
gclick1@/.cpr/browser.js:4352:17

ac_aac 0 ac_flac 1 ac_oth 1 au_compact 0 au_eq_amp 0 au_eq_chw 1 au_eq_gain [4,3,2,1,0,0,1,2,3,4] au_follow 0 au_fullpre 0 au_npclip 0 au_open 0 cpp_keynot rekobo_alnum csel 1 dotfiles 0 entreed tree filecols ["vq","aq","Vc","Ac","Fmt","Res","fps"] fman_clip 15 items fsearch 0 ganim slideIn griden 1 gridln 99 gridsel 0 gridsz 10 have_webp ya hfilecols [".vq",".aq","vc","ac","fmt","res",".fps"] icob icof 000 icot πŸ’– idxh 0 key_notation rekobo_alnum lifetime 1800 light 0 nthread 1 opmode search pb_mode next potato 0 sbars 1 theme bz b z thumbs 1 treesz 26 vmode S vol 1 wraptree 0

console:

    1700313997033 @
    1700313995391 log: h-repl /gremious/#q=name%20like%20*a*
    1700313995363 log: resetting fsort null
    1700313995363 log: failed to apply sort config: TypeError: nodes[b].href is undefined
    1700313990397 log: msg:iheight #epi 178
    1700313990298 log: h-push /gremious/
    1700313988315 log: msg:iheight #epi 502
    1700313988181 log: sha-ok
    1700313988174 log: h-repl /
    1700313988166 log: log-capture ok

readme states Python 2 and 3 supported - errors with Python 2.7.17 and Python 3.6.9

Describe the bug

Either docs are wrong or code is wrong:

copyparty/README.md

Line 5 in 9c28ba4

* server only needs Python (2 or 3), all dependencies optional

Doc claim Python 2 and 3 support. Setup.py only mentions 3, but mentions 3.3-3.12

Copyparty fails with:

  • Python 2.7.17
  • Python 3.6.9
  • Python 3.8.10

Does work with later version of Python, 3.11

To Reproduce

Checkout code, assume dependencies installed (e.g. bare minimum jinja), start server using:

python -m copyparty

from checkout directory.

Works fine for 3.11, fails with bad syntax for others. Possibly annotation / type hint related (Python 3.7 feature?)

Python 2

Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 163, in _run_module_as_main
mod_name, _Error)
File "/usr/lib/python2.7/runpy.py", line 111, in _get_module_details
import(mod_name) # Do not catch exceptions initializing package
File "copyparty/init.py", line 19
unicode: Callable[[Any], str] = str
^
SyntaxError: invalid syntax

Python3

Type error on type hints

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/clach04/code/py/copyparty/copyparty/__main__.py", line 28, in <module>
    from .authsrv import expand_config_file, re_vol, split_cfg_ln, upgrade_cfg_fmt
  File "/home/clach04/code/py/copyparty/copyparty/authsrv.py", line 16, in <module>
    from .bos import bos
  File "/home/clach04/code/py/copyparty/copyparty/bos/bos.py", line 6, in <module>
    from ..util import SYMTIME, fsdec, fsenc
  File "/home/clach04/code/py/copyparty/copyparty/util.py", line 36, in <module>
    from .stolen import surrogateescape
  File "/home/clach04/code/py/copyparty/copyparty/stolen/surrogateescape.py", line 35, in <module>
    def surrogateescape_handler(exc: Any) -> tuple[str, int]:
TypeError: 'type' object is not subscriptable

Expected behavior

Out of box doc instructions to allow start up.

Either doc needs update or type hints should be removed. For Python

Screenshots

NA

Server details

  • server OS / version: any platform
  • python version: 2.x 3.x - see above for details
  • copyparty arguments: none, default
  • filesystem (lsblk -f on linux): Any

Client details

NA

Additional context

Maybe doc last supported release for python version in docs? E.g. copyparty release ABC is the last release for Python 2.7, copyparty release XYZ is the last release for Python 3.3

NOTE copyparty-sfx.py download does work with older versions, this is specific to code checkout.

Product configuration wizard

is your feature request related to a problem? Please describe.
The product has so many configuration parameters with a lot of cross-dependencies between them already and new options keep coming with new releases. It would be not so easy task for a new user to go through all of them to configure a new product instance correctly according to the environment and business needs. Later, if the admin doesn't read each new release note, it would be difficult to stay on the same page and keep the product config in an optimal state.

Describe the idea / solution you'd like
I propose to implement some stand-alone script, which would represent an installation and migration wizard with the next flow:

  • in case of fresh deployment it should guide a user through all the configuration options subsequently (respecting dependencies) with a detailed and clear description of each option, its impact, use cases, and examples. There might be a lot of steps, so some progress indication, intermediate save, and ability to stop and later get back to the same point is needed.
  • in case of an upgrade from a previous version (in-place), it should get the previous config and guide a user through the delta between the versions: pretty the same, but with much fewer steps, and if something previously available was changed, it should be clearly indicated.
  • in case of a migration to another system (side-by-side), I'm not sure what is better: from 1 side asking a user to go through all the steps, as for fresh installation while a significant part of options from the source deployment might be the same looks like wasting time, but from another side, we couldn't be sure which options should be changed and which - not: answer to this question is from user's business area, not from technical one. However, I'm not an expert in the Copyparty, so I might not see the whole picture.
    After the wizard completion, there should be a config for Copyparty with all user answers reflected and a step-by-step guide on how to deploy the product and apply the config (all supported setups should be taken into account as part of the wizard)

Describe any alternatives you've considered
Keep it "as is" and force a user to manually configure the product putting all the responsibility of misconfiguration on the user's side

Additional context
This request came from my personal experience of analyzing a long list of config params with not-so-clear (as to me) descriptions, and I believe that my setup is not optimal (it is running in a local network behind a firewall, so it isn't so critical, but still...)
Feel free to ask some questions in comments in case you need to clarify something)

xvol allows symlinks to escape the root

not sure if this is design intent or not, but I am able to escape from a volume root via symlinks...

I created a symlink in ~/test pointing to ../../../backup.
cd ~/test && ln -s ../../../backup backup

Then I started copyparty in ~/test using the following args:
copyparty -a foo:bar -v .::r:rwdm,foo:c,xvol

I am able to see the backup folder in my root and can follow the symlink and escape from the volume root. This is also reproducible with absolute symlinks like:
cd ~/test && ln -s /backup backup

version info:
running on ubuntu 18.04 LTS
copyparty v1.6.15 "cors k" (2023-04-26)
CPython v3.6.9 on Linux64 [GCC 8.4.0]
sqlite v3.22.0*1 | jinja2 v3.0.3 | pyftpd v(None)

PKGBUILD improvements

Starting this so I don't forget

  • better pkgver() (first try didn't work so good, second try doesn't actually have the version, just the commit id) manual/automatic-on-release pkgver
  • the build process might be improved, looks a bit sus currently (still works tho) ("grab the web-dependencies" had some problems so didn't include it yet) just download the latest sfx instead of building
  • the config include thing mentioned in #15
  • add to AUR so I don't have to manually update (maybe when the PKGBUILD is good enough)
  • note what functionality optional deps are needed for
  • separate PKGBUILDs for release and git versions just release should be fine
  • ???

"Make symlink" button alongside the cut/paste ones

Hello!
I think it would be nice if right next to the cut/paste buttons when you select a file, there was also a "link" button. Very useful when running with no-dedup and wanting to selectively ""copy"" some files, esp. when sharing across volumes.

(And since windows is bad at symlinks I guess just don't show it unless you're admin/have hardlinks enabled on windo.)

WebDav support on Windows

There's no need for complicated (at least for normal users). This handy Windows registry hack (WebDav_Hack.txt, please note that you'll need to change .txt to .reg before importing it, because of GitHub complaining not supporting .reg files) can enable native WebDav support.

I may create a pull request with this.

Docker-Compose

Hello, you said you were unfamiliar with docker compose so I rewrote the run command in compose. If you'd like you can include this in the same section as the docker run command.

name: copyParty
services:
    ac:
        stdin_open: true
        tty: true
        user: "1000"
        ports:
            - 3923:3923
        volumes:
            - /mnt/nas:/w
            - $PWD/cfgdir:/cfg
        image: copyparty/ac

dropzones activated by dragging links

chrome bug -- affects current stable chrome 96.0.4664.45

appeared in chromium somewhere after 95.0.4631 and before 97.0.4666, disappeared again before chromium 98.0.4719

on windows, toolbar items become .url files when dragged, which upload successfully as windows shortcuts

possible workaround if chrome 98 doesn't fix it is checking dataTransfer.items[0].kind in the dragover handler, seems to be string if bad

(not doing anything about it for now because it looks like chrome is about to fix it)

Feature suggestion: Remember sort between page loads

I often sort by date.
Currently I have to toggle it to sort by date every time I reload the page.
I think it would be cool if it was remembered between page loads.
Although if this is implemented, then some way to reset sorting would be necessary, too. (Well, I suppose it sorts by name by default, and you can just toggle that, but might not be obvious at first).

Markdown placeholders

Describe the idea / solution you'd like
Placeholders for markdown files, specifically README.md that are obtained from the server (e.g. ip address) - to be rendered and shown below the file listing (where README.md is usually shown) - so for example, {request_ip_address} will be replaced with the IP address of the user accessing the file when it is rendered.

Additional context
I'm aware that this idea is kinda stupid, do close if you want :)

image

Poor message server

is your feature request related to a problem? Please describe.

I want to share "stuff" from my Android phone. Files, but also snippets, calendar entrires, and links.

I'd like a way to "see" these on a PC (in one place).

Describe the idea / solution you'd like

A crappy chat program :-p

Describe any alternatives you've considered

Full chat, which I do NOT want/need.

Additional context

I put together the following which works well enough for me. Any value in me contributing this?

#!/usr/bin/env python
# -*- coding: us-ascii -*-
# vim:ts=4:sw=4:softtabstop=4:smarttab:expandtab

import locale
import os
import sys
import json

"""
use copyparty as a dumb IRC messaging server
Sample usage:

    mkdir -p upload_here
    python dist/copyparty-sfx.py -v upload_here::rw --xm f,j,t10,bin/hooks/dumb_message_logger.py


Where:

    xm = execute on message-to-server-log
    f = fork so it doesn't block uploads
    j = provide message information as json; not just the text - this script REQUIRES json
    t10 = timeout and kill download after 10 secs

"""


def main(argv=None):
    if argv is None:
        argv = sys.argv

    message_info = json.loads(sys.argv[1])
    message_filename = os.environ.get('COPYPARTY_MESSAGE_FILENAME', 'messages.txt')  # .log gets downloaded, but .txt gets displayed in CopyParty web UI. Avoid .md to avoid being edited with Makrdown editor? (and .hist updates)
    # basic string template, works with (almost) any old version of Python
    message_logging_template = """
------
%(at)s  TODO ISO datetime format (or even datetime.now - either local time or UTC)
User: %(user)s
Client machine: %(ip)s  # TODO option for reverse DNS name lookup
%(txt)s
"""

    message_filename = os.path.join(message_info['ap'], message_filename)  # filename is actual shared file system shared
    f = open(message_filename, 'a')  # options binary and force write in ut8f, or use a codec
    try:
        f.write(message_logging_template % message_info)
    finally:
        f.close()

    #print(message_logging_template % message_info)  # assume output encoding is going to work fine (TODO force utf-8 - will do so for file output)

    return 0


if __name__ == "__main__":
    sys.exit(main())

html5 / js Web Share API PWA

Idea would be to write in the web client a small progressive web app (PWA) that implements the Share Web API client. This would mean users could install fro the web browser a small share application essentially what Party-Up does with significantly less code. Potentially also allows multiple share to apps to be installed (one per server, i.e. no need to edit config when switching between servers).

I believe this would be Android ONLY. I'm not sure what (if any plans) there are for Apple iOS implementations (I don;t have a device to test with).

This has the potential to replace Party-Up for some uses cases (unclear if this is a pro or a con, Party-Up works great so far with my experiments I'm not suggesting is needs to be replaced), specifically https.

Disadvantage of Web Share API is that it (understandably) requires https, Party-Up can work with plain http as well as https, so I don't think this idea would ever remove the need for Party-Up.

Unclear about basic-auth implications.

Resources

pypy3 bug

pypy3 produces duplicate rows on sqlite3 queries, usually partially populated
pypy2 from the same build is fine
no apparent db corruption occurs
gentoo chroot on alpine

pypy3 -m copyparty -v /export/::red -v /mnt/sdh_ov/inc/:/inc:aed --https-only -e2dsa

Python 3.6.12 (db1e853f94de42ad711bd930222bd2434e0f900d, Feb 09 2021, 01:37:44)
[PyPy 7.3.3 with GCC 9.2.0] on linux

Python 2.7.18 (a29ef73f9b32953753d0dd6d2a56255fa2892e24, Feb 09 2021, 02:02:43)
[PyPy 7.3.3 with GCC 9.2.0] on linux2

output from 9af64d6 with ~85k files

WARN: multiple entries: [/export] => [music/.../...nce/04. Impatience.ogg] |2|
('YwxmQTKXRLOxZ6QO6qFFJbt9DcyGU3d7x42CJM3W6DM', 1525061569, None, None, None)
('YwxmQTKXRLOxZ6QO6qFFJbt9DcyGU3d7x42CJM3W6DM', 1525061569, 5127515, 'music/.../...nce', '04. Impatience.ogg')

WARN: multiple entries: [/export] => [music/.../...04-30)/06. Gugun 955.m4a] |2|
('RAerl8BK2c2_p3eYHmZQ9EYLwNZCADIx4nwvAHuIp50', 1526496175, 9603800, 'music/.../...04-30)', '')
('RAerl8BK2c2_p3eYHmZQ9EYLwNZCADIx4nwvAHuIp50', 1526496175, 9603800, 'music/.../...04-30)', '06. Gugun 955.m4a')

WARN: multiple entries: [/export] => [music/.../guide/foo.jpg] |2|
('YXuDWM05p0YMed7FUlmsjiri4ukIIxF9KOR-4A6pt1M', 1290479772, 1341643, 'music/.../guide', 'foo.jpg')
('YXuDWM05p0YMed7FUlmsjiri4ukIIxF9KOR-4A6pt1M', 1290479772, 1341643, 'music/.../guide', 'foo.jpg')

Per volume size/upload limit.

is your feature request related to a problem? Please describe.

I want to share my copyparty instance with my friends so that they can use it for cloud storage/sharing, but I want to set e.g. a 1 TB upload limit for the volume I give to them.

Describe the idea / solution you'd like

I think the easier solution is definitely just a volflag like maxup/maxsize: 1tb which would just check the folder/volume size on upload, and if it's over the limit, deny the upload, no matter the IP.

Then you can simply set up a "user folder", and all works well.

Describe any alternatives you've considered

I suppose I can do maxb=1tb,9999999999... but that feels very hacky.

It is also different from checking volume size, since maxb is per ip and you can just use your phone or a vpn or something, not as hard of a limit, that's better at deterring random short-term users from messing around.

Also, maybe good idea to have a 50gb folder limit on like, a "shared"/"public" volume used by many random users. Also makes it possible to limit a volume used by two users, e.g. for a friend and their partner, where IP blocking won't work.

I've also consider per-user tracking, but:

  1. It would be far more annoying to do because you'd have to keep track of how much the user has uploaded/deleted somewhere (what if an upload breaks or the server internet cuts off?)
  2. It would not work for anon users e.g. friends-of-friends, where I provide a "shared" folder that has short-lived files so they can use it to go over e.g. Discord upload limits.

Additional context
This app has so many features....
It is also the best one I've found, and I really searched...good job !!

browser.js error

Describe the bug
getting this:
/.cpr/browser.js @947:7
TypeError: L is undefined

To Reproduce
no idea, just tried visiting the base url on localhost

Expected behavior
i dont get this error

Screenshots
see below

Server details
if the issue is possibly on the server-side, then mention some of the following:

  • server OS / version: 1.9.6

Client details
if the issue is possibly on the client-side, then mention some of the following:

  • the device type and model: windows
  • browser version: Firefox 117

Additional context
error

Tooltip delay

I think a short delay like maybe 1 second before a tooltip pops up would be nice, same with filenames.

A bunch of security questions and some requests

Hello again o/

I'm considering maybe having a lil, short-lived, public/ volume open to, well, the public, mostly for my friends to go over e.g. Discord size limits. However, should a malicious party stumble upon it, I'd like to be secure. Now, all the other volumes I have are locked behind accounts, so I'm quite happy with them e.g. generating thumbnails and the like, so even -s is not really nice for my day-to-day use.

Now, it might just be reasonable to instead give all my friends an account, or set a global password in nginx, so I'm considering doing that instead.

However, that still leads me to the question: If I wanted a fully publicly available folder, is that just a bad idea, period, or are there any things I could be doing to reasonably secure myself, while not restricting myself, i.e. ether only liming a) anon accounts, or b) on a per-volume basis?

Could copyparty implement some existing security features, like no-readme and no-thumb, but scoped, e.g. files uploaded to this specific volume should be marked, and will never generate thumbnails for anyone? (or an accs permission to generate thumbnails in the same vein) (though perhaps easier, maybe just generate an empty/"safe" thumbnail)

On that note, do folders with only G: * permissions still generate thumbnails?

Basically, perhaps going over some of the global security flags and seeing if any of them make sense per volume would be nice and worth doing. In an ideal world, the worst an anon user should be able to do it fill up the whole space with racism.png rather than exploit ffmpeg/<script>'s/symlinks, and I'd like to have the nice privileges still available to use for myself/trusted friends.

though maybe ppl should just avoid public folders entirely accessible by anons in the first place idk.

More sidebits:

  1. I checked the logs, I don't believe copyparty logs uploader IPs? It'd be nice to see the client IP right next to the uploaded file name. If some anon gets in my site and decided to be an unfunny guy and flood my shared folder with unfriendly images, I'd rather like to be able to just check the logs and block that IP in nginx or w/e.

  2. It would be nice to have a global option to set the default ban timer, so that I can set it to forever. Anybody who tries to to bruteforce a password or hits 50 404's in 60min is no friend of mine, and I'd rather not have to deal with them in 24 hour hours ether. And on that note, does copyparty log bans + ip, so that I can also just ban them in nginx generally? If they're malicious on copyparty, I don't want them on any of my services.

Thanks ❀️

Implement lazy loading for large directories

Sorry, I don't know if lazy loading is the right term but I'm sure you get it

is your feature request related to a problem? Please describe.
It takes a long time (up to or more than 5 seconds, and that's not including loading the thumbnails) to load a directory with over 10000 items (at least when using the grid)

Describe the idea / solution you'd like
Add an option to load a few hundred/thousand/user-defined amount of items at a time instead

Describe any alternatives you've considered
Deleting unnecessary files (unacceptable)

Additional context
Search would ideally still work on all items and splitting the items into pages would be a poor idea

Folders being deleted unexpectedly

Hi, Nice little project! Just testing it out and noticed some oddities regarding folders that may be design intent, but thought I'd inquire :)

  1. when deleting the only file in a folder, the folder is also deleted. this leaves the UI with a spinning christmas tree and the message "could not list files in folder: error 404: file not found". refreshing the UI shows that the folder has been deleted. up2k log indicates "deleted 1 files (and 1/1 folders)". Perhaps this is expected behaviour, and there is a config option to disable this pruning?
  2. when moving an empty folder, the folder is deleted. perhaps a similar cause as above. I came across this after creating 2 folders in a row: dir1 and then dir2. This created /dir1/dir2, which I wasn't expecting, so I tried moving /dir1/dir2 to /dir2, so I would have 2 folders at the top level of the root. Both folders ended up deleted. I also tried the opposite - creating /dir1 and /dir2 and then tried moving /dir2 into /dir1. I was left with only /dir1 (dir2 was deleted).
  3. renaming an empty folder causes it to be deleted. similar pruning to above is happening it seems. create /dir1. rename /dir1 to /dir2 - it disappears :)

Anyhow. Thanks for your time!

version info:
running on ubuntu 18.04 LTS
copyparty v1.6.15 "cors k" (2023-04-26)
CPython v3.6.9 on Linux64 [GCC 8.4.0]
sqlite v3.22.0*1 | jinja2 v3.0.3 | pyftpd v(None)

Open known text filetypes in editor

It would be nice if I could open not just .md files, but a few other files that are known text data wtih the editor.

Right now I have a music folder and I wanted to doublecheck an album.nfo, but it just opens it for download. Works good if I append ?edit to the link manually though.

I think just accepting txt/json/config/log/yaml/nfo/etc. would be nice, lets me quickly double check a config file in the browser.

Cut filenames to keep icon size consistant

i'm happy to see how much progress you are making on this i really admire your dedication

Like you can see below it can get quite messy if some filenames are really long

image

it would be nice to have an option to use the behavior google drive and mega use
image

Got myself banned from my own server because private files showed up in a search

I have a 'private' volume with permissions only to myself:

accs:
     rwmda: gremious
   flags:
     fk: 8
     fka,

(I now notice that fk is useless there but that's what it was)

  1. Opened new Firefox private window
  2. Searched "a" to test #60
  3. See files from said private volumes in the search, but they had no thumbnails
  4. Get concerned, click file
  5. "thank you for playing"
  6. ok

Client-side setting to use ctrl-click for multiselect

Right now, to select anything, I think your only option is to press the multiselect button explicitly and then start clicking files.

I would love if it was like any other file ui, from windows explorer to google photos, where you can ctrl+clicking to initiate multi-select. Just makes things faster and more intuitive.

Browsers by default make ctrl-click "open link in new tab" I think (same as middle click), at least firefox does, so it'd probably be annoying to users if that functionality was "removed". And it'd maybe be overkill as a full on config option (though I wouldn't be against a "enable by default" setting like with grid). I think it'd be nice to at least have that as a toggle in the little cog menu at the top.

Full, uncropped thumbnails

Currently, the generated thumbnails are square, and unfortunately this also cuts-off a lot of images. You can see this in the repo's own image example, where e.g. the text "Internet!" on the bottom left picture is cut off, because the image is "too tall".

image

I've tried fiddling with the css but then realized the generated thumbnails were just square. This is a bit inconvenient when browsing photos I've taken on my phone, since a lot of those tend to be in portrait. If I'm browsing e.g. mirror selfies, the crop will cut off at my face/show mainly my chest, which is less than ideal.

I'd like an option to generate the thumbnails without cropping, ones which show whole image, even if it doesn't neatly fit into the square, like e.g. Windows Explorer does:

image

Thumbnails are generated using third-party libs, but I do not know how copyparty uses them. If I'm supposed to add a config file for them instead or something I'm happy to do that, though I would appreciate some directions.

If it's up to copyparty to do this though, then I think it should provide a config flag that toggles what type of thumbnail to generate (crop to center vs fit). pyvips, for example, has a crop parameter which you can adjust. I personally think it's generally ok if e.g. it only works on pyvips, but perhaps ffmpeg/pillow also have such settings.

(I have pyvips installed, but I cannot actually tell what copyparty is using? Is there a config setting I'm expected to specify, or will it just always prefer pyvips over ffmpeg even if it finds both?)

Automatic per user directories?

I'm not sure if this is already possible or if this is effectively a feature request.

I have my users managed via the proxy. I'd like each user to get a directory that they'd have broad permissions to. Eg, for the user sue, I'd like them to have /mnt/files/sue/ (and bob would get /mnt/files/bob/). Ideally in this config this would be a single config entry with a placeholder, something like:

[/user]
  /mnt/files/{$USER}/

(it's ok if it needs to be a unique directory per user too, eg /user/${USER} in the above example)

Why

The problem this solves for me is that I have multiple ways for my users to already get at their files (syncthing, ssh, etc) but I'd like users to have a really powerful web interface (copyparty) that is just a lot more convenient for a bunch of my users. I don't want it to be the case that I have to manually update the config for each user. I'd like them to just be able to oauth in and go.

Document reverse proxy with Caddy

Work in progress. Works fine for web browser with basic auth (I'm not yet comfortable running this without basic auth in place).

Problems with basic auth and android app 9001/party-up#3

docker-compose.yml

#
#       docker-compose up
#

version: '3.3'

# based on:
#       * https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-20-04
#       * https://arjunphp.com/windows-server-docker-compose-caddy-server-reverse-proxy/
#       * https://github.com/lucaslorentz/caddy-docker-proxy - not actually used

services:
  # reverse proxy server
  # https://hub.docker.com/_/caddy
  caddy:
    image: caddy
    restart: unless-stopped
    hostname: caddy
    networks:
      - app_net
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/data:/data
    # NOTE only Caddy should be using ports, all others shuld use expose instead
    ports: 
      # NOTE 80 and 443 were needed to get cert. once had cert not needed - looks like http (80) was needed unclear about https (443)
      - "80:80"
      #- "443:443"
      - "2000:2000"


  #    mkdir -p /home/pi/data/copyparty/cfg /home/pi/data/copyparty/w
  #    chmod a+rwx /home/pi/data/copyparty/cfg /home/pi/data/copyparty/w
  copyparty:
    # TODO? -u 1000
    container_name: copyparty
    image: copyparty/min
    hostname: copyparty
    restart: unless-stopped
    expose: 
      - 3923
    networks:
      - app_net
    volumes:
      - "/home/pi/data/copyparty/cfg:/cfg"
      - "/home/pi/data/copyparty/w:/w"


networks: 
  app_net:
    external: false

Caddyfile

https://copyparty.YOUR.DOMAIN.HERE:2000 {
  # basicauth works fine for web browsers with CopyParty
  # it fails with Party-Up Android sharing app :-(
  # not figured out passwords scheme properly yet to want to remove this
  basicauth /* {
        username hashed_password
  }
  reverse_proxy copyparty:3923
}

/home/pi/data/copyparty/cfg/config.conf

Disable https, so that reverse proxy certificate is used

[global]
http-only

cannot specify username at login screen

Hi again.. thanks for the quick fixes for the other issues I raised :)

If I launch with:
copyparty -a user1:pass -a user2:pass
I was expecting it to allow 2 users, but entering "user1" or "user2" on the login screen logs "invalid password: user1".
If I enter only "pass", it logs me in as user2. It seems the "act" input is hidden by default. (as a suggestion, accepting only a password is a bit less secure, but seems to be design intent. not that I have more than one user :)
I haven't tried setting up a config file yet, since I'm just experimenting, but perhaps that un-hides the "act" field on the login page?

Show hidden files button does not show hidden files

Made a new folder called .dotfolder using the copyparty make a directory button, it made it fine and moved me into it.
I also made a .dotmd.md file.

Going back a page and refreshing hides it, as expected. But toggling the dotfiles setting on/off doesn't reveal any of the hidden files/folders.

The button does say "show hidden files (if server permits)" and I guess server does not permit - but I couldn't really find anything abt that in the readme. Do you happen to know what I need to change to allow it to show hidden files? Is there a volflag or something?

The permissions look like this, everyone should have at least read permissions.

gremious@hostname ~/snip/public> eza -la
drwxr-xr-x   - gremious  9 Dec 19:27 .dotfolder
.rw-r--r--  10 gremious  9 Dec 19:30 .dotmd
drwxr-xr-x   - gremious  3 Dec 20:39 .hist
drwxr-xr-x   - gremious  8 Dec 12:55 Misc
.rw-r--r-- 214 gremious  8 Jul 15:51 README.md
.rw-r--r-- 193 gremious 12 Jul 20:25 test.html

Hitting 404 rate limit when transfering large amount of files with WebDAV using GVFS

Describe the bug
When transfering a large amount of files with WebDAV (and possibly because how GVFS handles that) it's really easy to hit the 404 rate limit and have its IP get banned for 24h.

To Reproduce
Transfer a large amount of files, and you should see a log similar to the following in copyparty:

PFIND {path to file 1}
Not Found {path to file 1}
PFIND {path to file 1}
Not Found {path to file 1}
PUT {path to file 1}
<size of file 1>

PFIND {path to file 2}
Not Found {path to file 2}
PFIND {path to file 2}
Not Found {path to file 2}
PUT {path to file 2}
<size of file 2>

(repeating for each subsequent file)

You can see that the client here tries to check if the file it's about to transfer does not already exist (which is a safe way to do, imho), but it means that each PFIND that queries a nonexistent file will get a 404, and copyparty will record that as a brute force attempt.

Expected behavior
There's no perfect way to handle that, but maybe one workaround would be to "cancel" the recorded 404 generated by the PFIND for the client if it transfers a file at the same location immediately after.
This way it should be possible to transfer a huge amount of files without risking to hit the rate limit for wrong reasons.

Open videos with Streamlink

Hello,
Its is posible to open a link sent with Party Up! with Streamlink insted of the default browser?
Can I change this behaviour?

Expl: Streamlink is command-line interface which pipes streams from various services into a video player. Avoid resource-heavy and unoptimized websites, and still enjoy streamed content.

Thanks

Cutting and pasting a symlink loses it (does not relink?)

Describe the bug
If you cut and paste an existing link copy of a file into a different folder, it disappears in copyparty.

To Reproduce
I have the following config:

[global]
theme: 2
ftp: 3921
ftps: 3990
ftp-pr: 12000-13000
# Use hashed passwords
ah-alg: argon2
# Thumbnail view on by default
grid
# Enable general file indexing
# Maybe it's called "e2d..." because it uses an 'up2k' tree for the database...
e2dsa
# Enable metadata and tag indexing (ffprobe)
e2ts
# Full-sized image thumbnails
th-no-crop
# Don't crawl my website please google
force-js
no-robots
# Don't create symlinks to deduplicate, make copies of every file
# no-dedup
# Allow cross-volume symlinks for deduplication
xlink

[/protected]
/snip/protected
accs:
  rwmda: gremious
  G: *
flags:
  fk: 8
  fka,
  1. make folder /protected/one
  2. make folder /protected/two
  3. make folder /protected/two/three
  4. Upload a new image you don't have to protected/one
  5. Upload the same image to protected/two
    (so far everything is OK)
  6. Select said image and cut, then paste it into /protected/two/three
  7. Refresh

Notice image is not there in ether /protected/two/three or /protected/two

copyparty logs say:

Dec 08 13:14:46 hostname copyparty[1575637]: 13:14:46.399 scandir
[s] FileNotFoundError(2, 'No such file or directory') @ /snip/protected/two/three/noise.png

The link is still there on the filesystem (eza is just a fancy ls):

gremious@hostname ~> eza -l snip/protected/two/
drwxr-xr-x - gremious  8 Dec 13:11 three
gremious@hostname ~> eza -l snip/protected/two/three/
lrwxrwxrwx - gremious  8 Dec 13:10 noise.png -> ../one/noise.png

However, I think that should be ../../one/noise.png now and not ../one/noise.png?

Server details

  • server OS / version: Debian bullseye
  • python version: Python 3.9.2
  • filesystem (lsblk -f on linux): btrfs

statistics

a suggestion came up on discord to add statistics; making an issue for ideas / brainstorming

help-wanted: mostly for ideas (see labels below), but just in case someone feels adventurous enough to tackle the implementation as well, I was roughly planning to do it based on u2idx, doing direct sqlite3 queries

design

  • how should this work? There will be an html page with basic display of the metrics, but should probably also be made available over some api. I have briefly heard of things like nagios and grafana, but I don't know what kind of api these expect. What softwares can I use to test this / integrate with? Would appreciate suggestions! help-wanted
  • should this require user authentication / some specific access-level, or is it fine if anyone can read it? Some of the metrics would be expensive to compute, so requiring auth might be in order

metrics

List of metrics that are easy to add,

  • number of uploads and/or unique uploaders, in each volume or in total, during the last N hours/days/... Or within a given timespan (date-date)
  • number of files, total size, in each volume / in total
  • number of deduplicated files / amount of space saved due to dedup
  • number of currently banned IPs
  • Probably more... help-wanted

unlikely metrics

List of metrics that are unlikely to be added

  • visitor / downloads counter -- would likely have a performance impact even if the feature is not in use, and also non-trivial -- could probably be done by parsing nginx logs instead

Cannot connect to ftps server: type object 'FtpHandler' has no attribute 'hub'.

Heya, I cannot connect to the s(?)ftp server, with the following error in the server:

16:20:08.170 pyftpdlib             Traceback (most recent call last):
  File "/tmp/pe-copyparty.1000.5404.0/ftp/pyftpdlib/servers.py", line 175, in handle_accepted
    handler = self.handler(sock, self, ioloop=self.ioloop)
  File "/tmp/pe-copyparty.1000.5404.0/copyparty/ftpd.py", line 390, in __init__
    self.hub  = FtpHandler.hub
AttributeError: type object 'FtpHandler' has no attribute 'hub'

I have an android phone, trying to add the FTP server through the default file browser:

image

That server message pops up every time I try to press "Add". Phone doesn't complain about being unable to reach the server, like it would if I had put in a wrong port. So I believe it's something to do with copyparty/the ftp library.

My config contains the following:

ftps: 3990
ftp-pr: 12000-13000

I have opened all of those ports as well.

I'm using Python 3.9.2 on Debian 11.

Dude you’re awesome

This has to be the most over engineered project I’ve ever seen and it’s glorious.
Thanks for making it free and kudos to you for creating some great software.
I use it to host an archive of some rare music I like πŸ˜ƒ

Option to disable media player and default to download

I have a use case where the client app (DeoVR) needs to have the video file (I.E mp4) streamed directly in order to function. Is there a way to disable the media player globally or per-folder? I would still want to retain thumbnail function in gridview.

Question: What happens to duplicate files in different subfolders/with different names? Are they symlinked or duplicated?

The docs mention e.g. --hardlink: creates hardlinks instead of symlinks and that upload does symlink/discard duplicates (content-matching) but...when are symlinks created? (if I just missed a section of the docs please do say)

Please consider this example:

I have both e2dsa and e2ts in my global config.
I have uploaded a pictures/ folder with just all of my photos in it, to the same-named copyparty volume.
Then, on the OS, I make a new folder, pictures/my-may-album in which I copy photos from the pictures/ folder into, to make a little album.
Then I just drag-and-drop upload to copyparty's pictures/, so now it has pictures/my-may-album/ with copies of existing files.

Does copyparty deduplucate those? Are they symlinks now, can I delete ether end safely? Or does it only de-duplicate if they have the same name, directory AND contents?

The filekeys for both of those stay different, and doing a file search finds both copies as "0 diff", so it's hard to tell, and the mention of symlinks in regards to de-duplication got me a bit confused.

"database is locked"

suddenly unable to start db transactions (read or write)

has the following consequences if it happens (just seen once so far):

  • uploads don't get indexed
    • so upload deduplication is busted
    • but a restart with -e2ts fixes everything back up
  • search is disabled
  • otherwise harmless, just super annoying

uploads got stuck in _finish_upload at self.idx_wark giving clients a red toast saying database is locked before it automatically retried the handshake and successfully finished the upload, bypassing the indexing tho

searches and reindexing also failed similarly, guessing a thread died while holding a transaction hostage (is that possible?)

unfortunately grabbed the /?stack of another copyparty instance running on the same server so don't have any context

in case someone runs into this:

pls check server logs for a huge wall of base64 and post it -- that's a stacktrace of all threads which will probably help track this down -- alternatively visit /?stack on your server to take a look yourself

also include the lsof if there is one (only works on linux/macos)

original bug report (thanks!):

Screenshot_2022-07-27_17-07-47-fs8

nixpkg: autogenerated secrets

just a hunch that the recently added ~/.config/copyparty/fk-salt.txt is a bad fit for nixos

should figure this out soon -- definitely before starting any work on hashed passwords (which would be one or two more of those)

Considerations for dirkeys (like filekeys but for directories...)

Very simple thought, though it might be too much of a pain to implement idk:

I can share https:/myserver.com/photos/friends-only-album/image.jpg?k=1234, where https:/myserver.com/photos/friends-only-album/ goes to a 403 because there's only G permissions set for *.

it would be cool to be able to do https:/myserver.com/photos/friends-only-album/?k=1234 and share a whole folder

Then if a random person goes to https:/myserver.com/photos/friends-only-album/ it will 403, but I can selectively share it with trusted people.

Thoughts:

  1. I imagine if you share a folder, people should be able to open and view the files there - which will probably reveal the filekeys and let said people share those. This is probably fine cause if you're sharing something to anyone, you're supposed to trust them anyway.
  2. It will probably do the same for subdirectories, so ig be careful to not put sensitive file nested deep on some global shareable directory.
  3. If a new file is uploaded there, I guess the key should regenerate? But then that's a pain if it's a folder that is expected to have updates, so probably optional.

Consideratrion for not using filesize when generating filekeys (or, sorta-more-permanent-links)

Currently, from what I can tell, filekeys are generated using salt, fspath, fsize, and inode. They are also regenerated on file edit (?).

What this means is that if a text file is edited, the filekey is regenerated, and any links to it break permanently.

Now, this may be intentional as a "if I edit this file/delete it and upload a new unrelated one with the same name, link breaks", but it would be cool if links to files remained 'more-static' until the file is deleted/moved.

Use cases:

  1. I send notes to people, kinda like a pastebin, then edit the note cause I e.g. missed a word and now have to re-send because the link they already opened is now broken. Wish I could just say "oh ops refresh I updated it"
  2. I cannot have a nice <a href="path/to/shopping.md">πŸ›’ Shopping List </a> quicklink in my README.md
    because the file key changes every time I edit the file...

Perhaps instead of filesize, file creation timestamp or something can be used? (to prevent irrelevant re-uploads from having the same key). Or perhaps always generate a new filekey with size like normal, just don't regen if existing file is edited (idk how the code works to know if this is feasible).

[Help] Does copyparty support suburls?

I want to use copyparty as a simple file sharing service on my own server. And because I don't use domains (using NoIP currently :P) I wanted to ask if copyparty supports suburls? I've implemented a ApacheΒ² reverse proxy setup and it does not work correctly (as seen in the screenshot below). Can I somehow fix it (by implementing it manually or changing something in the args/config)?

A screenshot displaying a broken copyparty installation with DevTools network monitoring enabled:
broken copyparty screenshot

TODO when rclone v1.63 is out

update the rclone commands on the connect page to specify the new pacer-min-sleep param

and adjust the speed estimates in the readmes too cause this is gonna be fast

copyparty (help) output includes VT100 color escapes when redirected to a file

Describe the bug

Most tools that emit color only do so when dealing with a console/tty. Workaround is to supply --no-ansi flag.

To Reproduce

python  dist/copyparty-sfx.py --help > help.txt

help.txt file will have binary content.

Expected behavior

Readable content.

Screenshots
NA

Server details

NA

Client details

NA

Additional context

NA

Add cli/config argument to enable grid view by default

"I'm always frustrated when" I send my media folder to my friends and have to tell them they can enable grid view to actually see the images. Also, when connecting to a new device, I personally want grid mode on by default always anyway.

I scoured the help page, but I didn't find a setting for it. Did I miss it? If not, please consider adding a --grid or some such, thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.