tus / tusd Goto Github PK
View Code? Open in Web Editor NEWReference server implementation in Go of tus: the open protocol for resumable file uploads
Home Page: https://tus.github.io/tusd
License: MIT License
Reference server implementation in Go of tus: the open protocol for resumable file uploads
Home Page: https://tus.github.io/tusd
License: MIT License
my local upload demo is broken (upload gets stuck / is not resumable), need to figure it out / fix it.
Hi All,
I'm sorry if this is a noob question, but I can't find the default maximum file upload size for Tusd. I am not versed in Go so am unable to understand the flow of the binary.
Can someone help? Thanks!
Hello,
Few days ago I started port tus java client to C#. When I finished I tested it on mobile devices. I have tested 2 scenarios:
In S1 Golang server close connection correctly, copy already send data to the file and set correct offset for future requests.
In S2 Golang server is waiting for resuming connection (it looks like). As far as I know it has default infinity timeout and is waiting for resuming connection too long. In that case If I want to re-upload the file I must start from the scratch (HEAD requests gives me 0 offset).
I don't know Golang so I'm asking you: Is there any way you can handle this scenario?
We need a way to locally store files, which may be coming in as partial chunks. These chunks may overlap, which must cause no problems if the overlapping parts hold the same data.
My idea: For the first request create a file id, then create tus_data/<fileId>
and tus_data/<fileId>.log
. The first file has the full size of the expected is empty, and gets filled as data comes in. The second file is an append-only log that lists all chunks of the file that have been received. The format for this could be new-line separated JSON.
Otherwise we may fill up the disk on our demo server too quickly.
Basic plan: Trigger GC() after each upload, if total file size is too big, delete old files until enough space is available.
Would be awesome to get an officially maintained s3 storage plug using tusd.DataStore
.
Hi,
I want to use resumable upload functionality with PHP server to make it work over server, there are few things need like below -
I have configured tusd client but when installing/configuring tusd server in my local machine, it is not working as it is giving error when I'm trying to run main.go using below command -
go run cmd/tusd/main.go
Please try to help me out to solve the issue.
Thanks!
Need to figure out where to put it. Potentially as the first entry into the log file?
During my test, I have earned how to add key-value pairs into MetaData field contained into .info files.
But I'm not sure to have well understood the meaning of the field "IsFinal": I have read the comments into filestore.go, but I have seen these field always set to false, even if the upload is correctly finished. On the other hand the field "IsPartial" is always false even if I stop (restart and stop again) the upload before it is completed.
What I'm misunderstanding?
Who is responsible (which "go" class) to write the content of the .info files?
Again, thanks a lot in advance.
The protocol shows HEAD calls as returning 200 codes, which is also how the TUSKit library is implemented and what the HTTP spec defines.
https://github.com/tus/tus-resumable-upload-protocol/blob/master/protocol.md#example
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Hi,
Looking into the code and the go docs I couldn't figure out how to enable checksum extension.
The only place that mention this extension is the specification.
Is this supported by tusd? How can I enable ou start implementing this?
This depends on #9, once we have things refactored, we could easily spin up a test server for integration tests.
At Transloadit, we're currently generating the following Upstart file to launch tusd
:
$ cat /etc/init/tusd.conf
stop on runlevel [016]
respawn
respawn limit 10 5
limit nofile 32768 32768
pre-stop exec status "tusd" | grep -q "stop/waiting" && initctl emit --no-wait stopped JOB="tusd" || true
script
set -e
rm -f "/srv/shared/tusd-log-fifo"
mkfifo "/srv/shared/tusd-log-fifo"
( logger --tag "tusd" <"/srv/shared/tusd-log-fifo" & )
exec >"/srv/shared/tusd-log-fifo"
rm -f "/srv/shared/tusd-log-fifo"
exec bash -c "cd /srv/current \
&& source env.sh \
&& exec sudo -HEu transloadit-api2 PATH=\${PATH} LD_LIBRARY_PATH=\${LD_LIBRARY_PATH} \
/srv/current/stack/bin/tusd \
-host 127.0.0.1 \
-port 1080 \
-base-path /resumable/ \
-hooks-dir /srv/current/api2/bin/tusd-hooks \
-max-size 42949672960 \
-s3-bucket tmp.transloadit.com \
2>&1"
end script
Then our rsyslog is configured like so:
$ cat /etc/rsyslog.d/49-tusd.conf
if $programname == 'tusd' then /srv/shared/log/tusd.log
if $programname == 'tusd' then @logs.papertrailapp.com:34719
if $programname == 'tusd' then stop # no other actions to avoid duplicates
This handles things like logging (both to local file via syslog as well as centralized to papertrail), running tusd as a non-privileged user, injecting environments with a privileged user, and respawns.
Now this config may not work for everyone. I would also not be surprised if people in the tus community know better ways. And so the thought arises, maybe it's useful to offer a few community curated startup templates, initially for both Upstart and systemd (but that could later be expanded to support other platforms), so that the binary is easier deployed in production environments, and users run tus in a safer, and pretty much all in an identical way.
This could be provided by the tus community either in the form of:
./templates/
directory or similartusd os_startup
or similar, and it would ask some questions, and output the appropriate startup file, that the user could | sudo tee /etc/to/the/right/place
(avoiding tusd writing itself and having to ever run it as root)Thoughts?
Hey guys,
So I am trying to use TUS with S3 in my own project, since there were no examples, I looked into comments/code and tried came up with this piece of code:
cfg := &aws.Config{
Credentials: credentials.NewStaticCredentials(
viper.GetString("s3_access_key_id"),
viper.GetString("s3_secret_access_key"),
""),
}
s3Client := s3.New(session.New(), cfg)
store := s3store.New(viper.GetString("s3_bucket"), s3Client)
store.UseIn(composer)
When compiling this piece of code I get compile errors
cannot use s3Client (type *"github.com/aws/aws-sdk-go/service/s3".S3) as type s3iface.S3API in argument to s3store.New:
*"github.com/aws/aws-sdk-go/service/s3".S3 does not implement s3iface.S3API (wrong type for AbortMultipartUpload method)
have AbortMultipartUpload(*"github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadInput) (*"github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadOutput, error)
want AbortMultipartUpload(*"github.com/tus/tusd/vendor/github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadInput) (*"github.com/tus/tusd/vendor/github.com/aws/aws-sdk-go/service/s3".AbortMultipartUploadOutput, error)
What am I doing wrong here? Is it a bug? How can I fix this?
How can I return a custom http error code from a custom datastore ?
For instance, if the file extension is not allowed, I would like to send back a http/403 with a reason file extension not allowed
.
From the source, if my datastore returns an errors.New("403: File extension not allowed")
it will be returned as a 500
error.
Could the unrouted_handler provide an error structure to customize the status code ?
Like return tusd.NewHttpError(403, "File extension not allowed")
(if the error
implements the HttpError
interface, the status code is custom, otherwise it is set to 500).
The unrouted handler could also benefit from this (to remove the ErrStatusCodes
map):
ErrUnsupportedVersion = NewHttpError(http.StatusPreconditionFailed, "unsupported version")
ErrMaxSizeExceeded = NewHttpError(http.StatusRequestEntityTooLarge, "maximum size exceeded")
...
Hi @Acconut, as mentioned I made a version of tus client for go lang.
It was tested with 1.6, 1.7 and 1.8 (tip).
The code can be found in github.com/eventials/go-tus.
Feel free to check out, comment, or even send a PR. The lib was based a little in the JS version, but I want to add more features soon.
We need all of them : )
Hi!
I'm experimenting with the tusd
implementation as the basis for a production-grade TUS server. One of the things I've noticed in our tests is that at times the tusd.Handler
can get files to stay in locked state, with no way of being unlocked. We don't think we need this locking behavior in the handler, and can delegate this to the file store (we're using a DB as our back-end, which handles concurrent read/writes).
I have a (forthcoming) pull request which refactors the locking behavior to the filestore.Filestore
type.
Best,
Adam
For now we should probably make the most basic things configurable as environment variables:
Hello,
As the README says:
"In the future tusd may be extended with additional functionality to make it suitable as a standalone production upload server, but for now this is not a priority."
This issue deals with the standalone version of tusd.
Hi all,
I'm trying to upload a file that have size of 4.6 GB via tus. But after file is about 4.0 GB the PATCH method return an error message
read tcp 127.0.0.1:1080->127.0.0.1:61596: i/o timeout
Any idea?
In the lines above the path is concatenated with /
.
https://github.com/tus/tusd/blob/master/filestore/filestore.go#L187
https://github.com/tus/tusd/blob/master/filestore/filestore.go#L192
I think this will fail on Windows. And if I set the Path with slash it will add two slash. Most Linux don't break because of this, but the correct way is to use path.Join
function to concat paths.
If there's no specific reason for this I can send a PR to fix it.
Hello,
I would like to use tusd on Https protocol. I didn't success to used it. Is there possible to use this protocol. If it's possible, how do it this ?
Thank you for helping.
The current code could need a little more structure / types, which would also help with testing.
I was using the newest gb tool to fetch tusd, but it failed. Tusd is declared in the depfile. It turns out that tusd repo tags is not complied with the SemVer spec, so the gb tool failed. Can you guys tag the
tusd with a SemVer version?
It seams that tusd does not support "Range" headers for resuming interrupted downloads, or using a download manager to faster download the files.
It would be very nice to have this feature ;-)
This is about programs --some of them with the protocol explained-- that got great speed to upload files:
http://www.tcnj.edu/~bush/uftp.html
http://udt.sourceforge.net/
http://tsunami-udp.sourceforge.net/
And this information is to tune TCP to get high performance data transfers:
http://www.psc.edu/index.php/networking/641-tcp-tune
I hope it can be useful at some point.
Another issue (closed) says, regarding uploaded files being renamed to *.bin and *.info: "That's a good use case but a very specialized one. In general, I think, people will need more information than just the original file name and therefore have to read the additional data anyway. So I am sorry, but your suggestion won't make it into tusd."
I believe this is a major shortcoming for a "file uploader" - the uploaded file is essentially not there at all; this is a huge use case, not specialized in the least.
Is there a procedure for transferring "x.jpg" for example, to the server as "x.jpg"? Or, perhaps, a config hook to rename the file upon upload completion?
I know this may be a bit off-topic, but i have researched a lot and no clue.
The case is that I am working on a simple web application that allows people to upload files and edit them.
However I found that the uploaded files are owned as root and with permission 644, which cannot be edited by apache.
I have no idea how to allow apache edit the file that is not owned by it. Is it possible to change the owner or permission once tusd received the file?
many thanks!
Hi,
I find strange that store.UseIn(composer)
only acts on the composer (and not on the store).
It might be possible to create a UseAllExtensions
method on the composer
using some tests to know what the store
implements (see https://stackoverflow.com/a/20315155/3207406)
For instance:
func (composer *tusd.StoreComposer) UseAllExtensions(store interface{}) {
var ok bool
if _, ok = store.(DataStore);ok {
composer.UseCore(store)
}
if _, ok = store.(LockerDataStore);ok {
composer.UseLocker(store)
}
// ...
}
Now UnroutedHandler.absFileURL function uses r.Host
and scheme of server where the tusd handler is embedded. It would be great if the handler automatically detects proxied request using Forwarded
/X-Forwarded-Host
headers. I have to manually set r.Host
before calling tusd handler to workaround the cross-origin error.
Hi!
I have just succeded in tusd (go application) installation on an ubuntu machine. My idea is to to test "resumable file upload funcionality" with an Android client (using your sample Android application for now).
I have succesfully uploaded some files to my server througth Android application (pausing and resuming it).
It works like a charme! :)
Then I have stopped the tusd application and I have changed the upload directory from "data" to "uploadDirectory" in main.go.
I have started the server again and I have succesfully uploaded some OTHER files.
But when I try to upload one file that I have already uploaded in the previous tusd session (with upload dir="data") I have receiveed an "unexpected status code (404) while resuming upload".
I'm completly newby on go so forgive me in advance if I have miss some important details (and for my bad English).
Where is my mistake?
Is this the rigth place to post this questions?
Thanks in advance.
Currently, /files
is configurable.
If you where to place tusd behind a proxy, and route all URLs prefixed /resumable/
to your tusd server (and you're not stripping this prefix, because tusd needs to know and return the full path to its files), you might end with:
tusd \
-base-path /resumable/files/
However, since the metrics path is hardcoded under /metrics
this will now have become inaccessible, unless you are hacking around this with conditional stripping of prefixes.
Long story short we could solve this by making the metrics endpoint path configurable. And in our case, mount it to /resumable/metrics and then redirect all /resumable/* traffic to tusd without stripping or rewriting anything conditionally:
tusd \
-base-path /resumable/files/ \
-metrics-path /resumable/metrics
Or perhaps, if we're going down this path anyway, maybe we can make it apparent we are not talking about local paths, and also be more specific than base
:
tusd \
-files-endpoint /resumable/files/ \
-metrics-endpoint /resumable/metrics
Great project, we are just starting to use it. Hope uploading implementations are a thing of the past now 👍
The uploaded files are created with the executable bit true. For everyone! -rwxr-xr-x
I would prefer them not executable.
For me this could be one part of an attack strategy with some trojan. It's clear that there has to be a second exploit at least to complete the big picture. But early intervention is king here.
I don't know exactly if this could be changed by the os configuration or is caused by it (Ubuntu 16.04, pretty default install). Other uploads and tmp outside of tusd are not executable.
But maybe this should be cared about by tusd in every case to prevent people running into it.
urlPrefix is more specific / less generic imho
Hello,
I'm currently working on MongoDB and I need to upload some large files into DB. Is there a plan to support GridFS storage?
After successful upload, I need to move the uploaded file(.bin) to another directory and delete the .info file which is not need after that. Is there any call back function I can use to perform this operation.
And please let me know whether the following is feasible:
I have 'n' number of clients uploading files; I want to store the files in separate directory with respect to the client uploading it. This may be a feature of file server; but I foresee tus as a upload file server rather than just a uploading protocol.
When using the concatenation extension, partial uploads should not have an offset, which also implies that they should be PATCH
ed from offset 0 regardless of how many bytes have been received so far.:
https://github.com/tus/tus-resumable-upload-protocol/blob/master/protocol.md#concatenation
I can't run the server, nor demo files. I keep getting Error: Cannot find module 'google-cloud'
from node.
Is there any workaround? Am I doing something wrong?
a separate repository might be good for this:
README.md
we can use for docs for nowWhen you GET
on the root or the basePath
, it would be nice to display:
Welcome to tusd version x.y.z. For documentation and bug reports, please check https://github.com/tus/tusd
Instead of
Just more friendly and to let admins know they did something right, vs wrong.
Right now GET doesn't work until the full file is received. It'd be awesome if it would stream the amount of data that has been received so far, so that data can be streamed to the downloader as it's being uploaded!
I´m testing tus-js-client with tusd.
The tusd is running in Ubuntu Desktop 16.04 in Virtualbox and tus-js-cliente demo in Windows host with Firefox. I put the tusd in a AWS EC2 server and the same error occour.
The tusd is configured with parameter -s3-bucket,
All works fine, but an error always happen in upload process in big files (~ 300Mb). When click in "Ok" the process continues but after some minutes the error occurs again.
"Failed because: Error: tus: failed to upload chunk at offset 62914560
Do you want to retry?"
After many click´s in "Ok" button, the upload is completed and file is correct in AWS S3
What might is happening? How might I discover wher is the problem?
this way we can tell what files were uploaded, how fast, etc.
Hi all,
I tried to run the official binary distribution of Go language edition of tud with official Android sample code (client). I found a file name problem (or may be an enhancement / feature request) of current sample codes of server side and client side.
The uploaded file's file name on the server is convert to a random hashed file name (e.g. 5dcecfd73c6d77bf7dfcf4fbdd75275e.bin with 5dcecfd73c6d77bf7dfcf4fbdd75275e.info )
It's a good design to avoid the duplicate of file name of uploaded files.
But the server side do not know the original file name of uploaded file either in .info file. (.info file do not have the information of original file name)
The client side is also do not know the new file name of uploaded file.
So it is difficult to make the relationship of uploaded file name with original file name for further application (e.g. client side can call the server API to tell the server the uploaded file name for server to fetch the file for further processing after client side uploaded the file)
The file extensions (e.g. .png, .zip) are also changed to .bin. The server side is also difficult to know the original file extension name to do further use.
In conclusion, my suggestions are as below:
Right now it gives a 500 error with the explanation EOF which is confusing
i am a newbie to tus (and github), hope to get a right place to ask.
i am trying to start a tusd on an aws ec2 instance and client on my local computer.i have tried to use the binary and everything all right.
However, i would like to config the datastore path, so i used the sample code here and the below error occur,
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
checked i have to allow the access of the ec2 from other origin, but i am not sure how to achieve. thanks for any reply.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.