flowjs / flow-php-server Goto Github PK
View Code? Open in Web Editor NEWflow.js php server library, validates uploaded chunks and safely merges all chunks to a single file
Home Page: http://flowjs.github.io/ng-flow/
License: MIT License
flow.js php server library, validates uploaded chunks and safely merges all chunks to a single file
Home Page: http://flowjs.github.io/ng-flow/
License: MIT License
On every file upload I get an error 400. Tried to put some error_log() into the code, but nothing showed up. The error message comes from this:
if ($file->validateChunk()) {
$file->saveChunk();
} else {
// error, invalid chunk upload request, retry
header("HTTP/1.1 400 Bad Request");
return ;
}
But I don't know why there is an error...
(Using the example-code from the node-js sample (index.html))
I started working on version 1.0.1 with the help of my guide and I was met with this error: Class 'Flow\Config' not found
but In version 1.0.0, there was no such problem
my code:
require_once(APPPATH.'third_party'."/Flow/autoloader.php");
$config = new \Flow\Config();
.
.
.
I followed the docs closely. Uploading big files with multiple chunks keeps giving me UPLOAD_ERR_PARTIAL error. How do I fix this?
I have the flowjs client on localhost and then hosted the ng-flow server on cloud. Sending the file to the target of the server... Keep getting this error
sha1(): Passing null to parameter #1 ($string) of type string is deprecated"
But when I placed the server and client on same host, it works... but on different host it doesnt work and have added headers such as
header('Content-Type: application/json; charset=UTF-8');
header('Access-Control-Allow-Origin: *');
header('Access-Control-Max-Age: 3600');
For CORS but yet its not working.
On client i have
var flow = new Flow({
target:'https://my-server-link/upload',
query:{upload_token:'my_token'},
chunkSize: 1024*1024,
});
Please what is the workable method to use this?
This is a nasty code but It work for me for an AngularJs and Laravel 5 integration. I am just happy and want to share it for sme. It is a copy from one post here but modified.
public function getSubirArchivo($declaracion) {
error_log('Documento 22');
$request = new \Flow\Request();
$config = new \Flow\Config(array(
'tempDir' => './chunks_temp', //With write access
));
$file = new \Flow\File($config, $request);
$response = Response::make('', 200);
$destination = './archivos/' . $declaracion . '_' . $request->getFileName();
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
error_log('Documento GET');
if (!$file->checkChunk()) {
error_log('Documento GET2');
return Response::make('', 204);
}
} else {
if ($file->validateChunk()) {
$file->saveChunk();
error_log('Documento VaLido !!!');
} else {
error_log('Documento INVALIDO');
// error, invalid chunk upload request, retry
return Response::make('Error in chunck', 400);
}
}
if ($file->validateFile() && $file->save($destination)) {
error_log('EXITO!');
$response = Response::make('pass some success message to flow.js', 200);
}
return $response;
}
I'm currently working on a project which uses flow.js and flow-php-server.
9 times out of 10, when uploading the same 5mb PDF it transfers fine. However, sometimes it will arrive corrupted. A successful transfer would have a mime-type of "application/pdf", the corrupt transfers have a mime-type of "application/octet-stream".
I'm not sure if the root of the corruption is on the client or server side. Let me know what info you need and I'll pull together what I can.
When I upload a large file (e.g. 1.4 GB), I get a timeout (http 504) on one of the (post) requests. The file is completely uploaded and merged. However, the client does not know about it. On the client side, the transfer information ends with: progress:1, error:true, success: false, complete:true. The request is waiting for the response from the server. Maybe I can solve the problem by increasing some timeouts on the server side. However, I have already tried a lot without success. Would it be possible to flush something like a status from time to time on php side so that I don't run into the timeout?
Thanks for any advice.
Need thoughts on this:
$request = new \Flow\Request();
$config = new \Flow\Config();
$config->setTempDir('./');
$file = new \Flow\File($config, $request);
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
if ($file->checkChunk()) {
header("HTTP/1.1 200 Ok");
} else {
header("HTTP/1.1 204 No Content");
return ;
}
} else {
//print_r($file); // -> UNCOMMENTING THIS MAKES THIS CALL WORK...
if ($file->validateChunk()) {
$file->saveChunk();
} else {
// error, invalid chunk upload request, retry
header("HTTP/1.1 400 Bad Request");
return ;
}
}
if ($file->validateFile() && $file->save($request->getFileName())) {
// File upload was completed
} else {
// This is not a final chunk, continue to upload
}
I know the title is bit vague so let me explain.
During my tests I wanted to simulate big file uploads with relatively small files so on the front end I set:
simultaneousUploads: 3
(which is I believe default setting)
`chunkSize: 2048 (which is 1024*1024 by default)
This gave me kind of simulation of multiple chunk uploads. Using the library on the back end you have to follow four steps:
If you have multiple concurrent uploads this may lead to corrupted files. Lets take a case of last two chunks being uploaded where x axis is time:
chunk1: |---validateChunk---||---saveChunk---||---validateFile---||---save---|
chunk2: |---validateChunk---||---saveChunk---||---validateFile---|
when validateFile
is called on chunk1 the saveChunk
for chunk2 already begun so chunk1::validateFile will return true and proceed to save file with chunk2 not fully saved.
I saw this error in my logs already few times. The above example will also lead to double save
call. And in my case it leads to duplicate key error in the database.
To fix the problem library would have to implement locking not only in save
but in validateFile
methods . Its not enough to call file_exists
in validateFile
. We have to know the chunk is fully uploaded.
I'm working on new version of on-line photo printing software. So I tested your package. In this case often happens that users send the same file.
I enabled duplicate uploads via JS:
var flow = new Flow({
target: 'upload.php',
allowDuplicateUploads: true,
});
When you select same file multiple times until first upload is finished only one file is saved.
To easy reproduce use this code (upload.php):
<?php
require_once './vendor/autoload.php';
$config = new \Flow\Config();
$config->setTempDir('./tmp');
$request = new \Flow\Request();
$uploadFolder = './upload/'; // Folder where the file will be stored
$uploadFileName = uniqid()."_".$request->getFileName(); /
$uploadPath = $uploadFolder.$uploadFileName;
if (\Flow\Basic::save($uploadPath, $config, $request)) {
// file saved successfully and can be accessed at $uploadPath
} else {
// This is not a final chunk or request is invalid, continue to upload.
}
sleep(10);
And add sleep(5)
to /vendor/flowjs/flow-php-server/src/Flow/File.php
, line 192.
sleep(5);
if ($this->config->getDeleteChunksOnSave()) {
$this->deleteChunks();
}
Just before chunks are merged.
Explanation: Flow.js creates same identifier for same fille. Flow-PHP-Server uses this identifier for chunk names (after hasing) = same chunk names for different uploads. When first file is merged chunks for next one are deleted. So another file cannot be created.
Hi,
I am using this code and filename save it as upload.php as follows:
But after submitting the form through html and where to pass uploaded_file (i.e.,) $_FILES['uploaded_file']['tmp_name'] and $_FILES['uploaded_file']['name'] in backend how to use.
require_once 'vendor/autoload.php';
$config = new \Flow\Config();
$config->setTempDir(storage_path() . '/temp');
$destination = 'uploads/';
$file = new File($config);
$request = new \Flow\Request();
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
if ( ! $file->checkChunk() ) {
return \Response::make('',204);
}
} else {
if ($file->validateChunk()) {
$file->saveChunk();
} else {
return \Response::make('', 400);
}
}
if ($file->validateFile() && $file->save($destination . $request->getFileName())) {
return \Response::make('File upload was completed', 200);
}
It's probably just me, but uploading larger files (3GB+) is not really possible currently. Chunking files and uploading each junk to the server works just fine, even for very large files. But the larger the file the more chunks need to be put together by the server. This process seems to be very time consuming (in PHP) and if it takes too long the script just timeouts.
For some people it's possible to just increase the timeout limit, but even then I'm not sure if that's the best solution. So I'm wondering if we could just bypass this problem by uploading the chunks in the correct order and appending them to a "temp" file (after a chunk was successfully uploaded). In this case there shouldn't be a timeout problem at all, no matter how large the file is. At least with my very limited understanding of PHP.
If there aren't any drawbacks I'm wondering if you could imagine adding something like that?
I have two servers setup, A & B, with urls A & B pointing to the servers respectively. Both servers contain the same codebase.
If I go directly to url B on server B, and initiate a flowjs upload, everything works as expected.
If I go to url A (server A), but have the target url for the flowjs be url B (server B), instead of uploading the file, I just got three requests with an empty response but a 302 Found Header.
Any insight on getting this working cross-server without me digging into the codebase?
All SSL's are wildcard, and sessions are cross-domain to a redis server.
Hi,
Is there a small example available how to use the flow-php-server from scratch?
Thanks
Tom
Hello,
I'm trying to implement flow-php-server in Slim framework, but it is not working.
Here is my code:
$app->any('/upload', function(Request $req, Response $resp, $args){
try {
$response = [
'type' => 'error',
'message' =>'no data'
];
$request = new \Flow\Request();
$dest = 'files/' . $request->getParam('path');
$chunks_temp_folder = $request->getParam('chunks_temp_folder') ?: './chunks_temp_folder';
if (!is_dir($chunks_temp_folder) || !is_writable($chunks_temp_folder)) {
throw new Exception("Chunks directory {$chunks_temp_folder} does not exists or is not writeable");
}
$config = new \Flow\Config(['tempDir' => $chunks_temp_folder ]);
if (!is_dir($dest) || !is_writable($dest)) {
throw new Exception("Destination directory {$dest} does not exists or is not writeable");
}
$dest .= DIRECTORY_SEPARATOR . $request->getFileName();
if (\Flow\Basic::save($dest, $config, $request)) {
$file = $request->getFile();
$response = [
"type" => "success",
"name" => $file['name'],
"filetype" => $file['type'],
"error" => $file['error'],
"size" => $file['size'],
"readablesize" => self::human_filesize(filesize($dest), 2),
"fullpath" => $dest
];
} else {
throw new Exception("\Flow\Basic::save failed");
}
} catch (Exception $e) {
$response['type'] = 'error';
$response['message'] = $e->getMessage();
}
echo json_encode($response);
return;
});
The "\Flow\Basic::save failed"
exception is being thrown. Anyone has been successful in doing this?
the current implementation is based on the obsolete mongo lib.
this should be changed to mongodb/mongodb.
I'd be willing to provide a PR :)
The script currently overwrites if there is an identically named file in the final upload directory. It should be able to rename the file (numbered perhaps like filename-2.ext
) if it encounters this kinds of situation.
Hey there,
and thank you for the great work on this library.
This issue can be deleted. In case someone else is wondering, in Laravel 5 the standard Request::input('parametername')
does the trick.
Cheers
I'm using the basic sample and I always get a 404 error (probably because of chunk checking)
but no file is saved even using the basic code.
http://tinyurl.com/keu5gbk
No chunk files are created nor the file is uploaded.
The demo presented on this page works just fine:
https://github.com/flowjs/flow.js/blob/master/samples/Backend%20on%20PHP.md
Thanks
there are some commits after the 1.0.3 release, which I'd need. Would it be possible to cut a "fresh" release, please?
The PHP server only support the default multipart
option, and not the octet
one.
This should be added to the documentation right now,
and maybe added to the library as contributions go :)
$fileArray = [
'name' => $requestParams['flowFilename'],
'type' => '',
'tmp_name' => 'php://input',
'error' => UPLOAD_ERR_OK,
'size' => $requestParams['flowCurrentChunkSize'],
];
move_uploaded_file
but instead read 'php://input'
and file_put_contents
itIs there somewhere a config where I can prevent that metadata like portrait modus are not stripped from uploaded images? After uploading an iOS iPhone portrait modus photo, it tilts the images.
When I enable testChunks
optsion in my JS script, I get a 404 error but when I set it to false it works just fine, what am I doing wrong?
I can't find any way to get the mime type of the file from either the Flow\Request or the Flow\File object? If this functionality does not exist, I'd be glad to make a pull request for it, but I don't want to go reinventing the wheel and it seems like a pretty important thing to omit.
Seems that the lib keep the original file name and there is no way that the server can tell ng-flow the new file name, my server output this on complete:
{
"success":true,
"flowTotalSize":58110,
"flowIdentifier":"10152987551505362-2e68232139983ba857669baf3c2d45c7_development.jpg-58110",
"flowFilename":"4498b3d20302c1c3b75ce37c1e5fefac",
"flowRelativePath":"http:\/\/localhost\/images\/users\/4498b3d20302c1c3b75ce37c1e5fefac"
}
my upload script:
<?php
$path = __DIR__ . '/../public/images/users/';
$tmp = __DIR__ . '/../tmp/';
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
$chunkDir = $tmp . DIRECTORY_SEPARATOR . $_GET['flowIdentifier'];
$chunkFile = $chunkDir.'/chunk.part'.$_GET['flowChunkNumber'];
if (file_exists($chunkFile)) {
header("HTTP/1.0 200 Ok");
} else {
header("HTTP/1.0 404 Not Found");
}
}
// $file = $path . (isset($_FILES['file']) ? $_FILES['file']['name'] : $_GET['flowFilename']); // this is using original name, but in this case the file sooner or later will be overrite by other user that upload a file with the same name
$file = $path . md5(time()); // The way I want, it need to add the extension but it was a test only
if (isset($_FILES)) {
copy($_FILES['file']['tmp_name'], $file);
echo json_encode([
'success' => true,
//optional
'flowTotalSize' => $_FILES['file']['size'],
'flowIdentifier' => $_FILES['file']['name'] . '-' . $_FILES['file']['size'],
'flowFilename' => basename($file),
'flowRelativePath' => 'http://localhost/images/users/' . basename($file)
]);
} else {
if (\Flow\Basic::save($file, $tmp)) {
echo json_encode([
'success' => true,
//optional
'flowTotalSize' => $_GET['flowTotalSize'],
'flowIdentifier' => basename($file),
'flowFilename' => basename($file),
'flowRelativePath' => 'http://localhost/images/users/' . basename($file)
]);
}
}
I'm using PHP 5.4 and when using the Flow\Request object I get the error that the abstract class method 'wgetCurrentChunkNumber' must be declared. Which it isn't... I'd submit a fix, but I'm not even sure what the function would do...
Cheers
Ben
How are the file names / flowIdentifier sanitized before being used ? Maybe I'm wrong, but looking at the code, I can't see any checks being made that would prevent a directory traversal attack for example.
This is a bit of an abstract concept. I'd need to get a list of all files which are successfully uploaded, but not dynamically, i.e. not right after they are completed i.e. not using fileSuccess
event.
Context:
i.e. those visible in the 'completed' table, e.g. one featured in https://github.com/flowjs/ng-flow/blob/master/samples/basic/index.html
and listed by <tr ng-repeat="file in $flow.files">
I am versed with PHP, and only green with Javascript. Any help and possibly a sample code (what to add to app.js) would be very welcome. Many thanks.
The current Request is not compatible with the new PSR-7: http://www.php-fig.org/psr/psr-7/
This could provide better compatibility with Symfony / Zend3... and a lot more.
Hi,
I am using
composer require flowjs/flow-php-server
and getting following notice
[InvalidArgumentException]
Could not find package flowjs/flow-php-server at any version for your minimum-stability (stable).
Check the package spelling or your minimum-stability
What could be wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.