flowjs / flow-php-server Goto Github PK

On every file upload I get an error 400. Tried to put some error_log() into the code, but nothing showed up. The error message comes from this:

if ($file->validateChunk()) {
     $file->saveChunk();
} else {
    // error, invalid chunk upload request, retry
    header("HTTP/1.1 400 Bad Request");
    return ;
}

But I don't know why there is an error...
(Using the example-code from the node-js sample (index.html))

I started working on version 1.0.1 with the help of my guide and I was met with this error: Class 'Flow\Config' not found

but In version 1.0.0, there was no such problem

my code:
require_once(APPPATH.'third_party'."/Flow/autoloader.php");
$config = new \Flow\Config();
.
.
.

I followed the docs closely. Uploading big files with multiple chunks keeps giving me UPLOAD_ERR_PARTIAL error. How do I fix this?

I have the flowjs client on localhost and then hosted the ng-flow server on cloud. Sending the file to the target of the server... Keep getting this error

sha1(): Passing null to parameter #1 ($string) of type string is deprecated"

But when I placed the server and client on same host, it works... but on different host it doesnt work and have added headers such as

header('Content-Type: application/json; charset=UTF-8');
header('Access-Control-Allow-Origin: *');
header('Access-Control-Max-Age: 3600');

For CORS but yet its not working.

On client i have

var flow = new Flow({
  target:'https://my-server-link/upload', 
  query:{upload_token:'my_token'},
chunkSize: 1024*1024,
});

Please what is the workable method to use this?

This is a nasty code but It work for me for an AngularJs and Laravel 5 integration. I am just happy and want to share it for sme. It is a copy from one post here but modified.

    public function getSubirArchivo($declaracion) {
        error_log('Documento 22');

        $request = new \Flow\Request();
        $config = new \Flow\Config(array(
            'tempDir' => './chunks_temp', //With write access
        ));

        $file = new \Flow\File($config, $request);
        $response = Response::make('', 200);

        $destination = './archivos/' . $declaracion . '_' . $request->getFileName();
        if ($_SERVER['REQUEST_METHOD'] === 'GET') {
            error_log('Documento GET');
            if (!$file->checkChunk()) {
                error_log('Documento GET2');
                return Response::make('', 204);
            }
        } else {
            if ($file->validateChunk()) {
                $file->saveChunk();
                error_log('Documento VaLido !!!');

            } else {
                error_log('Documento INVALIDO');
                // error, invalid chunk upload request, retry
                return Response::make('Error in chunck', 400);
            }
        }
        if ($file->validateFile() && $file->save($destination)) {
            error_log('EXITO!');
            $response = Response::make('pass some success message to flow.js', 200);
        }
        return $response;
    }

I'm currently working on a project which uses flow.js and flow-php-server.

9 times out of 10, when uploading the same 5mb PDF it transfers fine. However, sometimes it will arrive corrupted. A successful transfer would have a mime-type of "application/pdf", the corrupt transfers have a mime-type of "application/octet-stream".

I'm not sure if the root of the corruption is on the client or server side. Let me know what info you need and I'll pull together what I can.

When I upload a large file (e.g. 1.4 GB), I get a timeout (http 504) on one of the (post) requests. The file is completely uploaded and merged. However, the client does not know about it. On the client side, the transfer information ends with: progress:1, error:true, success: false, complete:true. The request is waiting for the response from the server. Maybe I can solve the problem by increasing some timeouts on the server side. However, I have already tried a lot without success. Would it be possible to flush something like a status from time to time on php side so that I don't run into the timeout?

Thanks for any advice.

Need thoughts on this:

            $request = new \Flow\Request();

    $config = new \Flow\Config();

    $config->setTempDir('./');

    $file = new \Flow\File($config, $request);

    if ($_SERVER['REQUEST_METHOD'] === 'GET') {
        if ($file->checkChunk()) {
            header("HTTP/1.1 200 Ok");
        } else {
            header("HTTP/1.1 204 No Content");
            return ;
        }
    } else {
    //print_r($file); // -> UNCOMMENTING THIS MAKES THIS CALL WORK...
      if ($file->validateChunk()) {
          $file->saveChunk();
      } else {
          // error, invalid chunk upload request, retry
          header("HTTP/1.1 400 Bad Request");
          return ;
      }
    }
    if ($file->validateFile() && $file->save($request->getFileName())) {
        // File upload was completed
    } else {
        // This is not a final chunk, continue to upload
    }

I know the title is bit vague so let me explain.

During my tests I wanted to simulate big file uploads with relatively small files so on the front end I set:

simultaneousUploads: 3 (which is I believe default setting)
`chunkSize: 2048 (which is 1024*1024 by default)

This gave me kind of simulation of multiple chunk uploads. Using the library on the back end you have to follow four steps:

1. validateChunk()
2. saveChunk()
3. validateFile()
4. save()

If you have multiple concurrent uploads this may lead to corrupted files. Lets take a case of last two chunks being uploaded where x axis is time:

chunk1:  |---validateChunk---||---saveChunk---||---validateFile---||---save---|
chunk2:               |---validateChunk---||---saveChunk---||---validateFile---|

when validateFile is called on chunk1 the saveChunk for chunk2 already begun so chunk1::validateFile will return true and proceed to save file with chunk2 not fully saved.

I saw this error in my logs already few times. The above example will also lead to double save call. And in my case it leads to duplicate key error in the database.

To fix the problem library would have to implement locking not only in save but in validateFile methods . Its not enough to call file_exists in validateFile. We have to know the chunk is fully uploaded.

I'm working on new version of on-line photo printing software. So I tested your package. In this case often happens that users send the same file.

I enabled duplicate uploads via JS:

var flow = new Flow({
    target: 'upload.php',
    allowDuplicateUploads: true,
});

When you select same file multiple times until first upload is finished only one file is saved.

To easy reproduce use this code (upload.php):

<?php

require_once './vendor/autoload.php';

$config = new \Flow\Config();
$config->setTempDir('./tmp');
$request = new \Flow\Request();
$uploadFolder = './upload/'; // Folder where the file will be stored
$uploadFileName = uniqid()."_".$request->getFileName(); /
$uploadPath = $uploadFolder.$uploadFileName;
if (\Flow\Basic::save($uploadPath, $config, $request)) {
  // file saved successfully and can be accessed at $uploadPath
} else {
  // This is not a final chunk or request is invalid, continue to upload.
}
sleep(10);

And add sleep(5) to /vendor/flowjs/flow-php-server/src/Flow/File.php, line 192.

sleep(5);
if ($this->config->getDeleteChunksOnSave()) {
    $this->deleteChunks();
}

Just before chunks are merged.

Explanation: Flow.js creates same identifier for same fille. Flow-PHP-Server uses this identifier for chunk names (after hasing) = same chunk names for different uploads. When first file is merged chunks for next one are deleted. So another file cannot be created.

Hi,

I am using this code and filename save it as upload.php as follows:
But after submitting the form through html and where to pass uploaded_file (i.e.,) $_FILES['uploaded_file']['tmp_name'] and $_FILES['uploaded_file']['name'] in backend how to use.

require_once 'vendor/autoload.php';

$config = new \Flow\Config();
$config->setTempDir(storage_path() . '/temp');
$destination = 'uploads/';
$file = new File($config);
$request = new \Flow\Request();

    if ($_SERVER['REQUEST_METHOD'] === 'GET') {

        if ( ! $file->checkChunk() ) {
            return \Response::make('',204);
        }
    } else {
        if ($file->validateChunk()) {
            $file->saveChunk();
        } else {
            return \Response::make('', 400);
        }
    }
    if ($file->validateFile() && $file->save($destination . $request->getFileName())) {
        return \Response::make('File upload was completed', 200);
    }

It's probably just me, but uploading larger files (3GB+) is not really possible currently. Chunking files and uploading each junk to the server works just fine, even for very large files. But the larger the file the more chunks need to be put together by the server. This process seems to be very time consuming (in PHP) and if it takes too long the script just timeouts.

For some people it's possible to just increase the timeout limit, but even then I'm not sure if that's the best solution. So I'm wondering if we could just bypass this problem by uploading the chunks in the correct order and appending them to a "temp" file (after a chunk was successfully uploaded). In this case there shouldn't be a timeout problem at all, no matter how large the file is. At least with my very limited understanding of PHP.

If there aren't any drawbacks I'm wondering if you could imagine adding something like that?

I have two servers setup, A & B, with urls A & B pointing to the servers respectively. Both servers contain the same codebase.

If I go directly to url B on server B, and initiate a flowjs upload, everything works as expected.
If I go to url A (server A), but have the target url for the flowjs be url B (server B), instead of uploading the file, I just got three requests with an empty response but a 302 Found Header.

Any insight on getting this working cross-server without me digging into the codebase?

All SSL's are wildcard, and sessions are cross-domain to a redis server.

Hi,

Is there a small example available how to use the flow-php-server from scratch?

Thanks
Tom

Hello,
I'm trying to implement flow-php-server in Slim framework, but it is not working.
Here is my code:

$app->any('/upload', function(Request $req, Response $resp, $args){

  try {

    $response = [
      'type' => 'error',
      'message' =>'no data'
    ];

    $request = new \Flow\Request();

    $dest = 'files/' . $request->getParam('path');
    $chunks_temp_folder = $request->getParam('chunks_temp_folder') ?: './chunks_temp_folder';

    if (!is_dir($chunks_temp_folder) || !is_writable($chunks_temp_folder)) {
      throw new Exception("Chunks directory {$chunks_temp_folder} does not exists or is not writeable");
    }

    $config = new \Flow\Config(['tempDir' => $chunks_temp_folder ]);

    if (!is_dir($dest) || !is_writable($dest)) {
      throw new Exception("Destination directory {$dest} does not exists or is not writeable");
    }

    $dest .= DIRECTORY_SEPARATOR . $request->getFileName();

    if (\Flow\Basic::save($dest, $config, $request)) {

      $file = $request->getFile();

      $response = [
        "type" => "success",
        "name" => $file['name'],
        "filetype" => $file['type'],
        "error" => $file['error'],
        "size" => $file['size'],
        "readablesize" => self::human_filesize(filesize($dest), 2),
        "fullpath" => $dest
      ];
    } else {
      throw new Exception("\Flow\Basic::save failed");
    }

  } catch (Exception $e) {
    $response['type'] = 'error';
    $response['message'] = $e->getMessage();
  }

  echo json_encode($response);
  return;
});

The "\Flow\Basic::save failed" exception is being thrown. Anyone has been successful in doing this?

the current implementation is based on the obsolete mongo lib.
this should be changed to mongodb/mongodb.
I'd be willing to provide a PR :)

The script currently overwrites if there is an identically named file in the final upload directory. It should be able to rename the file (numbered perhaps like filename-2.ext) if it encounters this kinds of situation.

Hey there,

and thank you for the great work on this library.

This issue can be deleted. In case someone else is wondering, in Laravel 5 the standard Request::input('parametername') does the trick.

Cheers

I'm using the basic sample and I always get a 404 error (probably because of chunk checking)
but no file is saved even using the basic code.
http://tinyurl.com/keu5gbk

No chunk files are created nor the file is uploaded.
The demo presented on this page works just fine:
https://github.com/flowjs/flow.js/blob/master/samples/Backend%20on%20PHP.md

Thanks

there are some commits after the 1.0.3 release, which I'd need. Would it be possible to cut a "fresh" release, please?

The PHP server only support the default multipart option, and not the octet one.

This should be added to the documentation right now,
and maybe added to the library as contributions go :)

• the Request must fake an upload array:

    $fileArray    = [
        'name'     => $requestParams['flowFilename'],
        'type'     => '',
        'tmp_name' => 'php://input',
        'error'    => UPLOAD_ERR_OK,
        'size'     => $requestParams['flowCurrentChunkSize'],
    ];

• the File::_move_uploaded_file method must not use move_uploaded_file but instead read 'php://input' and file_put_contents it

Is there somewhere a config where I can prevent that metadata like portrait modus are not stripped from uploaded images? After uploading an iOS iPhone portrait modus photo, it tilts the images.

When I enable testChunks optsion in my JS script, I get a 404 error but when I set it to false it works just fine, what am I doing wrong?

I can't find any way to get the mime type of the file from either the Flow\Request or the Flow\File object? If this functionality does not exist, I'd be glad to make a pull request for it, but I don't want to go reinventing the wheel and it seems like a pretty important thing to omit.

Seems that the lib keep the original file name and there is no way that the server can tell ng-flow the new file name, my server output this on complete:

{
  "success":true,
  "flowTotalSize":58110,
  "flowIdentifier":"10152987551505362-2e68232139983ba857669baf3c2d45c7_development.jpg-58110",
  "flowFilename":"4498b3d20302c1c3b75ce37c1e5fefac",
  "flowRelativePath":"http:\/\/localhost\/images\/users\/4498b3d20302c1c3b75ce37c1e5fefac"
}

my upload script:

<?php
    $path = __DIR__ . '/../public/images/users/';
    $tmp = __DIR__ . '/../tmp/';

    if ($_SERVER['REQUEST_METHOD'] === 'GET') {
      $chunkDir = $tmp . DIRECTORY_SEPARATOR . $_GET['flowIdentifier'];
      $chunkFile = $chunkDir.'/chunk.part'.$_GET['flowChunkNumber'];
      if (file_exists($chunkFile)) {
        header("HTTP/1.0 200 Ok");
      } else {
        header("HTTP/1.0 404 Not Found");
      }
    }

    // $file = $path . (isset($_FILES['file']) ? $_FILES['file']['name'] : $_GET['flowFilename']); // this is using original name, but in this case the file sooner or later will be overrite by other user that upload a file with the same name
    $file = $path . md5(time()); // The way I want, it need to add the extension but it was a test only

    if (isset($_FILES)) {
      copy($_FILES['file']['tmp_name'], $file);

      echo json_encode([
          'success' => true,
          //optional
          'flowTotalSize' => $_FILES['file']['size'],
          'flowIdentifier' => $_FILES['file']['name'] . '-' . $_FILES['file']['size'],
          'flowFilename' => basename($file),
          'flowRelativePath' => 'http://localhost/images/users/' . basename($file)
        ]);
    } else {
      if (\Flow\Basic::save($file, $tmp)) {
        echo json_encode([
          'success' => true,
          //optional
          'flowTotalSize' => $_GET['flowTotalSize'],
          'flowIdentifier' => basename($file),
          'flowFilename' => basename($file),
          'flowRelativePath' => 'http://localhost/images/users/' . basename($file)
        ]);
      }
    }

I'm using PHP 5.4 and when using the Flow\Request object I get the error that the abstract class method 'wgetCurrentChunkNumber' must be declared. Which it isn't... I'd submit a fix, but I'm not even sure what the function would do...

Cheers
Ben

How are the file names / flowIdentifier sanitized before being used ? Maybe I'm wrong, but looking at the code, I can't see any checks being made that would prevent a directory traversal attack for example.

This is a bit of an abstract concept. I'd need to get a list of all files which are successfully uploaded, but not dynamically, i.e. not right after they are completed i.e. not using fileSuccess event.

Context:

1. A user uploads a few files. Then deletes some. Then uploads some more, etc.
2. When they are satisfied with the result, the files can be now processed further, in PHP.
3. The user clicks on a "ready" button and PHP automagically receives a list of files registered with ng-flow,

i.e. those visible in the 'completed' table, e.g. one featured in https://github.com/flowjs/ng-flow/blob/master/samples/basic/index.html and listed by <tr ng-repeat="file in $flow.files">

I am versed with PHP, and only green with Javascript. Any help and possibly a sample code (what to add to app.js) would be very welcome. Many thanks.

The current Request is not compatible with the new PSR-7: http://www.php-fig.org/psr/psr-7/

This could provide better compatibility with Symfony / Zend3... and a lot more.

Hi,

I am using

composer require flowjs/flow-php-server

and getting following notice

[InvalidArgumentException]
Could not find package flowjs/flow-php-server at any version for your minimum-stability (stable). 
Check the package spelling or your minimum-stability

What could be wrong?