GraphicsMagick Steams Failing? about form-data HOT 44 CLOSED

Just stumbled across this when I thought I was going to lose my mind. Probably not too helpful to you two at this point but if some weary traveller finds this thread, adding the filetype solved my issue:

.stream('png', function( err, stdout, stderr ) {

    var buf = new Buffer(0);

    stdout.on('data', function(data) {
        buf = Buffer.concat([buf, data]);
    });

    stdout.on('end', function() {

        var data = {
            Bucket: 'my-bucket-name',
            Key: imageFilename,
            Body: buf
        };

        s3.putObject(data, function( err, response ) {
            if ( err ) {
                console.log( err );
            } else {
                console.log( chalk.green('Image #' + iterator + ' successfully generated for ' + text ) );
                iterator++;
                setImmediate( generateImages );
            }
        });
    });

from form-data.

What node version do you use?

from form-data.

0.8.14 currently, is there another version that this might work better with?

from form-data.

Thanx and I don't know yet, just want to test it with same environment as yours.
I'll get back to you.

from form-data.

In my actual code (not the bug report above), which uses the same gm stream, it acts different on 0.10.4 than it did on 0.8.14. No errors, and the request.post() calls back with undefined.

For reference, my code is based off of the Content-Length "fix" in the #316 discussion, eg:

var FormData = require('form-data');
var request = require('request')
var form = new FormData();
form.append('api_key', api_key);
form.append('product_id', 19);
form.append('file', fs.createReadStream(path.join(__dirname, '/IMG_0859.png')));

form.getLength(function(err,length){
    var r = request.post('http://your-site.com', { headers: { 'content-length': length } }, function(err, res, body){ console.log(body) });
     r._form = form
});

I have not tried the code in the OP example with 0.10.4 yet

from form-data.

In my tests it throws no errors (node 0.8.14) but form-data ends with timeout.
When I tried to point stdout from your example into local file,
I ended up with empty file.

Can you test that you can out put your stdout into local file?

from form-data.

Writing to a file works for me. I've been using this same gm code for a while anyway with no issues, i was piping the stdout into an ftp client which would upload it to another server without troubles

from form-data.

But yea, i couldn't make sense of this issue.

My workaround, for the time being.. is to not try and use the GM stream into FormData. GM reads the image, and then writes the image to disk. FormData then opens up a new ReadStream for that file, and uploads that.

It works, but obviously it's twice as costly.. not really optimal :/

from form-data.

Can you give me your actual code, so I can play with it?
Thanx.

from form-data.

My actual form-data code is spread out among multiple files and libraries, but during tests i use the code i gave. The gm code is identical to what i gave.

Am i doing something wrong, is it working in your tests?

from form-data.

In my tests gm does weird stuff,
just simple example from their docs:

var fs       = require('fs')
  , gm       = require('gm')
  , filename = './input.png'
  ;

gm(filename)
  .stream(function(err, stdout, stderr)
  {
    var f = fs.createWriteStream('./output.png')

    var e = fs.createWriteStream('./output.txt')

    stderr.pipe(e);

    stdout.pipe(f);
  });

In result:

output.png is empty and in output.txt I have following: execvp(): No such file or directory

You said local streaming to the file works for you,
can you show me your code, just to make sure gm is playing nice.

Thanx.

PS. Trying same files with fs.create(Read/Write)Stream works fine.

from form-data.

The graphicsmagick part is identical for me.. that's what i mean :/

gm(req.files.image.path)
  .resize(100, 100)
  .noProfile()
  .stream(function(err, stdout, stderr) {

That is literally the code i use:/

I then send stdout to another function to simplify the upload interface, which uses

var FormData = require('form-data');
var request = require('request')
var form = new FormData();
form.append('api_key', api_key);
form.append('product_id', 19);
form.append('file', my_stout_stream);

form.getLength(function(err,length){
    var r = request.post('http://your-site.com', { headers: { 'content-length': length } }, function(err, res, body){ console.log(body) });
     r._form = form
});

Though, obviously this part doesn't work (hence the issue hah). So currently i am writing the image back to disk, and then after the form is creating it's own stream from disk.

from form-data.

Maybe you need to actually have GM do something to the image for it to output a stream? Sounds silly i know.. but i can't think of why the demo gm code wouldn't work for you :/

from form-data.

Originally I tried with resize() it was the same result.

Can you show me part of your code where you write to disk and it works,
because it doesn't work for me and I think it might gm problem not form-data.

Thank you.

from form-data.

If you despise Coffee, let me know i can change it for you haha.

      gm(req.files.avatar.path)
        .resize(260, 260)
        .noProfile()
        .write "#{req.files.avatar.path}_main.png", (err) ->
          if err? then return next err
          rfs.upload "#{req.files.avatar.path}_main.png", "target.png", (err, res, body) ->

rfs.upload is just the wrapper around form-data, no gm code is in there.

Also note that this is simply my workaround, and obviously not using Streams. Streams did work for me (since writing to disk works without problem either way), but i am using .write() because it gives me a free, clean callback for when the write is done. Rather than using .stream() and having to create a write stream, piping, and callback.

from form-data.

Obvious question, but you do have GraphicsMagick installed, correct? If the node-gm can't find GraphicsMagick, i'm not sure what it would do.. maybe it just returns nothing..

Though, i should hope it would throw a massive error heh.

from form-data.

Yep, when I tried write() it actually told me that I haven't. :) And stream's stderr wasn't as verbose.
I thought I did have :) most likely it didn't survive last spring cleaning :)

While I'm take care of it, can you try to write to disk via streaming interface,
just to make sure it works for you.

Thanx.

from form-data.

While I'm take care of it, can you try to write to disk via streaming interface,
just to make sure it works for you.

I did before, and it did work. :)

from form-data.

I simply used fs.createWriteStream

And, like i mentioned earlier i had been using stdout into an node-ftp connection with no issues for weeks. :)

from form-data.

Hey, I've found the problem and fixed it. Going to submit PR shortly. :)

from form-data.

@leeolayvar Please take a look if it works for you.

Example of extra options you need to pass:

form.append('gm-file', stdout, {
  filename: 'output.png',
  contentType: 'image/png',
  knownLength: 13135
});

If everything is fine I'll update npm.

from form-data.

Hmm, is knownLength required? I have the issue with the length that uses this hack:

var FormData = require('form-data');
var request = require('request')
var form = new FormData();
form.append('api_key', api_key);
form.append('product_id', 19);
form.append('file', my_stout_stream);

form.getLength(function(err,length){
    var r = request.post('http://your-site.com', { headers: { 'content-length': length } }, function(err, res, body){ console.log(body) });
     r._form = form
});

So i don't know the length at the time of append()

from form-data.

The hack/fix is from request/request/issues/316

from form-data.

I just used for my tests,
so if that hack worked before it should.

There was problem with form-data not setting proper boundary headers without filename and mime-type.

Can you try new version with your code and see if it's working?

Thank you.

PS. I'll investigate mikeal's hack.

from form-data.

Are you using request to send stuff out?

Problem described in request/request#316 is problem with request not with form-data, it calculates right length even from the appended file and request (input, don't confuse with using request as transport) streams, former have file descriptor and latter from Content-Length header.

And in your case it's a GraphicsMagick stream with has no file descriptor attached nor it has precalculated filesize property. And if you want to send it out as it goes, you need to know final size in advance.

So it will be question to gm how to retrieve final filesize from their stream object.

from form-data.

Going thru gm's repo I found this "solution" aheckmann/gm/issues/124,
not sure if it would be much help, but seems like it's the closest we can get.

from form-data.

Are you using request to send stuff out?

Problem described in request/request#316 is problem with request not with form-data, it calculates right length even from the appended file and (input, don't confuse with using request as transport) request streams, former have file descriptor and latter from Content-Length header.

And in your case it's a GraphicsMagick stream with has no file descriptor attached nor it has precalculated filesize property. And if you want to send it out as it goes, you need to know final size in advance.

So it will be question to gm how to retrieve final filesize from their stream object.

Yea i am using request to actually submit the data. It had an easier callback to retrieve the returned data than i could find with form-data, so i chose request.

So, i am attempting to make your pull work, but am getting 411 errors. Attempting to work it out

edit: Note i am using gm(stdout).filesize(function (err, filesize){}) to calculate filesize for knownLength

from form-data.

Hmm, i'm looking into it now.. it seems gm().filesize() is a pretty.. annoying function heh. It returns filesize.. in human readable format. I'm trying to see if there is a way around this very odd behavior.

from form-data.

Yeah, I tried it myself and it's weird :) It returns you approximate size :)

from form-data.

So this entire thing hinges on finding the byte size of a stream that both has no size indicator, and the library providing the stream provides a useless filesize() function.. jesus hah.

from form-data.

Another wild idea, try to set arbitrary size, something that is bigger for sure, like original image size.
In the age of chunked requests it might work :)

from form-data.

And it's hard to know stream size in advance, you know because it's stream :)

from form-data.

Yea, i figured it buffered the stream or calculated the end size somehow :s

from form-data.

But yea, not providing the stream size is the optimal solution.. and it would work fine, but this whole problem with request and 411, is that older/annoying servers are configured in such a way that they are not supporting chunked multipart data. Gah.

from form-data.

Yeah, I'll read more about chunked requests, maybe something like that can be incorporated into form-data.

I assume you don't want to keep the whole thing in the memory, right? Otherwise there is streamToBuffer method which would accumulate it so you can measure it.

from form-data.

It would be bad form.. and possible break things, yea.

Frankly, i think this whole thing hinges now on gm. If it can't reliably predict output size, then there is no way for these 411 problems to resolve themselfs.. i think. Writing to file and reading it back might be the only way. Ugh

from form-data.

Ok, I close issue for now and if later I'll get something going with chunked post, I'll ping you here.

from form-data.

With that said, regarding your PR.. i can only assume it works. With no way to set the options.knownLength value from GraphicsMagick, i can't confirm anything :/

Thanks for all the help!

from form-data.

It works. :) I have test for it. :)
I wanted to know if it's useful. :)

Thank you.

from form-data.

Hey, idea. If you do it thru request, you can control http headers you send.
So set knownLength to anything on append and instead 'Content-Length' header in request options add {'Transfer-Encoding': 'chunked'}. It might work for you. In my tests some server were unhappy about it, but my local node.js server (and formidable) parsed it no problem. (Yet to fully test it with gm's stream).

from form-data.

I haven't tried that! I'll give it a go and see if it flies with my problematic servers

from form-data.

Btw, I noticed that gm stream hangs if the there is no end event listener.

This is my code that works with node.js (+ formidable) server on the other side:

gm(filename)
  .stream(function(err, stdout, stderr)
  {
    stdout.on('end', function(){});

    var form = new FormData();
    form.append('gm-file', stdout, {filename: 'output.png', contentType: "image/png"});
    form.submit(endpoint, resHandler);
  });

from form-data.

No luck here, though it's the servers fault for not supporting chunked, not the client, so no real big surprise.

from form-data.

Sad. Good thing you can stream to disk and back, so you can show some kind of progress to the user.

from form-data.