Giter VIP home page Giter VIP logo

busboy's Introduction

co busboy

NPM version Node.js CI Test coverage npm download

busboy multipart parser using co or koa.

Example

const parse = require('co-busboy')

app.use(async (next) => {
  // the body isn't multipart, so busboy can't parse it
  if (!this.request.is('multipart/*')) return await next()

  const parts = parse(this)
  let part
  while (part = await parts()) {
    if (part.length) {
      // arrays are busboy fields
      console.log('key: ' + part[0])
      console.log('value: ' + part[1])
    } else {
      // otherwise, it's a stream
      part.pipe(fs.createWriteStream('some file.txt'))
    }
  }
  console.log('and we are done parsing the form!')
})

Note that parts will be delievered in the order they are defined in the form. Put your CSRF token first in the form and your larger files last.

If you want co-busboy to automatically handle the fields, set the autoFields: true option. Now all the parts will be streams and a field object and array will automatically be populated.

const parse = require('co-busboy')

app.use(async (next) => {
  const parts = parse(this, {
    autoFields: true
  })
  let part
  while (part = await parts()) {
    // it's a stream
    part.pipe(fs.createWriteStream('some file.txt'))
  }
  console.log('and we are done parsing the form!')
  // .field holds all the fields in key/value form
  console.log(parts.field._csrf)
  // .fields holds all the fields in [key, value] form
  console.log(parts.fields[0])
})

Example for csrf check

Use options.checkField hook function(name, val, fieldnameTruncated, valTruncated) can handle fields check.

const parse = require('co-busboy')

app.use(async (next) => {
  const ctx = this
  const parts = parse(this, {
    checkField: (name, value) => {
      if (name === '_csrf' && !checkCSRF(ctx, value)) {
        var err =  new Error('invalid csrf token')
        err.status = 400
        return err
      }
    }
  })
  let part
  while (part = await parts()) {
    // ...
  }
})

Example for filename extension check

Use options.checkFile hook function(fieldname, file, filename, encoding, mimetype) can handle filename check.

const parse = require('co-busboy')
const path = require('path')

app.use(async (next) => {
  const ctx = this
  const parts = parse(this, {
    // only allow upload `.jpg` files
    checkFile: (fieldname, file, filename) => {
      if (path.extname(filename) !== '.jpg') {
        var err = new Error('invalid jpg image')
        err.status = 400
        return err
      }
    }
  })
  let part
  while (part = await parts()) {
    // ...
  }
})

API

parts = parse(stream, [options])

const parse = require('co-busboy')
const parts = parse(stream, {
  autoFields: true
})

options are passed to busboy. The only additional option is autoFields.

Note: If busboy events partsLimit, filesLimit, fieldsLimit is emitted, will throw an error.

part = await parts()

Await the next part. If autoFields: true, this will always be a file stream. Otherwise, it will be a field as an array.

  • Readable Stream

    • fieldname
    • filename
    • transferEncoding or encoding
    • mimeType or mime
  • Field[]

    1. fieldname
    2. value
    3. valueTruncated - Boolean
    4. fieldnameTruncated - Boolean

If falsey, then the parser is done.

parts.field{}

If autoFields: true, this object will be populated with key/value pairs.

parts.fields[]

If autoFields: true, this array will be populated with all fields.

License

The MIT License (MIT)

Copyright (c) 2013 Jonathan Ong [email protected] Copyright (c) 2015 cojs and other contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

busboy's People

Contributors

atian25 avatar bytemain avatar dead-horse avatar eivindfjeldstad avatar fengmk2 avatar haoxins avatar jonathanong avatar killagu avatar mscdex avatar xieren58 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

busboy's Issues

fileSize limit bug

I found that fileSize works well if we set it large enough , but when we have a small fileSize limit , it not works , for instance if you set fileSize=1024 or fileSize=1 , there is no limit , you can upload file with no size limit , I test it and I see that I can upload 5MB files

I guess , When we set fileSie limit less than highWaterMark it occures

enter the busboy module twice,but the opations.headers is different?

I am studying the koa example--multipart,there is my code:

    var parts = parse(this.request,{
        autoFields:true
    });
    var tmpdir = path.join(os.tmpdir(),uid());

    yield fs.mkdir(tmpdir);
    var files = [];
    var file;
    var part;
    while(part = yield parts){
        files.push(file = path.join(tmpdir,part.filename));
        yield saveTo(part,file);
    }

there is a error "Missing content type" when i test use:

<form method="post" action="http://localhost:3000" enctype="multipart/form-data">
        <input type="file" name="test">
        <input type="submit" value="test">
    </form>

so I log the opations.headers,before new Busboy(options) i found the headers log twice and they are different.

why they log twice and why they are differents?

thanks i am a fresher

the part.length is undefined and can't catch error


var parse = require('co-busboy');
app.use(function* (next) {
  if (!this.request.is('multipart/*')) return yield next;
  var parts = parse(this, {
          checkFile: function (fieldname, filestream, filename) {
          if (path.extname(filename) !== '.dat') {
            return new Error('invalid filename extension');
          }
        }
   });
  var part;
try {
  while (part = yield parts) {
    if (part.length) {
      console.log('key: ' + part[0]);
      console.log('value: ' + part[1]);
    } else {
      part.pipe(fs.createWriteStream('some file.txt'))
    }
  }
catch (e) {
}
});

when upload a image file, the 'part' is FileStream instance.but it is no length method.
and then can't catch checkFile error. my request url's status always 'pending'.
i don't know why, pls help

PR #23 test fails due to bug in co-busboy

Right now co-busboy ignores files/fields with names like 'getOwnProperty'.
That's just wrong. There's no legal reason to ditch fields with valid names.

In theory, such names never happen. But if you think about a specific service for programmers, such names are perfectly normal. I had a problem with that thing and made PR to fix it.

PR #23 removes this technical restriction and fixes this behavior. Guess, the tests should be fixed too.

Lost latest field while multipart form

I use multipart form to upload file and edit fields, when i upload a file i will get all field values, but i don't upload a file there will always lost a field.

pls check is it busboy bug or something else.
thanks.

Waiting for streams

Hey – not sure where to file this – but I had a quick question. Suppose I am using a koa middleware that looks like this:

app.use(function* (next) {
  var parts = parse(this), part;
  while (part = yield parts) part.pipe(/* somewhere */);
  // Here
});

I would like to get the part stream into a buffer, which I've used a library like accum or concat-stream (https://github.com/jeffbski/accum, https://github.com/maxogden/concat-stream) for before. But it would be really great if I could end up at the // Here comment, so I could stay in Koa's scope, etc.

Is there a way to do this?

this.req || this

maybe so we can do parse(this) == parse(this.req) koa special sauce

Queue parts

Need to create a queue of parts in case users yield too long between parts.

Payload Too Large

I have a 1MB limit for files , When I upload a 200KB file with a field like following

------WebKitFormBoundaryOmz20xyMCkE27rN7
Content-Disposition: form-data; name="data";
Content-Type: application/json

{
  "description": "description"
}
------WebKitFormBoundaryOmz20xyMCkE27rN7
Content-Disposition: form-data; name="file"; filename="image.jpg"
Content-Type: image/jpeg


FILE_DATA
------WebKitFormBoundaryOmz20xyMCkE27rN7

everything is OK and works , But the following has Payload Too Large

------WebKitFormBoundaryOmz20xyMCkE27rN7
Content-Disposition: form-data; name="data"; filename="blob"
Content-Type: application/json

{
  "description": "description"
}
------WebKitFormBoundaryOmz20xyMCkE27rN7
Content-Disposition: form-data; name="file"; filename="image.jpg"
Content-Type: image/jpeg


FILE_DATA
------WebKitFormBoundaryOmz20xyMCkE27rN7

the diffrence is filename="blob" when I set filename for json part I have Payload Too Large error

About file size limit ?

if i want to validate file size , index.js line 90 of source code 。can i right??

at index.js line 90: modified source code bellower:

function onFile(fieldname, file, filename, encoding, mimetype) {
if (checkFile) {
// var err = checkFile(fieldname, file, filename, encoding, mimetype)
// if (err) {
// // make sure request stream's data has been read
// var blackHoleStream = new BlackHoleStream();
// file.pipe(blackHoleStream);
// return onError(err)
// }
checkFile(fieldname, file, filename, encoding, mimetype, ## function (err) {
if (err) {
// make sure request stream's data has been read
var blackHoleStream = new BlackHoleStream();
file.pipe(blackHoleStream);
return onError(err)
}
})

}

======then at my app code =========

parts = parse(this, {
"checkFile": function (fieldname, file, filename, encoding, mimetype,cb) {
file.on("limit", function () {
var err = new Error('=====file size ======')
err.code = 'Request_fileSize_limit'
err.status = 400;
cb(err);
//return err;
});

        }, "limits": { "fileSize": 10000000 }
    });

high risk dependency

the dependencies busboy 0.2.x has a dependency dicer that contains a HIGH level vulnerability CVE-2022-24434

Are you free to upgrade it?

Best Regards👍

it couldn't catch file stream error

when request protocol is illegal, it will emit Part terminated early due to unexpected end of multipart data in node_modules/busboy/node_modules/dicer/lib/Dicer.js

maybe remove delegation

not a huge deal, but that's one thing in the spec im not a huge fan of, IMO users shouldn't have to worry about wether or not they should be delegating

autoFields and arrays

Idk if you want to add more features to this lib, but right now you'd need a lot of boilerplate code to handle arrays. It's a one-line fix if we do it here . Let me know what you think

co-busboy was unable to parse ctx using multipart/form-data in koa2

I plan to test a file upload using multipart/form-data in koa2, the below was my code:

var parts = parse(ctx.request);
// or var parts = parse(ctx);
var part;
while (part = await parts) {
var stream = fs.createWriteStream(path.join(os.tmpdir(), Math.random().toString()));
console.log('part===>', part);
part.pipe(stream);
console.log('uploading %s -> %s', part.filename, stream.path);
}

the below was the dump of ctx in log:

ctx===> { request:
   { method: 'POST',
     url: '/upload',
     header:
      { host: '127.0.0.1:3000',
        connection: 'keep-alive',
        'content-length': '2316',
        'cache-control': 'max-age=0',
        accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        origin: 'http://127.0.0.1:3000',
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36 Core/1.47.277.400 QQBrowser/9.4.7658.400',
        'content-type': 'multipart/form-data; boundary=----WebKitFormBoundaryJl7OKIVqHwYSA5fz',
        referer: 'http://127.0.0.1:3000/',
        'accept-encoding': 'gzip, deflate',
        'accept-language': 'zh-CN,zh;q=0.8' } },
  response: { status: 404, message: 'Not Found', header: {} }

but I get the below error as below,my question is :how to get the uploading file name and the content of uploading file ?
[TypeError: part.pipe is not a function]

Unsupported content type:multipart/mixed

LINECONTENT:nodejs.Error: Unsupported content type: multipart/mixed; boundary=409ed440-5bfe-4c7e-b317-1a0735da6e8a
    at Busboy.parseHeaders (/data/lifekh-mp-nodejs-mobile-app-composition-sit/ROOT/node_modules/busboy/lib/main.js:75:9)
    at new Busboy (/data/lifekh-mp-nodejs-mobile-app-composition-sit/ROOT/node_modules/busboy/lib/main.js:22:10)
    at module.exports (/data/lifekh-mp-nodejs-mobile-app-composition-sit/ROOT/node_modules/co-busboy/index.js:31:16)

Deferring stream pipe

I'm looking for a practical and performant method to defer piping of streams until I've validated other parts of the multipart request.

Essentially, I want to limit the files that are stored (streamed to a location on disk, or to the cloud), and then pipe the matching file field streams once everything else is done.

//...
while (part = yield parts) {
   //... part is a stream here. 
}
//... I should have access to file streams here. 

Does anyone know how to achieve this?

It gets stuck if parts were not piped somewhere

For unknown reason if parts of the form were not piped to another stream in the same while-loop (like it was done here: https://github.com/koajs/examples/blob/master/upload/index.js), but instead they were collected to array for further working with them out of this while-loop, then it gets stuck after it went through all the variables of the form.

E.g.

var yieldedParts = [];
while (part = yield parts) {
    yieldedParts.push(part);
    console.log('yielded');
}
console.log('went out of while loop');

In this case it will show 'yielded' as many times as you have form variables, but never 'went out of while loop'

How to use Busboy (special) events

Hello,

In the doc, I did not found these relevant infos. I check the source code of this module as well. These events seems not to be registered or handled. How could I use co-busboy in Koa.js together with the limits check?

Best regards,
Kevin

Plans for async/await compatibility

async-csp would pretty much be a drop in replacement for the chan module which as far as I can tell doesn't seem compatible with async/await. I've ran a few tests and async-csp works great so far. Would a PR make sense ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.