Large file uploads crash Yaws about yaws HOT 7 CLOSED

Also tested same with Yaws 1.88.

Also, with temp_file, there is no problem.

It's obviously not very smart to not use temp file if large files are expected, but to me it seems somewhat excessive to require over 1GB of RAM for a 12MB upload, I think.

from yaws.

Thanks for providing all the details -- I've duplicated it on the latest github bits as well.

I'll investigate why it's trying to use so much memory.

from yaws.

Turns out the multipart/form-data handling code is quite old, almost 10 years as a matter of fact. It converts all the posted data into lists, keeps multiple copies of the lists, and works through them byte by byte. Together all this adds up to significant memory usage.

from yaws.

I didn't know what a web form was back then. :D

from yaws.

:-) Yes, I was surprised to find the code had not really been touched since then.

Improvements will be committed shortly.

from yaws.

I've pushed changes to try to address this problem. Uploads that used to cause Yaws to run out of memory on my laptop now work, so memory usage is better, but it's still too high. Unfortunately backwards compatibility makes this hard to fix due to the need to convert results to lists, but with my next commit I hope to add a new option to the multipart handling functions to tell Yaws to leave the data as binaries. This should allow apps willing to use the new option to handle large uploads.

from yaws.

Added new binary option for handling uploaded file data as binaries rather than lists, though for backward compatibility the list form is still the default. Given this addition together with the previous fix, I'm closing this issue.

from yaws.