Memory overload when downloading large files. about telegraph HOT 10 OPEN

Great to hear it! Have you dig into a new Network framework?

Yeah, I'm working on a wrapper for the Apple Network framework and a DispatchIO implementation for iOS 11 and lower versions. The SSL implementation is going to be a nightmare, but I'll try 😅

from telegraph.

Hi @aplekhanov, Yes big files are fully loaded in memory. The implementation of the file handler is pretty basic.

Unfortunately to properly send big files to your client you need chunked transfer encoding, I haven't implemented that in the HTTP classes. I'm not really sure how much work it would be to add it to the framework, I haven't explored the topic in too much detail yet.

Perhaps I can have a look at it in the weekend.
Or if you want to show off your coding skills, pull requests are most welcome 😉 😄

from telegraph.

I've just pushed a commit that adds support for range requests on files. That might help in some cases and for example allows for streaming video on iOS. It contains FileHandle code to read parts of the files.

However this does not resolve the memory issue yet. To fix that we need to build in proper streaming, for example by adding something like a bodyStream variable to HTTPResponse.

I will probably add that later on. The problem is that the socket layer (CocoaAsyncSocket) makes it difficult to implement this. I'll probably have to replace that first.

from telegraph.

Just start unsecured server with .serveDirectory on iPad and get file using URLSession from 'http://localhost:9000/big.pdf'. The same picture on macOS. Seems it load the whole file into memory for every downloading session.

from telegraph.

Hi @Zyphrax,
Yeah, I found already that part of code in the HTTPFileHandler response.body = try Data(contentsOf: fileURL) . I guess there must be NSFileHandle to get chunked data from file... and some special http header for this case... right? Very tempting 😈

But right now I need to finish A LOT OF other interesting work. Let's say it is not high priority and most important thing is that it could be realized by you, by me or by someone else in future 🍻

from telegraph.

I will probably add that later on. The problem is that the socket layer (CocoaAsyncSocket) makes it difficult to implement this. I'll probably have to replace that first.

Great to hear it! Have you dig into a new Network framework?

from telegraph.

@Zyphrax I am interested to hear whether or not you decided to pursue this. Not an urgent feature for me, but would be nice to have 😄.

from telegraph.

Hi @JUSTINMKAUFMAN. Darwin sockets are quite complex and especially the SecureTransport SSL layer is going to be a challenge. I'm still working on it, but haven't had much time to spend on it.

As soon as I have a unified network layer, I'll dive into proper buffering for large files.

from telegraph.

Hi @JUSTINMKAUFMAN. Darwin sockets are quite complex and especially the SecureTransport SSL layer is going to be a challenge. I'm still working on it, but haven't had much time to spend on it.

As soon as I have a unified network layer, I'll dive into proper buffering for large files.

Hey

Do you have a progress with this issue?

PS
Telegraph is a Great work!
Thank you

from telegraph.

I'm checked Kitura and Vapor for big files transfering
Kitura seems has the same issue

Vapor support streams for files
I'm not sure, but maybe next link help you
https://github.com/vapor/vapor/tree/main/Sources/Vapor

ChannelResponseBodyStream in
https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/HTTP/Server/HTTPServerResponseEncoder.swift

BodyStreamWriter in
https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/Response/Response%2BBody.swift

https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/HTTP/BodyStream.swift

https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/Request/Request%2BBodyStream.swift

from telegraph.