Error when loading large file to local server about datacomets HOT 16 OPEN

Can you provide the line number and file the error is from.

from datacomets.

As I recall, it said parse:1

from datacomets.

So it's coming from the backend and likely means Pyulog is failing to read the file somehow. I'm not sure I can debug this one without the log file. Could you share it?

from datacomets.

Unfortunately I cannot share the particular log file. However, I might be able to find another one and test with that as well. I will also look into the parsing with python.

from datacomets.

I understand that no worries, if you can run the ulog2csv command it is most likely a network issue and not a issue with the log.

from datacomets.

It is definitely not an issue with the log. It doesn’t fail every time. If I try multiple times, it will succeed at some point.

from datacomets.

I see, definitely a network issue then, I am working on a javascript parser which would solve these issue but if it seems like it will take a lot of time I will invest more time into better netcode.

from datacomets.

The log is fine, if I try multiple times with the same log, eventually it will succeeed.

from datacomets.

With the larger logs i have noticed it can sometimes fail to send all the data to the python sever correctly. With a JavaScript parser that step is not necessary any more and there shouldn't be network failures since everything would happen on the client side.

from datacomets.

In the mean time, I could setup datacomets to accept pre-processed logs. This way for very large logs you can parse them separately with a quick python command and then load the log into data comets. This would also be faster for very large logs since you dont have to deal with the network. Less convenient but a stop gap for now, what do you think?

from datacomets.

Sounds great. Please let me know if I can be of any assistance.

from datacomets.

@dsaffo I missed the comment about preprocessed logs. How would I preprocess the logs then? It doesn't seem to be the "parsing" that fails, but the uploading. Which would still have to take place as the server is plotting?

Please correct me if I am wrong.

from datacomets.

Keep in mind, that when I say "local server", it is a server I started using your suggestion with gunicorn.

from datacomets.

from datacomets.

Actually after doing so more research I think I will try to implement this plugin instead https://github.com/blueimp/jQuery-File-Upload/blob/master/README.md#features. It features file chunking and progress bars so might get 2 for 1 solution here.

from datacomets.

@dsaffo That would definitely be the preferred solution.

from datacomets.