Giter VIP home page Giter VIP logo

node-red-contrib-ui-upload's Introduction

node-red-contrib-ui-upload

Node-RED Dashboard UI widget node for uploading a file content by WebSocket (Socket.io) streaming.

Supports: browse for file or drag & drop; pause & resume, replay; custom chunk size; backpressure.

Meant to be combined with node-red-contrib-chunks-to-lines.

Screenshot in the Node-RED Dashboard:

Node-RED Dashboard upload widget

In a Node-RED flow, this Upload node (using text transfer type) can advantageously be connected to some standard nodes such as:

  • Split node using the option Handle as a stream of messages: to read one line at a time (works well, also for very large uploaded files)
  • Join node using the automatic mode: to reassemble the uploaded chunks into one single message / string (only for relatively small uploaded files, which can fit in memory)

Example: flow.json

Node-RED flow

But this node really shines when combined with the node-red-contrib-chunks-to-lines node, which can split in a more efficient way, e.g. safe for Unicode and with built-in backpressure for automatic buffering optimisation.

Example: flow.json

Node-RED flow

See also a more advanced example of upload of a large CSV file to an SQL database.

Transfer types

  • When using binary transfer type, the data is uploaded as chunks of byte arrays. This mode can safely transmit any type of file, but requires a transformation before further use (which node-red-contrib-chunks-to-lines can handle).
  • When using text transfer type, the data is uploaded as chunks of ASCII (more precisely Windows-1252 encoding). This mode is not able to properly transmit non-ASCII data such as multibyte strings (e.g. Unicode / UTF-8) or binary files, but is convenient for basic ASCII text.

Backpressure

This node supports backpressure / flow control: it can wait for a tick before uploading the next chunk of data, to make sure the rest of your Node-RED flow is ready to process more data, instead of risking an out-of-memory condition.

To make this behaviour potentially automatic (avoiding manual wires), this node declares its ability by exposing a truthy node.tickConsumer for downstream nodes to detect this feature. Using this approach, the backpressure is done automatically when using the node-red-contrib-chunks-to-lines node.

For a manual approach, just send { tick: true } to the node for triggering the upload of the next chunk of data.

By default, in absence of wired input on this node, a tick will be automatically generated upon full reception of a chunk of data to trigger the upload of the next one.

This backpressure mechanism also gives time to other nodes communicating on the same shared WebSocket and avoids disconnections when uploading large files.

Credits

License: Apache 2.0, 2020-2022.

Originally made by Alexandre Alapetite at the Alexandra Institute.

node-red-contrib-ui-upload's People

Contributors

alkarex avatar dependabot[bot] avatar eamon0989 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

node-red-contrib-ui-upload's Issues

This node takes over the complete ui

HI
I have the strange effect that as soon as i integrate this node into a flow the rest of my configured ui disapears.
I use W10 and no matter what browser, just this one element is on screen in ui.
Deactivate is not enough, I have to restart node-red afterwards to get the rest of my ui back.

THX in advance

upload fast but consume slow (ui upload + chunks to line)

Hello,
I am using the ui-upload & chunks-to-lines in a node-red setup and the Process is as follows:
Upload -> chunks-to-lines -> process of filtering & writing data to files -> visualize on dashboard
Where the uploaded file ~400Mbytes. & my Node-red server is not the fastest.

Case 1:
If the upload chunk size is too big -> processing data takes long -> dashboard times out -> page refreshes & the upload stops!
Pros: upload is fast
Cons: processing gets choked & upload stops at a random points in time.
Case 2:
If upload chunk size is too small -> uploading takes too long -> high chance of network drops & browser timeout -> upload stops
Pros: processing is not choked
Cons: upload is too slow and often times out and the whole process stops.

Case Ideal:
Ideally, I would like..
Upload -> buffer data -> chunks to lines -> process -> visualize
So, Upload happens as fast as possible
And, Processing consumes data at its own pace thanks to the chunks to lines
Pros: Upload is fast & processing is done at server’s own pace
Cons: none!

Therefore in summary, I guess I would like the hunks-to-lines to pull data from the upload node as fast as possible but push the data in a line by line in the normal controlled manner (every time msg.tick = true).
Any tips how to achieve that?

Can't send file with same name and different content.

Hi

Amazing package! Using it daily, however, in my application, I'm always sending files with the same name, just changing the content, which requires a webpage refresh every time, because for some reason it doesn't let you send a file with the same name as the previous one but with a different content. In my case its .txt files with a few duzen characters.

You this get solder would be amazing, meanwhile, i'm looking for a way to make node-red automatically refresh the dashboard after each send.

Thank you.

Cheers, Luís Pereira.

file type filter

Hi, thanks for making this node public!

It is more of a question, rather that issue. How can I add file type/extension filter to the node configuration, so that only particular file type (e.g. *.CSV) could be selected in Open File dialog box?

Thanks

Write to file?

Is it possible to write the upload file to disk? I tried with the file node in append mode, but it truncate the end of the file :(

md-card width problem

Hello, My page is 14 wide but the ui-ipload md-card takes all the width and not only the desired one.
image

Dark theme issue

Hi, cool node ! But when using the dark theme, the "play" button disappears after chosing a file.

Using this node in a subflow crashes the node-red server

I've tried adding this node to a subflow and every time I do, once I click the upload button it crashes the server. Here is an example of the subflow I am using (but it should be reproducible any time you use it in a subflow):

[
    {
        "id": "ab143ae234639771",
        "type": "subflow",
        "name": "Subflow 5",
        "info": "",
        "in": [
            {
                "x": 60,
                "y": 80,
                "wires": [
                    {
                        "id": "4191a6f96fbbdef6"
                    }
                ]
            }
        ],
        "out": [
            {
                "x": 780,
                "y": 80,
                "wires": [
                    {
                        "id": "f6138561e5ba2cc8",
                        "port": 0
                    }
                ]
            }
        ]
    },
    {
        "id": "4191a6f96fbbdef6",
        "type": "ui_upload",
        "z": "ab143ae234639771",
        "group": "06fcee6ae387d692",
        "title": "upload",
        "name": "",
        "order": 3,
        "width": 0,
        "height": 5,
        "chunk": 256,
        "transfer": "binary",
        "x": 190,
        "y": 80,
        "wires": [
            [
                "c04ecb54869f1234"
            ]
        ]
    },
    {
        "id": "f6138561e5ba2cc8",
        "type": "function",
        "z": "ab143ae234639771",
        "name": "Allow access to certificate globally",
        "func": "global.set('certificate', msg.payload);\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 560,
        "y": 80,
        "wires": [
            []
        ]
    },
    {
        "id": "c04ecb54869f1234",
        "type": "json",
        "z": "ab143ae234639771",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": false,
        "x": 330,
        "y": 80,
        "wires": [
            [
                "f6138561e5ba2cc8"
            ]
        ]
    },
    {
        "id": "06fcee6ae387d692",
        "type": "ui_group",
        "name": "API input",
        "tab": "998ca7fbf9a4874e",
        "order": 1,
        "disp": true,
        "width": "10",
        "collapse": false,
        "className": ""
    },
    {
        "id": "998ca7fbf9a4874e",
        "type": "ui_tab",
        "name": "API input",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false
    },
    {
        "id": "9a9cca779db28544",
        "type": "subflow:ab143ae234639771",
        "z": "2a80dbf55bb7c970",
        "x": 680,
        "y": 280,
        "wires": [
            [
                "5af85ebf24ddfbf2"
            ]
        ]
    }
]

Here is the error that I get when it crashes:

19 May 10:24:48 - [error] TypeError: Cannot read properties of undefined (reading 'id')
    at /Users/eamon/work/node-red-api-flow/node_modules/node-red-contrib-ui-upload/ui_upload.js:341:74
    at findOutputNodeId (/Users/eamon/work/node-red-api-flow/node_modules/node-red-contrib-ui-upload/ui_upload.js:288:20)
    at Object.beforeSend (/Users/eamon/work/node-red-api-flow/node_modules/node-red-contrib-ui-upload/ui_upload.js:341:26)
    at EventEmitter.handler (/Users/eamon/work/node-red-api-flow/node_modules/node-red-dashboard/ui.js:288:22)
    at EventEmitter.emit (node:events:538:35)
    at Socket.emit (node:events:526:28)
    at Socket.emitUntyped (/Users/eamon/work/node-red-api-flow/node_modules/socket.io/dist/typed-events.js:69:22)
    at /Users/eamon/work/node-red-api-flow/node_modules/socket.io/dist/socket.js:466:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

I using Mac OS version 12.3.1, here are the versions of the nodes and node-red:

		"node-red": "^2.2.2",
		"node-red-contrib-ui-upload": "~0.5.1",
		"node-red-dashboard": "~3.1.7"

It works perfectly for me when used normally, but as soon as I use it in a subflow and try to upload something it crashes.

Chunk size less than 1kb

Hi, thanks for your great work!
I know that the chunk size is 1kb minimum but is there a way (by modifying ui.upload.js for example) to make the minimum chunk size in bytes instead of kilo bytes, like 128 or 256 bytes ?
Thanks! 😄

It messes up the whole dashboard pt2

Hello,
I saw the issue #5 ticked as solved, but actually I have the exact same problem:
before:
dashboard ok
after:
error upload node

I'm also using NSSM to use node-red like a service, with projects enabled.
I tried it with a fresh new installation 3.1.0 of node-red, with node 18.18.2 (as the MSSQL contrib I use doesn't support any newer).

I didn't get what the other guy meant with "redoing the complete directory node-modules works like a charm", any idea?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.