dcodeio / node-memcpy Goto Github PK

View Code? Open in Web Editor NEW

54.0 4.0 11.0 216 KB

Copies data between node Buffers and/or ArrayBuffers up to ~75 times faster than in pure JS.

Home Page: http://dcode.io

License: Apache License 2.0

Python 1.61% C++ 41.13% JavaScript 56.89% C 0.37%

node-memcpy's Introduction

node-memcpy

Copies data between node Buffers and/or ArrayBuffers up to ~75 times faster than in pure JS.

Wait, what?

node.js utilizes a non-standard concept of I/O buffers and thus has both its Buffer as well as ArrayBuffer support. While Buffers are nice because they are a lot faster than V8's ArrayBuffers, transferring data between those two types can be ridiculously slow. This is where a node module like memcpy comes into play.

// C++ binding

 i memcpy.100k > cc Buffer -> Buffer: 22.756ms
 i memcpy.100k > cc Buffer -> ArrayBuffer: 23.861ms

 i memcpy.100k > cc ArrayBuffer -> Buffer: 22.955ms
 i memcpy.100k > cc ArrayBuffer -> ArrayBuffer: 23.273ms

// Native JS

 i memcpy.100k > js Buffer -> Buffer: 21.617ms
 i memcpy.100k > js Buffer -> ArrayBuffer: 993.361ms

 i memcpy.100k > js ArrayBuffer -> Buffer: 940.273ms
 i memcpy.100k > js ArrayBuffer -> ArrayBuffer: 1626.182ms

API

memcpy(target[, targetStart=0], source[, sourceStart=0[, sourceEnd=source.length]):bytesCopied

Argument	Type	Optional	Description
target	Buffer \| ArrayBuffer		Target buffer to copy to
targetStart	number	omittable	Target offset to begin copying to, defaults to `0`
source	Buffer \| ArrayBuffer		Source buffer to copy from
sourceStart	number	optional	Source offset to begin copying from, defaults to `0`
sourceEnd	number	optional	Source offset to end copying from, defaults ot `source.length`
@returns	number		Number of bytes copied

Source and target regions may overlap.

Usage

npm install memcpy

var memcpy = require("memcpy"), // C++ binding if available, else native JS
    memcpy_binding = memcpy.binding, // C++ binding or NULL if not available
    memcpy_native = memcpy.native; // Native JS
...

Please keep in mind that - besides the nice numbers - this is still to be considered experimental. I'd love if you'd review the C++ code to validate that it's safe. I can't yet think of a sane use case, though, as just sticking with Buffers on node.js and ArrayBuffers in the browser should be best practice.

Contributors

Denys Khanzhiyev (node 0.11.13+ support with nan)

License: Apache License, Version 2.0

node-memcpy's People

Contributors

Stargazers

Watchers

Forkers

xdenser hdachev mcanthony prejr-dev nyxtom pps83 13812453806 mrmugiwara robertjchristian 5l1v3r1 chriss-0x01

node-memcpy's Issues

npm install --save memcpy Error

$ npm install --save memcpy

[email protected] install /Users/haoyin/workspace/code/idorp/ozj/test/angular2-seed/node_modules/memcpy
node-gyp configure build

CXX(target) Release/obj.target/memcpy/src/memcpy.o
In file included from ../src/memcpy.cc:18:
../node_modules/nan/nan.h:342:68: error: too many arguments to function call, expected at most 2, have 4
return v8::Signature::New(v8::Isolate::GetCurrent(), receiver, argc, argv);
~~~~~~~~~~~~~~~~~~ ^~~~~~~~~~
/Users/haoyin/.node-gyp/6.3.1/include/node/v8.h:4798:3: note: 'New' declared here
static Local New(
^
In file included from ../src/memcpy.cc:18:
../node_modules/nan/nan.h:424:3: error: redefinition of 'NanNew'
NanNew<v8::StringObject, v8::Handlev8::String >(
^
../node_modules/nan/nan.h:417:3: note: previous definition is here
NanNew<v8::StringObject, v8::Localv8::String >(
^
../node_modules/nan/nan.h:442:36: error: redefinition of 'NanNew'
NAN_INLINE v8::Localv8::RegExp NanNew(
^
../node_modules/nan/nan.h:436:36: note: previous definition is here
NAN_INLINE v8::Localv8::RegExp NanNew(
^
../node_modules/nan/nan.h:454:36: error: redefinition of 'NanNew'
NAN_INLINE v8::Localv8::RegExp NanNew(
^
../node_modules/nan/nan.h:448:36: note: previous definition is here
NAN_INLINE v8::Localv8::RegExp NanNew(
^
../node_modules/nan/nan.h:623:19: error: no type named 'ExternalAsciiStringResource' in 'v8::String'; did you mean 'ExternalStringResource'?
v8::String::ExternalAsciiStringResource *resource) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
ExternalStringResource
/Users/haoyin/.node-gyp/6.3.1/include/node/v8.h:2230:19: note: 'ExternalStringResource' declared here
class V8_EXPORT ExternalStringResource
^
In file included from ../src/memcpy.cc:18:
../node_modules/nan/nan.h:622:36: error: redefinition of 'NanNew'
NAN_INLINE v8::Localv8::String NanNew(
^
../node_modules/nan/nan.h:617:36: note: previous definition is here
NAN_INLINE v8::Localv8::String NanNew(
^
../node_modules/nan/nan.h:637:27: error: redefinition of '_NanEscapeScopeHelper'
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Local val) {
^
../node_modules/nan/nan.h:632:27: note: previous definition is here
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Handle val) {
^
../node_modules/nan/nan.h:653:12: error: no matching function for call to '_NanEscapeScopeHelper'
return NanEscapeScope(NanNew(v8::Undefined(v8::Isolate::GetCurrent())));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../node_modules/nan/nan.h:641:43: note: expanded from macro 'NanEscapeScope'

define NanEscapeScope(val) scope.Escape(_NanEscapeScopeHelper(val))

                                      ^~~~~~~~~~~~~~~~~~~~~

../node_modules/nan/nan.h:637:27: note: candidate template ignored: substitution failure [with T = v8::Primitive]
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Local val) {
^
../node_modules/nan/nan.h:658:12: error: no matching function for call to '_NanEscapeScopeHelper'
return NanEscapeScope(NanNew(v8::Null(v8::Isolate::GetCurrent())));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../node_modules/nan/nan.h:641:43: note: expanded from macro 'NanEscapeScope'

define NanEscapeScope(val) scope.Escape(_NanEscapeScopeHelper(val))

                                      ^~~~~~~~~~~~~~~~~~~~~

../node_modules/nan/nan.h:637:27: note: candidate template ignored: substitution failure [with T = v8::Primitive]
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Local val) {
^
../node_modules/nan/nan.h:663:12: error: no matching function for call to '_NanEscapeScopeHelper'
return NanEscapeScope(NanNew(v8::True(v8::Isolate::GetCurrent())));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../node_modules/nan/nan.h:641:43: note: expanded from macro 'NanEscapeScope'

define NanEscapeScope(val) scope.Escape(_NanEscapeScopeHelper(val))

                                      ^~~~~~~~~~~~~~~~~~~~~

../node_modules/nan/nan.h:637:27: note: candidate template ignored: substitution failure [with T = v8::Boolean]
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Local val) {
^
../node_modules/nan/nan.h:668:12: error: no matching function for call to '_NanEscapeScopeHelper'
return NanEscapeScope(NanNew(v8::False(v8::Isolate::GetCurrent())));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../node_modules/nan/nan.h:641:43: note: expanded from macro 'NanEscapeScope'

define NanEscapeScope(val) scope.Escape(_NanEscapeScopeHelper(val))

                                      ^~~~~~~~~~~~~~~~~~~~~

../node_modules/nan/nan.h:637:27: note: candidate template ignored: substitution failure [with T = v8::Boolean]
NAN_INLINE v8::Local _NanEscapeScopeHelper(v8::Local val) {
^
../node_modules/nan/nan.h:701:20: error: no type named 'GCEpilogueCallback' in 'v8::Isolate'
v8::Isolate::GCEpilogueCallback callback
~~~~~~~~~~~~~^
../node_modules/nan/nan.h:707:20: error: no type named 'GCEpilogueCallback' in 'v8::Isolate'
v8::Isolate::GCEpilogueCallback callback) {
~~~~~~~~~~~~~^
../node_modules/nan/nan.h:712:20: error: no type named 'GCPrologueCallback' in 'v8::Isolate'
v8::Isolate::GCPrologueCallback callback
~~~~~~~~~~~~~^
../node_modules/nan/nan.h:718:20: error: no type named 'GCPrologueCallback' in 'v8::Isolate'
v8::Isolate::GCPrologueCallback callback) {
~~~~~~~~~~~~~^
../node_modules/nan/nan.h:889:13: error: no member named 'smalloc' in namespace 'node'
, node::smalloc::FreeCallback callback
~~~~~~^
../node_modules/nan/nan.h:900:12: error: no matching function for call to 'New'
return node::Buffer::New(v8::Isolate::GetCurrent(), data, size);
^~~~~~~~~~~~~~~~~
/Users/haoyin/.node-gyp/6.3.1/include/node/node_buffer.h:46:40: note: candidate function not viable: 2nd argument ('const char ') would lose const qualifier
NODE_EXTERN v8::MaybeLocalv8::Object New(v8::Isolate isolate,
^
/Users/haoyin/.node-gyp/6.3.1/include/node/node_buffer.h:34:40: note: candidate function not viable: no known conversion from 'const char ' to 'v8::Localv8::String' for 2nd argument
NODE_EXTERN v8::MaybeLocalv8::Object New(v8::Isolate isolate,
^
/Users/haoyin/.node-gyp/6.3.1/include/node/node_buffer.h:31:40: note: candidate function not viable: requires 2 arguments, but 3 were provided
NODE_EXTERN v8::MaybeLocalv8::Object New(v8::Isolate* isolate, size_t length);
^
/Users/haoyin/.node-gyp/6.3.1/include/node/node_buffer.h:39:40: note: candidate function not viable: requires 5 arguments, but 3 were provided
NODE_EXTERN v8::MaybeLocalv8::Object New(v8::Isolate* isolate,
^
In file included from ../src/memcpy.cc:18:
../node_modules/nan/nan.h:904:12: error: no viable conversion from returned value of type 'v8::MaybeLocalv8::Object' to function return type 'v8::Localv8::Object'
return node::Buffer::New(v8::Isolate::GetCurrent(), size);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/haoyin/.node-gyp/6.3.1/include/node/v8.h:216:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'v8::MaybeLocalv8::Object' to 'const v8::Localv8::Object &' for 1st argument
class Local {
^
/Users/haoyin/.node-gyp/6.3.1/include/node/v8.h:216:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'v8::MaybeLocalv8::Object' to 'v8::Localv8::Object &&' for 1st argument
/Users/haoyin/.node-gyp/6.3.1/include/node/v8.h:220:13: note: candidate template ignored: could not match 'Local' against 'MaybeLocal'
V8_INLINE Local(Local that)
^
In file included from ../src/memcpy.cc:18:
../node_modules/nan/nan.h:911:26: error: no member named 'Use' in namespace 'node::Buffer'
return node::Buffer::Use(v8::Isolate::GetCurrent(), data, size);
~~~~~~~~~~~~~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Release/obj.target/memcpy/src/memcpy.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack at emitTwo (events.js:106:13)
gyp ERR! stack at ChildProcess.emit (events.js:191:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "build"
gyp ERR! cwd /Users/haoyin/workspace/code/idorp/ozj/test/angular2-seed/node_modules/memcpy
gyp ERR! node -v v6.3.1
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm ERR! Darwin 15.4.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--save" "memcpy"
npm ERR! node v6.3.1
npm ERR! npm v3.10.3
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: node-gyp configure build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp configure build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the memcpy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp configure build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs memcpy
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls memcpy
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /Users/haoyin/workspace/code/idorp/ozj/test/angular2-seed/npm-debug.log

Node 0.12: Not a valid kExternalUnsignedByteArray

Hi,
I get a TypeError: Illegal source: Not a valid kExternalUnsignedByteArray error when using node 0.12?

var memcpy = require("memcpy"); 
var ab = new ArrayBuffer(8);
var buffer = new Buffer(ab.byteLength);
memcpy(buffer, ab);

Function to rewrap memory without memmove?

First off, I'm glad this library exists, thanks. Conversions between Buffer and ArrayBuffer are frustrating.

I just made a modification to simply point the target's underlying ArrayData to the source's underlying data:

target->SetIndexedPropertiesToExternalArrayData(
    source->GetIndexedPropertiesExternalArrayData(),
    v8::kExternalUnsignedByteArray,
    len
);

Thoughts?

Your memmove method is fast; this is instant.
There could be unexpected consequences if both the Buffer and ArrayBuffer are manipulated after the call, and the user expects them to be independent. However, this is exactly how typed array views work and I think it's manageable.

I'd be happy to make a PR for a new function, but would want your syntax preference given that this library has a single exported function. (1) New argument like copy (default true), which determines whether memmove or the pointer assignment is performed. (2) Separate library, single function like this. (3) Like your memcpy.binding and memcpy.native -- var membind = require("memcpy").membind or something.

Incompatible with node > 0.12

This package is great, but cannot be installed on node versions after 0.12. Do you have any plans to update this to be compatible with newer versions?

The javascript code is still valid on later versions, but since the build fails the module isn't added and therefore cannot be used.

Do you have dependencies on the python verson?

If I have python 2.6.9 or less, npm install memcpy will fail. Please see error attached.
if I have python 2.7x , npm install memcpy will not encounter the follow issue.
You may want to update the readme on requirements if this is an indirect dependencies.
Thank you.

[testMemCpy]$ npm install memcpy

[email protected] install /usr/bot/testMemCpy/node_modules/memcpy
node-gyp configure build

Traceback (most recent call last):
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 18, in
sys.exit(gyp.script_main())
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 534, in script_main
return main(sys.argv[1:])
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 527, in main
return gyp_main(args)
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 503, in gyp_main
options.circular_check)
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 129, in Load
params['parallel'], params['root_targets'])
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2692, in Load
variables, includes, depth, check, True)
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 362, in LoadTargetBuildFile
includes, True, check)
File "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 219, in LoadOneBuildFile
None)
File "binding.gyp", line 1
{
^
SyntaxError: invalid syntax
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:462:16)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:810:12)
gyp ERR! System Linux 2.6.32-431.17.1.el6.s390x
gyp ERR! command "node" "/usr/bot/T64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "build"
gyp ERR! cwd /usr/bot/testMemCpy/node_modules/memcpy
gyp ERR! node -v v0.10.32
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok

npm ERR! [email protected] install: node-gyp configure build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the memcpy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp configure build
npm ERR! You can get their info via:
npm ERR! npm owner ls memcpy
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 2.6.32-431.17.1.el6.s390x
npm ERR! command "/usr/bot/T64/bin/node" "/usr/bot/T64/bin/npm" "install" "memcpy"
npm ERR! cwd /usr/bot/testMemCpy
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

npm install memcpy fail

Installing this package causes this error

gyp ERR! configure error
gyp ERR! stack Error: Command failed: C:\Users\_____\AppData\Local\Programs\Python\Python37-32\python.EXE -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack                                ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:294:12)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "build"

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.