Comments (41)
With the last few changes in #1691 using Pyodide in Node.js with local files should work.
I'm looking for input on what you would like to see in the Pyodide npm package (related discussion in #328 and #1403), particularly when running Pyodide in Node.js.
-
We could install something like node-fetch, to add the missing fetch API in Node and share mostly the same code base for loading Pyodide and PyPi/micropip packages. The issue is unlike in the browser those downloads will then not be cached locally which is not good.
-
For instance in https://www.npmjs.com/package/@pyodide/pyodide all build artifacts are included in the node package. This has the advantage of not having to deal with version compatibility between packages and the npm library, however it results in quite large downloads. Which would in addition, I imagine, not be very useful for people integrating this npm package into a frontend app that would still use the CDN for packages. Also this does not address the question of caching PyPi packages in Node.
- 2b. A variation of the above is to upload each Pyodide package individually and let npm do the dependency resolution. It's something that we could explore in the future, but for now I don't think it would solve any of the issues we currently have.
-
The npm package contains just the JS+TypeScript code. When used in Node, downloads of Pyodide and PyPi packages are cached locally with npm.im/make-fetch-happen (or similar) and it's also possible to load those packages from the local file system. This would require dealing with version compatibility between Pyodide packages and the npm package though.
Feedback welcome. For now I'm considering going with option 3, unless there are objections.
The other thing that's not clear to me, is how to combine in the same NPM package the use case a) of frontend app development and producing some .js files with rollup b) running Pyodide in Node. Say we need some arbitrary package for running in Node, and add it to the dependencies. Is this potentially an issue for use case a) or it wouldn't matter?
cc @hoodmane @dmondev @teonbrooks @bollwyvl @jtpio @rebornix
from pyodide.
how to combine in the same NPM package the use case a) of frontend app development and producing some .js files with rollup b) running Pyodide in Node. Say we need some arbitrary package for running in Node, and add it to the dependencies
Yeah I would be interested to know the answer to this, my impression was that this is inconvenient but I have only ever used node for front end stuff.
Here is one possible solution:
https://stackoverflow.com/a/61283419/1942152
from pyodide.
For my highly-biased purposes (entirely informed by jupyterlite), I could care less whether pyodide
works with nodejs
... aside from:
- available via normal means e.g.
npmjs.com
and assorted mirrors - reproducibly built, to the extent possible (hard with custom toolchain)
- easily manageable with
yarn <2
- imports well into
typescript >=4.2
- works painlessly, is tree-shakeable with
webpack 5
I am, however, interested in being able to run these in wasmtime
and wasmer
backends, but haven't even begun to look into it.
from pyodide.
Okay I moved the vscode discussion over to #1712 (I don't think I can actually move peoples' comments so I just quoted them over).
from pyodide.
Not as far as I'm aware, but you could adapt the .circleci/config.yml for that, which lists necessary instructions to setup the enthronement with the circleci/python:3.7.0-stretch-browsers image.
Great!
Also I wonder if somewhere in emscripten code base there isn't already some abstraction layer to do part of that (e.g. it does to it for file system).
It is strange indeed that it does some environment check for the file system but not for the data.js
files.
these lines make sure you will have an error thrown if you're not in a browser
var PACKAGE_PATH;
if (typeof window === 'object') {
PACKAGE_PATH = window['encodeURIComponent'](window.location.pathname.toString().substring(0, window.location.pathname.toString().lastIndexOf('/')) + '/');
} else if (typeof location !== 'undefined') {
// worker
PACKAGE_PATH = encodeURIComponent(location.pathname.toString().substring(0, location.pathname.toString().lastIndexOf('/')) + '/');
} else {
throw 'using preloaded data can only be done on a web page, web worker or in nodejs';
}
A possible change would be something like this
var PACKAGE_PATH;
if (typeof window === 'object') {
PACKAGE_PATH = window['encodeURIComponent'](window.location.pathname.toString().substring(0, window.location.pathname.toString().lastIndexOf('/')) + '/');
console.warn('Browser environment'); // BROWSER
} else if (typeof location !== 'undefined') {
// worker
PACKAGE_PATH = encodeURIComponent(location.pathname.toString().substring(0, location.pathname.toString().lastIndexOf('/')) + '/');
console.warn('Web worker environment'); // WEB WORKER
} else if (typeof process === "object" && typeof require === "function") {
console.warn('Node environment'); // NODE
} else {
throw 'using preloaded data can only be done on a web page, web worker or in nodejs';
}
the above code is me already working on some changes for file_packager.py
.
from pyodide.
The most recent discussion about related stuff was in #1477. The strict mode issues in #1477 have been dealt with and I think we exclusively use globalThis
to access globals, so some of the basic issues have been addressed. We also have been working toward having less code with better code quality. (I think all of our Javascript code runs in strict mode right now, including all the emscripten glue code.)
That said, I don't think any of the current contributors has much knowledge of or experience with any of these platforms (node, deno, electron), and there is a limited amount of developer energy. So basically we've been resolving issues that are known blockers for node integration, but we haven't been directly trying to get it working ourselves.
An important step would be just to try to run Pyodide on your favored platform and open issues when you run into trouble. We are very open to including changes to increase the range of platforms supported. Once we have basic support, if we added at least some basic node tests to the CI it would upgrade node support from an afterthought to more of a first-class feature.
Is there any more information about this? Glad to jump in to help if there is.
I can answer specific questions about how the Pyodide runtime works. I'm not sure what sort of information you need. Help is very welcome.
from pyodide.
we've actually been working on migrating our Python requirement in our Electron app from native Python to Pyodide (makebrainwaves/BrainWaves#157). We've currently done this by downloading the entire versioned zip, and serve the files from a utils folder with the app (added to our webpack config).
our biggest requirements have been that:
- it works in a webworker
- integrate with our build process (based on the electron-react-boilerplate)
- uses yarn
- supports ts
for our package dependencies, we don't need all the supported packages but we haven't had a need to prune it.
related question: is there a way to serve wheels locally to micropip? I could imagine a scenario where i could require additional python packages for our app but wouldn't necessarily want to go through the build process to make its .js/.data
version for wasm.
cc: @jdpigeon can you think of anything else we would need from a node-compatible pyodide?
from pyodide.
Thanks for the feedback @bollwyvl @teonbrooks! It's helpful to have your input on these different use cases. We likely make an alpha release for the npm package in the near future, and it will be more clear then if there are any issues for your use cases.
from pyodide.
should not cause any issues except if the final Pyodide build for NodeJS uses native NodeJS modules
FWIW it is possible to use native node modules in VS Code but this gets troublesome for extensions if the platform specific binaries have to be built and bundled with the VS Code extension. If the native node modules in question use Node-API there's no issue.
if the Pyodide instance in VSCode can be shared via an API
As a first step, we can publish an extension which exposes the Pyodide instance via an API, and then other extensions can declare a dependency on that with extensionDependencies
in package.json, and access the API via https://code.visualstudio.com/api/references/vscode-api#Extension.exports
from pyodide.
@joyceerhl thanks for that--can we discuss further in a separate issue under vscode to reduce noise here?
(VSCode-relevant discussion below)
As a first step, we can publish an extension which exposes the Pyodide instance via an API, and then other extensions can declare a dependency on that with
extensionDependencies
in package.json, and access the API via https://code.visualstudio.com/api/references/vscode-api#Extension.exports
I wonder if there was any discussion of making extensionDependencies
an object with extension ID keys and version values? (like normal package.json
dependencies
).
from pyodide.
Hi @mdboom , i made an attempt to load pyodide.asm.wasm
on nodejs
but no success so far, i installed a nice npm package for the boilerplate npm install webassembly
the code looks like this
const path = require("path");
const wa = require("webassembly");
const wasmPath = path.join(__dirname, '/wasm/pyodide.asm.wasm');
const memory = new WebAssembly.Memory({ initial: 8192, maximum: 8192 });
const table = new WebAssembly.Table({ initial: 16384, element: "anyfunc" });
const importObj = { memory: memory, table: table };
wa.load(wasmPath, { imports: importObj }).then((module) => {
console.log(module.exports);
}).catch((err) => {
console.log(`some error: ${err}`);
});
i'm getting this error
LinkError: Import #4 module="env" function="DYNAMICTOP_PTR" error: global import must be a number
can't find a solution for this, not sure what's causing it, maybe you have some idea?
from pyodide.
I wouldn't expect this one to be easy, but it would be really cool if we can find a solution ;)
There are a bunch of things beyond just the Memory and Table that need to be set up to import the WebAssembly. In general, I wouldn't expect to be able to run emscripten-generated code directly like this. These variables are set up in pyodide.asm.js
, and get passed to us in the Module.instantiateWasm
callback in pyodide.js
.
In general, I would look at the code in pyodide.js
to see how all this gets bootstrapped and try to emulate that in the node environment.
Even then, I think there are probably some browser-specific things that emscripten is generating here that may break things in node. You may have to recompile pyodide with different settings. I would look at the emscripten docs for info on porting wasm from browser to node in general.
from pyodide.
Would you have the pyodide.asm.js
prettified version
i started adding stuff manually but i think it's too much stuff
const importObj = {
global: {
NaN: NaN,
Infinity: Infinity
},
env: {
memory: memory,
table: table,
tableBase: 0,
memoryBase: 0,
DYNAMICTOP_PTR: 2042576,
tempDoublePtr: 2041040,
STACKTOP: 2042592,
STACK_MAX: 7285472,
gb: 1024,
fb: 0,
abort: function(cb) {},
enlargeMemory: function(){},
getTotalMemory: function(){}
}
};
i could just copy from a pretty pyodide.asm.js
Edit: don't mind, i managed to prettify here
from pyodide.
did you write pyodide.asm.js
or it was generated?
from pyodide.
pyodide.asm.js
is generated by emscripten. It contains a lot of the emscripten
standard library and pyodide.asm.wasm
couldn't work without it.
from pyodide.
hmm ok
from pyodide.
i'm almost there
My code
const path = require('path');
const fs = require('fs');
const wa = require('webassembly');
const pyodidejs = require('./src/wasm/pyodide.asm.js');
const wasmPath = path.join(__dirname, '/src/wasm/pyodide.asm.wasm');
const baseURL = path.join(__dirname, '/src/wasm/');
let pyodide;
function loadLanguage() {
return new Promise((resolve, reject) => {
let Module = {};
fetch_node(wasmPath).then((buffer) => buffer.arrayBuffer()).then((arrayBuffer) => {
let wasm_promise = WebAssembly.compile(arrayBuffer);
Module.instantiateWasm = (info, receiveInstance) => {
wasm_promise
.then(module => WebAssembly.instantiate(module, info))
.then(instance => receiveInstance(instance));
return {};
};
Module.filePackagePrefixURL = baseURL;
Module.postRun = () => {
Module = null;
console.log('will resolve'); // never gets here
resolve();
};
pyodide = pyodidejs(Module);
})
});
}
async function init() {
await loadLanguage();
console.log(pyodide.runPython);
console.log('Python loaded');
}
function fetch_node(file) {
return new Promise((resolve, reject) =>
fs.readFile(file, (err, data) => err ? reject(err) : resolve({ arrayBuffer: () => data })));
}
init();
got this error
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00000000 (most recent call first):
exception thrown: abort(). Build with -s ASSERTIONS=1 for more info.
let me know if you have any ideas
from pyodide.
Great progress!
That looks like it's not finding the Python standard library (all the .py
files that make the Python interpreter work). That gets shipped to the browser (or node) in the pyodide.asm.data
file, with the metadata (what files go where) in pyodide.asm.data.js
. If I recall correctly, you have to load pyodide.asm.data.js
before pyodide.asm.js
or bad things happen.
from pyodide.
Hey @mdboom , there must be an option to generate pyodide.asm.js
/pyodide.asm.data.js
to target node.js
those files are loaded with methods that depends on the browser.
i went to the source to remove some error throw ''
's when there is no browser and override a lot of file system methods to be able to download the python standard library i.e FS.createPath() FS.createReadFile()
etc...
I managed to download the whole library but there is an internal method called addOnPreRun(cb)
that gets called in the end and loads the library files dynamically, this method receives a callback but this callback is not exposed in Modules
so i couldn't override it, the callback is using the browser fetch
method too and its not possible to override.
it's been a pretty long journey today but i believe i've overridden the maximum things i could to make some stuff work in NodeJS
. there must be an option some where to generate asm.js
and asm.data.js
that can work there, i don't know.
from pyodide.
This is how the code looks like so far
https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/test.js ...
to be able to download the python standard library files you need to uncomment
line 98 // Module.calledRun = true;
otherwise it will throw an error before getting to Module.postRun()
pyodide.asm.data.js
is a little bit different as well, just added support to NodeJS
https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/pyodide.asm.data.js
from pyodide.
Wooooooooooooooohhh i did it... i'll push to github and send you the link for the final code
from pyodide.
That's great news! It's really nice to have those details worked out.
Are you interested in working that into a pull request against Pyodide? We would need to find a way to work these changes into the build system -- either by applying the right compiler flags to emscripten to generate the right things, or possibly contributing changes to emscripten itself to emit cross-browser-and-node-compatible code.
from pyodide.
Yes, i still need to figure out how to load packages, but i can write python in nodejs
already, which is a BIG win.
Unfortunately i had to manually add nodejs
support inside pyodide.asm.data.js, same thing for numpy.js
(the .js
files for the packages have the same structure as pyodide.asm.data.js
, they're just there to load the dependencies).
Maybe there is a flag on emscripten to support nodejs
for asm.data.js
files.
pyodide.asm.js
is ok though, it supports multiple environments.
when i have everything tidy up and loadPackage
working i can put together a PR
by the way, have you seen my PR on iodide-example
, i wrote mic recording support on the notebook
from pyodide.
Awesome!
from pyodide.
It's working, i managed to load python
and numpy
in NodeJS.
The main class: https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/PyodideNode/PyodideNode.js
i test on: https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/test.js
That's awesome. Minor support for NodeJS
was required on pyodide.asm.data.js
and numpy.js
files, the optimal scenario would be to generate those files with this support out of the box, but in any case, it's not hard to do that by hand, just a few lines of code on the loadPackage
function and some other few lines in other places.
I thought we would need to download the whole python library but there is no need for that, i think it gets loaded in the webassembly memory, i'm still reading the docs to understand webassembly better, i'm very new to this technology but i think it's very impressive.
from pyodide.
Very nice work on this!
Maybe there is a flag on emscripten to support nodejs for asm.data.js files.
According to the emscripten docs, at least emcc should generate code that also runs on node by default, cf for instance, https://github.com/kripken/emscripten/blob/6cc4c664e4db2d0e4172c298d4ec88519bce1997/src/settings.js#L321-L328
As to pyodide.asm.data.js
, packaged/generated by file_packager.py
, I couldn't see anything in the code or its docs that says that it's not suitable for nodejs.
override a lot of file system methods to be able to download the python standard library i.e FS.createPath() FS.createReadFile() etc...
If I understand the docs correctly, in browser the MEMFS
system would be used while in node it would be NODEFS
. In the above PyodideNode.js
, there are things like,
var fs = require('fs');
which talks to the node FS directly, while IMO to be environment independent it should be done via emscripten. Looking at pyodide.asm.js
, there are things like,
if (ENVIRONMENT_IS_NODE) {
scriptDirectory = __dirname + "/";
var nodeFS;
var nodePath;
[...]
so some node specific code seems to be included.
You mentioned it before, but could you give an example of code that should be adapted for node? I couldn't find anything that looks like FS.createPath()
or FS.createReadFile()
mentioned above in the generated folder except for Module["FS_createPath"](...)
. Though I may be missing something and I know very little about node :)
from pyodide.
Hi @rth , thank you. pyodide.asm.data.js
comes only with support for browsers to download the library, the same goes for numpy.js
or any other package, on my repository if you take a look at pyodide.asm.data.js
i had to include Module
inside the node process process['Module'] = Module
and pass this Module to the var Module
in the 1st line.
var Module = typeof Module !== "undefined" ? Module : typeof process.Module !== 'undefined' ? process.Module : {};
there is no problem with pyodide.asm.js
, it comes with Nodejs support out of the box.
this is an example of code modification included in pyodide.asm.data.js
and numpy.js
var fetch = require('isomorphic-fetch'); // <-- for external resources
var loadPackage = function(metadata) {
var PACKAGE_PATH;
var PACKAGE_NAME = "build/pyodide.asm.data";
var REMOTE_PACKAGE_BASE = "pyodide.asm.data";
if (typeof Module["locateFilePackage"] === "function" && !Module["locateFile"]) {
Module["locateFile"] = Module["locateFilePackage"];
err("warning: you defined Module.locateFilePackage, that has been renamed to Module.locateFile (using your locateFilePackage for now)")
}
var REMOTE_PACKAGE_NAME = Module["locateFile"] ? Module["locateFile"](REMOTE_PACKAGE_BASE, "") : REMOTE_PACKAGE_BASE;
var REMOTE_PACKAGE_SIZE = metadata.remote_package_size;
var PACKAGE_UUID = metadata.package_uuid;
console.log(REMOTE_PACKAGE_NAME)
function fetchRemotePackage(packageName, packageSize, callback, errback) {
function fetch_node(file) { // <-- for local resources
var fs = require('fs');
return new Promise((resolve, reject) =>
fs.readFile(file, (err, data) => err ? reject(err) : resolve({ arrayBuffer: () => data })));
}
fetch(packageName).then((buffer) => buffer.buffer()).then((packageData) => {
if (!Module.dataFileDownloads) Module.dataFileDownloads = {};
Module.dataFileDownloads[packageName] = {
loaded: packageSize,
total: packageSize
}
var total = 0;
var loaded = 0;
var num = 0;
for (var download in Module.dataFileDownloads) {
var data = Module.dataFileDownloads[download];
total += data.total;
loaded += data.loaded;
num++
}
total = Math.ceil(total * Module.expectedDataFileDownloads / num);
console.log(`Downloaded ${packageName} data... (${total}/${total})`);
callback(packageData);
if (Module["setStatus"]) Module["setStatus"]("Downloading data... (" + total + "/" + total + ")");
}).catch((err) => {
console.error(`Something wrong happened ${err}`);
throw new Error(`Something wrong happened ${err}`);
});
}
.
.
.
.
i had to replace some arrayBuffer
checking because the one that existed before was just not working!
function processPackageData(arrayBuffer) {
if(!arrayBuffer) throw "No input to processPackageData";
Module.finishedDataFileDownloads++;
if(!arrayBuffer instanceof ArrayBuffer) arrayBuffer = arrayBuffer.buffer ? arrayBuffer.buffer : null; // <- this is better
if(!arrayBuffer) throw "bad input to processPackageData";
i believe this is all i had to by hand to make pyodide work on Node.js, of course it looks easy now but i spent hours and hours debugging those modules to figure out what had to be done hehe, i almost gave up =D
Conclusion
i believe that var loadPackage
should have nodejs support out of the box, today it throws a big red error if you're not in a browser, i believe this should be enought
from pyodide.
Thanks for your response @gabrielfreire !
The question is how we can integrate your valuable contributions into pyodide :)
As far as I understand the code in numpy.js
etc is generated by emscripten's file_packager.py
: essentially it does some string templating (cf. e.g. here). So to do the changes you propose automatically we would need to patch that file. To avoid rebuilding emsdk for each modification, a first solution could be to just copy file_packager.py
from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py
,
Makefile:FILEPACKAGER=$(PYODIDE_ROOT)/emsdk/emsdk/emscripten/tag-1.38.10/tools/file_packager.py
tools/buildpkg.py: Path(os.environ['EMSCRIPTEN']) / 'tools' / 'file_packager.py',
Once we have this working, we can propose these modifications upstream in emscripten (or possibly it might be useful to open an issue in emscripten to start with). There is a number of open issues about node js in emscripten github repo (cf e.g. emscripten-core/emscripten#5237).
I think a first step might be to just use load packages from some remote locations same as it happens in the browser.
I was also wondering how to test this. One possibility could be to create another Selenium instance alongside Chrome/Firefox here, that would interact with Node JS via the Selenium Webdriver API. This would allow to run all tests also in NodeJS without much effort. However, I'm not sure this can work in practice: one possibility I saw for Node+Webdriver API was webdriver.io but I'm not sure how to call it from the Selenium-Python package (cf its documentation). Unless testing in that way would defeat the purpose of running in Node.js?
from pyodide.
a first solution could be to copy file_packager.py from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py,
@gabrielfreire Would you be interested in opening a PR for it? (If you think this is the right approach to apply your changes automatically).
from pyodide.
a first solution could be to just copy file_packager.py from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py,
I really like this solution, i believe it is better to have another file_packager.py
for pyodide's needs since maybe supporting Nodejs might not be on emsdk's Road map, plus they are very packed with issues and PRs.
I think a first step might be to just use load packages from some remote locations same as it happens in the browser.
Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js
browser code for node.js code, which is basically what i did.
This would allow to run all tests also in NodeJS without much effort
i wasn't familiar on how you guys wrote the tests, i took a quick look and it's all python using selenium, if you want to re-use those tests for node.js i'm not sure how you would call it from python, the first thing that comes to my mind is ZeroMQ or RabbitMQ but i believe it should be pretty easy to re-write in pure node.js using karma
, mocha
or something like that.
Would you be interested in opening a PR for it?
Well, i was working with files generated by @mdboom so i don't have 'pyodide' or any emsdk
pre-requisite setup, i think i'll try to setup everything in a Docker
machine, for this project, should be easy. Let me know if you already have a Docker
configuration for this.
Once i have everything setup i'll apply the changes and generate something to see if it works well, then i open a PR.
from pyodide.
Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js browser code for node.js code, which is basically what i did.
i think you mean, AFTER the new file_packager.py
, right ? hehe... yes i agree
from pyodide.
Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js browser code for node.js code, which is basically what i did.
I guess I was hoping we could arrive at a data.js
that might work in both cases (possibly by detecting some environment variables specific to Node JS and adding some code specific to that case). Also I wonder if somewhere in emscripten code base there isn't already some abstraction layer to do part of that (e.g. it does to it for file system).
Let me know if you already have a Docker configuration for this
Not as far as I'm aware, but you could adapt the .circleci/config.yml
for that, which lists necessary instructions to setup the environement with the circleci/python:3.7.0-stretch-browsers
image.
from pyodide.
@rth , i'm getting a strange error with the new pyodide.asm.js
, the one i have locally works well
exception thrown: ReferenceError: window is not defined,ReferenceError: window is not defined
at _hiwire_get_global (C:\Users\gabriel.freire\Documents\workspace\neuralnet.js\PyodideNode\pyodide.asm.js:1552:36)
at wasm-function[312]:290
at wasm-function[10266]:10
at wasm-function[488]:348
at wasm-function[2631]:32923
at wasm-function[14870]:7
at wasm-function[2626]:3888
at wasm-function[2625]:31
at wasm-function[2587]:676
at wasm-function[11339]:7
It gets fixed when i change this function
function _hiwire_get_global(idname) {
var jsname = UTF8ToString(idname);
return Module.hiwire_new_value(window[jsname])
}
to
function _hiwire_get_global(idname) {
var jsname = UTF8ToString(idname);
return Module.hiwire_new_value(jsname)
}
but this change could brake something else so i don't think this is the problem because the old pyodide.asm.js
has the same function
from pyodide.
This is great work!
I think it makes sense to have a local copy of file_packager.py
for now, and try to update it so the generated code works in both the browser and node. I think we definitely want to have "universal" builds that would work in all contexts.
I can help get that merged back upstream into emscripten if there are any delays.
As for the references to window
: I think we'll just need an abstraction that uses window
in the browser and global
in Node. That should keep things explicit and hopefully not break anything.
As for testing, ideally we'd find a way to use the same tests in all contexts. We could try writing some tests in a Node-based test harness (rather than py.test) so we could run code in-process in Node itself, and then use webdriver.io
to run the same code in the browsers. If that works, we would then have to port all of the tests to the new style. An alternative is to stick with what we have, but call out to Node -- you shouldn't have to use anything fancy like zeromq -- node can be communicated with through Unix pipes.
from pyodide.
What happened to this exciting development? Is running pyodide under nodejs possible/supported these days?
from pyodide.
See the latest status in #183 (comment) which is still up to date.
from pyodide.
#183 is now closed
Nodejs support would be a huge plus for many electron/node apps depending on Python applications--examples include VSCode extensions that no longer need to bundle a whole Python installation and what not to enable features.
Is there any more information about this? Glad to jump in to help if there is.
from pyodide.
Hi, we've been investigating how to make Pyodide and VS Code work, and allowing Pyodide to run in Node.js would be very helpful since VS Code is based on Node. It would be great to have the the ability to use Pyodide with VS Code through Node support, not only for us but also for our users. Thanks!
from pyodide.
@eleanorjboyd just a heads up that VSCode is Electron (which bundles Node); this (I assume) should not cause any issues except if the final Pyodide build for NodeJS uses native NodeJS modules (which should not be the case, otherwise Pyodide shouldn't run on Web at all!)
(VSCode-relevant discussion below)
I however wonder if there is separate discussion for bundling Pyodide in VSCode and exposing it in some sort of API to extension devs (since Pyodide is large). I currently maintain a VSCode extension that depends on Python as well, so if the Pyodide instance in VSCode can be shared via an API, it would be ideal. Another problem is that some Python-dependent extensions are dependent on the Python versions used (for example the official Python extension asks for the path to the Python executable used for the current project)--how would this be resolved if Pyodide was used?
from pyodide.
I am closing this as resolved. We have been running the full test suite in node as part of the CI for months now. Please feel free to open a new issue if your use case isn't supported.
from pyodide.
This is also being tested nightly against CPython main
: https://github.com/ethanhs/python-wasm/actions/workflows/ci.yml?query=branch%3Amain
from pyodide.
Related Issues (20)
- Debug pyodide release artifacts (and pyodide build debug support)
- "Building and testing Python packages out of tree" commands need some dev tools installed as prerequisites HOT 2
- Add esbuild-py HOT 4
- Add `PyMuPDF` package
- runPython return missing
- cmake -E capabilities is broken HOT 1
- Release 0.26.0 HOT 5
- datasets (huggingface) HOT 6
- Cannot create a WebGL2 context when using SDL HOT 3
- request to add pandasai HOT 1
- request to add llama-index
- request to add langchain HOT 1
- request to add Boto3 HOT 4
- fastobo
- loadPackage() not to access the file system if the cache already exists HOT 2
- cvxpy-base 1.4.3 HOT 2
- html5_canvas_backend no longer works. HOT 1
- RFC Package unvendored stdlibs to sdist / wheel? HOT 3
- `make` in docker container `pyodide/pyodide-env` fails with `could not create work tree dir 'emsdk'` HOT 8
- DOM types in pyodide lead to build issues with Angular and typescript. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyodide.