Bug: [email protected] nodejs installation fails about usearch HOT 69 OPEN

I guess I was overzealous on files[] part of the package.json. Will fix.

from usearch.

See #383

from usearch.

These don't include prebuilt binaries (you won't find a .node file among them):

https://www.npmjs.com/package/better-sqlite3?activeTab=code
https://www.npmjs.com/package/duckdb?activeTab=code
https://www.npmjs.com/package/hnswlib-node?activeTab=code
https://www.npmjs.com/package/@tensorflow/tfjs-node?activeTab=code

As opposed to usearch:

https://www.npmjs.com/package/usearch?activeTab=code

Frustrating.

Anyhow, I will look at ncc source code for ncc and see what they are doing for .node files.

from usearch.

Oh, they are the same size, so definitely not right.

from usearch.

bash-5.2# npm install @sroussey/usearch

added 5 packages, and audited 6 packages in 6s

found 0 vulnerabilities

The install passed! Haven’t tried running sample code yet though

from usearch.

I’m home a bit today, so I’ll comment out that section until I have a working version.

from usearch.

Works fine on MacOS though.

from usearch.

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 60 --slave /usr/bin/g++ g++ /usr/bin/g++-12 fixed it, thanks.

from usearch.

At any rate, I don't use aws lambda, but if you give me some steps to reproduce locally, I'll have a look.

from usearch.

Sure, the easiest way to locally reproduce without getting docker involved is:

mkdir usearch-bundle
cd usearch-bundle
npm init -y
npm install [email protected]

To get the entrypoint, run node -p "require.resolve('usearch')" or just run the commands below:

Then, try bundling with @vercel/ncc (using --external doesn’t make sense in this case, because our goal is to bundle usearch, not skip it):

npx --yes @vercel/ncc build node_modules/usearch/javascript/dist/cjs/usearch.js -o ncc-usearch-bundled --debug

or with esbuild:

npx --yes esbuild node_modules/usearch/javascript/dist/cjs/usearch.js --bundle --outdir=esbuild-usearch-bundled --loader:.node=file --platform=node

Bundling works great with many other native addons that have prebuilds, such as:

• better-sqlite3

	let addon;
	if (nativeBinding == null) {
		addon = DEFAULT_ADDON || (DEFAULT_ADDON = require('bindings')('better_sqlite3.node'));
	} else if (typeof nativeBinding === 'string') {
		// See <https://webpack.js.org/api/module-variables/#__non_webpack_require__-webpack-specific>
		const requireFunc = typeof __non_webpack_require__ === 'function' ? __non_webpack_require__ : require;
		addon = requireFunc(path.resolve(nativeBinding).replace(/(\.node)?$/, '.node'));
	} else {
		// See <https://github.com/WiseLibs/better-sqlite3/issues/972>
		addon = nativeBinding;
	}

• duckdb

var binary = require('@mapbox/node-pre-gyp');
var path = require('path');
var binding_path = binary.find(path.resolve(path.join(__dirname,'../package.json')));
var binding = require(binding_path);

module.exports = exports = binding;

• hnswlib-node

const addon = require('bindings')('addon');
module.exports = addon;

• @tensorflow/tfjs-node
• And many more…

It should be possible to add prebuilds without breaking bundling. @sroussey hopefully some of those package examples can be inspiration. I’ll look at it more, but I’m not familiar enough with prebuilds and node-gyp to immediately see what the solution is.

from usearch.

Great! I'll have a look.

Hopefully one of them uses prebuildify. I think the sqlite one (based on my poor memory) relies on a lifecycle script to download a prebuilt binary, and for security reasons people are moving away from having some deep dependency run arbitrary code on install on your machine (and another reason for dev containers). Bun, for example, does not run them.

I remember having issues since typescript doesn't have __dirname etc. I know some better ways around that now however.

from usearch.

Those can cross compile, so faster to do it that way.

from usearch.

Also, I don't think GitHub has arm64 for those.

from usearch.

cosmetic changes

Maybe a red herring? It uses windows-latest so maybe that changed instead?

Looking at the changeset, I cannot see why the error would come up. You might try branching that off and do a force push of the last known good and see if that now fails.

from usearch.

I try to let the bundlers do that, because I’ve had problems with other libraries where the paths change due to bundling and it’s better just to let them handle it.

Yeah, looks like ncc will change it to be relative to the combined js file. So the copy of the prebuilds folder is adjacent to the index.js file (which just happens to be where we expect it to be).

from usearch.

You could disable for windows which should cause everyone to rebuild on their machine on that platform.

If they don't allow lifecycle scripts it will fail, same with bun, etc. They will need to make changes to accommodate.

I'm not a big windows build expert but I'll have another look this weekend. I'm curious if other people are having windows build issues on GitHub actions in 2024. What was the date it started failing? Might be a clue.

from usearch.

That's good to know! Another related issue I am solving is the lack of Doxygen support for TypeScript. Maybe we should provide both binding.gyp and prebuilt, as well as TypeScript + JavaScript?

from usearch.

It is odd since there should be a pre-built binary for linux-arm64

https://www.npmjs.com/package/usearch?activeTab=code

from usearch.

I did cross compile it so maybe i did something wrong

from usearch.

See #379

from usearch.

🎉 This issue has been resolved in version 2.10.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

from usearch.

I will try it as soon as 2.10.4 is released on npm 😄

from usearch.

@jlarmstrongiv, sadly, the build fails. @sroussey, would you be able to double-check the cross-compilation commands? I am also looking to extend cross-compilation in Rust as well (for #378).

from usearch.

I will try it as soon as 2.10.4 is released on npm 😄

Try npm install @sroussey/usearch

as a test.

from usearch.

@jlarmstrongiv, sadly, the build fails. @sroussey, would you be able to double-check the cross-compilation commands? I am also looking to extend cross-compilation in Rust as well (for #378).

Might remove the cross compile for now while I do more investigation.

from usearch.

How is it going, @sroussey? How should I change the CI for builds to pass?

from usearch.

See c6fb875

It has what i think is the fix, but I am not setup to test it.

So farther down I have it skip the cross compiling.

from usearch.

I have a docker setup on x86_64 but I can't get it to compile normally, so I can't test cross compiling.

@ashvardanian Have any idea how i set things up wrong?

from usearch.

from usearch.

root@6a1e9ab171d0:/usr/usearch# uname -a 
Linux 6a1e9ab171d0 6.4.16-linuxkit #1 SMP PREEMPT Thu Nov 16 10:49:20 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

apt install -y cmake build-essential libjemalloc-dev libomp-dev gcc-12 g++-12 
apt install -y gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu g++-aarch64-linux-gnu

root@6a1e9ab171d0:/usr/usearch# git submodule
 0a92994d729ff76a58f692d3028ca1b64b145d91 fp16 (heads/master)
 127ead1da7c39957b30a50dd85e74814edb022d6 simsimd (v4.2.2)
 c2848c1ba777c541af2903501f3d6344d711638b stringzilla (v3.7.0)

from usearch.

The _ph suffixes are for half-precision. GCC 12 must support it. You download it, but seems like you are not using it, @sroussey 🤷

from usearch.

cross compile says it goes fine, but it doesn't contain a build

from usearch.

Oh, it did build it:

root@6a1e9ab171d0:/usr/usearch# file build/Release/obj.target/usearch.node
build/Release/obj.target/usearch.node: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=c0650f24bd82a89cca3a4a773455913f2477a789, not stripped

It just didn't get moved over. I wonder if a strip issue.

from usearch.

Indeed, something went wrong with strip. Manually works though. I can work around this...

from usearch.

@jlarmstrongiv is the issue resolved now?

from usearch.

Hmm, I get errors that:

Error: src/usearch.ts(11,5): error TS2322: Type '"cos"' is not assignable to type 'MetricKind'.
Error: src/usearch.ts(13,5): error TS2322: Type '"f32"' is not assignable to type 'ScalarKind'.

Did the types change?

from usearch.

Fixed the types… testing

from usearch.

It seems my bundling for usearch does not work with the new package structure. I am using https://www.npmjs.com/package/@vercel/ncc, I wonder why that would break and not detect the usearch.node file anymore

Uncaught Exception	{
  "errorType": "Error",
  "errorMessage": "Could not find native build for usearch",
  "stack": [
    "Error: Could not find native build for usearch",
    "    at getBuildDir (/opt/nodejs/node_modules/usearch/index.js:1080:15)",
    "    at getBuildDir (/opt/nodejs/node_modules/usearch/index.js:1081:12)",
    "    at getBuildDir (/opt/nodejs/node_modules/usearch/index.js:1081:12)",
    "    at getBuildDir (/opt/nodejs/node_modules/usearch/index.js:1081:12)",
    "    at getBuildDir (/opt/nodejs/node_modules/usearch/index.js:1081:12)",
    "    at 128 (/opt/nodejs/node_modules/usearch/index.js:605:48)",
    "    at __nccwpck_require__ (/opt/nodejs/node_modules/usearch/index.js:1117:43)",
    "    at /opt/nodejs/node_modules/usearch/index.js:1137:37",
    "    at Object.<anonymous> (/opt/nodejs/node_modules/usearch/index.js:1140:12)",
    "    at Module._compile (node:internal/modules/cjs/loader:1376:14)"
  ]
}

from usearch.

These tools are not smart enough to deal with .node or .wasm files (generally). They will require extra steps on your side.

from usearch.

Oh, and if it has something like "external" you might use that. Bun can make a single file as well, and it has something like that. I use externals for transformers.js since it has wasm files.

from usearch.

@sroussey 64-bit Arm is a popular platform. We should make sure it works fine there. Any other ways we can improve our cross-compilation scripts?

from usearch.

I think it compiles fine now. This is a separate issue of bundling.

from usearch.

I’ll look at the package.json as well as try using esbuild bundle and see if that works (it also supports .node files)

The package https://www.npmjs.com/package/@vercel/ncc handles bundling with .node files, and worked with usearch before 2.10.x. Example: usearch__arm64__nodejs20__2_9_2.zip

If it can’t be bundled, it can’t be used in serverless like AWS Lambda.

from usearch.

I think @vercel/ncc is failing because the way the bindings import has changed from something bundlers can analyze:

const compiled = require('bindings')('usearch');

To dynamic code that bundlers can’t analyze:

const compiled: Compiled = build(getBuildDir(getDirName()));

interface Compiled {
  CompiledIndex: CompiledIndex;
  exactSearch(
    dataset: VectorOrMatrix,
    queries: VectorOrMatrix,
    dimensions: number,
    count: number,
    metric: MetricKind
  ): CompiledSearchResult;
}

function getBuildDir(dir: string) {
  if (existsSync(path.join(dir, "build"))) return dir;
  if (existsSync(path.join(dir, "prebuilds"))) return dir;
  if (path.basename(dir) === ".next") {
    // special case for next.js on custom node (not vercel)
    const sideways = path.join(dir, "..", "node_modules", "usearch");
    if (existsSync(sideways)) return getBuildDir(sideways);
  }
  if (dir === "/") throw new Error("Could not find native build for usearch");
  return getBuildDir(path.join(dir, ".."));
}

function getDirName() {
  try {
    if (__dirname) return __dirname;
  } catch (e) { }
  return getRoot(getFileName());
}

@sroussey is it possible to use the previous import syntax with the bindings?

In the meantime, I will try to see if esbuild bundling works

Note to self, add "./package.json": "./package.json" back to package.json field exports

from usearch.

Yeah, prebuilt binaries have a different algorithm to find the binary, since there are several.

Have you tried --external ?

from usearch.

One thing I notice with better-sqlite3, duckdb, and @tensorflow/tfjs-node is that they don’t download the binaries for all platforms on install—they install the correct binary for the particular platform (named the same package-name.node file) at a well-known location for common libraries like bindings or @mapbox/node-pre-gyp to find.

from usearch.

I am not sure if I broke something, but potentially after these cosmetic changes the Windows CI failed. @sroussey I think it might be a good ide to deconvolute the build commands for MacOS, Linux, and Windows, similar to how I do for other platforms in release.yml? I am also curious, why are we excluding those targets:

      matrix:
        arch:
          - x64
          - x86
        os:
          - macos-latest
          - ubuntu-latest
          - windows-latest
        exclude:
          - arch: x86
            os: macos-latest
          - arch: x86
            os: ubuntu-latest

from usearch.

Then, try bundling with @vercel/ncc (using --external doesn’t make sense in this case, because our goal is to bundle usearch, not skip it):

Yeah, kinda. bundling and packaging are kinda being used interchangeably. A clearer way to think of it is that bundling is like inlining code (this is kinda sorta how the bundle ends up as one file), and packaging is taking this bundle and other assets and doing something with them. So external, kinda-sorta meaning don't inline, not so much don't include. I say kinda-sorta, because this is not C and linking and standard ABIs, etc., as things are much more... fluid... in the JS space, so every tool goes its own way.

from usearch.

Then, try bundling with @vercel/ncc (using --external doesn’t make sense in this case, because our goal is to bundle usearch, not skip it):

npx --yes @vercel/ncc build node_modules/usearch/javascript/dist/cjs/usearch.js -o ncc-usearch-bundled --debug

OK, what should I expect? I get

❯ npx --yes @vercel/ncc build node_modules/usearch/javascript/dist/cjs/usearch.js -o ncc-usearch-bundled --debug
ncc: Version 0.38.1
ncc: Compiling file index.js into CJS
Skipping asset emission of /Users/steve/Code/usearch-bundle/package.json for /Users/steve/Code/usearch-bundle/node_modules/bindings/bindings.js as it is outside the package base /Users/steve/Code/usearch-bundle/node_modules/bindings
41kB  ncc-usearch-bundled/index.js
41kB  [304ms] - ncc 0.38.1

from usearch.

OK, what should I expect? I get

That looks about right (I don’t remember the package.json warning). Though, before 2.10.x, it also included the .node file in the output

These don't include prebuilt binaries (you won't find a .node file among them):

Perhaps not in the npm package, but sometimes they are stored in github releases or other cache locations

Anyhow, I will look at ncc source code for ncc and see what they are doing for .node files.

Thank you for checking! I look forward to being able to use usearch again

from usearch.

Perhaps not in the npm package, but sometimes they are stored in github releases or other cache locations

Yeah... unfortunately that means having an install script run.

Thank you for checking! I look forward to being able to use usearch again

Oh, you should be able to just copy theprebuilds folder over, right? People usually do this with webpack.

From the ncc dev: "We specifically include support for bindings and node-pre-gyp". I am pretty sure they don't anticipate multiple architectures. But I'll see what they do.

from usearch.

Or this:

❯ npx --yes @vercel/ncc build test.js -o ncc-usearch-bundled --debug --external usearch
ncc: Version 0.38.1
ncc: Compiling file index.js into CJS
2kB  ncc-usearch-bundled/index.js
2kB  [113ms] - ncc 0.38.1
❯ cp -rp node_modules/usearch ncc-usearch-bundled
❯ node ncc-usearch-bundled/index.js
Index {}

That will contain some extra files though (all the stuff to build it)

from usearch.

for reference: vercel/ncc#1183

from usearch.

From the ncc dev: "We specifically include support for bindings and node-pre-gyp". I am pretty sure they don't anticipate multiple architectures.

Those libraries support prebuilt binaries with multiple architectures though right?

It’s just that only the correct binary for the particular os/architecture is downloaded using npm scripts at install time, rather than prebuildify’s new pattern of including all the architectures in the npm package.

But I'll see what they do.

I doubt @vercel/ncc or esbuild will move quickly to support prebuildify. They already bundle most .node files successfully if they follow the common pattern.

The only solution I can think of so far is to have a build script to move the correct binary to a location that common bundlers like esbuild, @vercel/ncc, and webpack expect.

If build scripts are enabled and the correct binary is renamed and moved to the expected location, all should work fine. If build scripts are disabled, then the new logic for locating the binary can be the fallback.

That way, bundling will still work with build scripts enabled and usearch will still work even if build scripts are disabled.

What do you think @sroussey ? I’m trying to think of other compromises, but that’s the only one that’s come to mind. If you have any other ideas or workarounds, please share!

from usearch.

I asked them what I can do on my side. I don't expect them to write any code, and would likely take too long anyhow.

I've been trying to use the folders they expect, but no success so far. Next try is to also add a require to something they expect and see if that triggers it. If I do find a folder and require combo that works, we may have to rename all the .node files since they all have the same name. Then I can collapse the folder structure. Github actions doesn't like that though. It likes assets to have their own folder. Maybe I can add another step after the builds are done but before the packaging.

from usearch.

BTW: ncc uses webpack-asset-relocator-loader.

So, some such around here:
https://github.com/vercel/webpack-asset-relocator-loader/blob/bda4108a3aeb672fe42da85650349ded73fba6e1/src/asset-relocator.js#L207-L215

and https://github.com/vercel/webpack-asset-relocator-loader/blob/bda4108a3aeb672fe42da85650349ded73fba6e1/src/asset-relocator.js#L907-L952

My thinking at the moment is that I don't use the prebuildify library, but instead create my own. Instead of using variables to figure out the path, I will switch on them and have hard coded paths that the code scanner will find. I think this will work across build tools.

I don't think it will be hard, but it will have to wait until tomorrow.

from usearch.

BTW: have you tried this?

npx --yes @vercel/ncc build test.js -o ncc-usearch-bundled --debug --external usearch
cd ncc-usearch-bundled
npm init -y
npm i usearch

That should work. At least to get you through the weekend. Also you see how external works and helps in tricky situations.

from usearch.

I asked them what I can do on my side. I don't expect them to write any code, and would likely take too long anyhow.

My apologies. I’m sorry I didn’t fully read the linked issue. You’re entirely right.

My thinking at the moment is that I don't use the prebuildify library, but instead create my own. Instead of using variables to figure out the path, I will switch on them and have hard coded paths that the code scanner will find. I think this will work across build tools.

That should work. If the paths can be analyzed, they can be bundled.

In previous js bundlers, a partial dynamic syntax was supported, but esbuild, vite, and others have not implemented it evanw/esbuild#700 so I think your method is best.

If I do find a folder and require combo that works, we may have to rename all the .node files since they all have the same name. Then I can collapse the folder structure. Github actions doesn't like that though. It likes assets to have their own folder. Maybe I can add another step after the builds are done but before the packaging.

I’ve done the same thing where I add another build step to move the assets before the publishing. It’s good to do if needed.

BTW: have you tried this?

I like your code examples! (especially the last one with --external flag) But, the main package I’m trying to bundle is usearch. I’ve already excluded it from the build, and I just need to bundle the package together into a single js + single .node file.

from usearch.

So, if I add this dummy code, it will work in ncc. But unfortunately, not esbuild:

if (process.uptime() < 0) {
    require(__dirname + "/../../../prebuilds/darwin-arm64+x64/usearch.node");
    require(__dirname + "/../../../prebuilds/linux-arm64/usearch.node");
    require(__dirname + "/../../../prebuilds/linux-x64/usearch.node");
    require(__dirname + "/../../../prebuilds/win32-ia32/usearch.node");
    require(__dirname + "/../../../prebuilds/win32-x64/usearch.node");
}

from usearch.

and maybe something like

require(__dirname + "/../../../build/Release/usearch.node");

in case people did their own build.

from usearch.

Sadly

require(__dirname + `/../../../build/${os.platform}-${os.arch}/usearch.node`);

did not work.

from usearch.

BTW: did you try just copying over the prebuilds folder in your adventures?

from usearch.

#397

from usearch.

cosmetic changes

Maybe a red herring? It uses windows-latest so maybe that changed instead?

@ashvardanian -- I did a force push of the last commit had a good windows build.

Both of these are the same commit:

https://github.com/unum-cloud/usearch/actions/runs/8625104966/job/23641178068

https://github.com/sroussey/usearch/actions/runs/8681208236/job/23803506624

This is a good reason to pin your dependencies and not use -latest etc.

from usearch.

@sroussey I am not sure I understand correctly.
The diff between our versions is ubuntu-latest vs ubuntu-22.04.
But in our case only the Windows runner is failing, Ubuntu is fine.
Moreover, ubuntu-latest and ubuntu-22.04 should map to the same config.

from usearch.

and maybe something like require(__dirname + "/../../../build/Release/usearch.node"); in case people did their own build.

That’s smart

Sadly require(__dirname + /../../../build/${os.platform}-${os.arch}/usearch.node);` did not work.

Still waiting to support variables in dynamic imports for esbuild evanw/esbuild#700

Would using relative paths allow esbuild to bundle correctly?

BTW: did you try just copying over the prebuilds folder in your adventures?

I try to let the bundlers do that, because I’ve had problems with other libraries where the paths change due to bundling and it’s better just to let them handle it.

from usearch.

But in our case only the Windows runner is failing, Ubuntu is fine.

My guess is that they changed something in the windows runner.

from usearch.

Seems like you are right, all of our Windows prebuild pipelines fail across repos. @sroussey whats the downside of disabling prebuild pipeline?

from usearch.

@sroussey started failing about a month ago, same in SimSIMD. Thank you!

from usearch.