Giter VIP home page Giter VIP logo

node-sqlite3-wasm's Introduction

node-sqlite3-wasm

npm SQLite

WebAssembly build of SQLite3 for Node.js

node-sqlite3-wasm is a port of SQLite3 to WebAssembly for Node.js with file system access. node-sqlite3-wasm brings SQLite3 to your Node.js environment without recompiling on every target platform. This is especially useful for Electron applications.

The port to WebAssembly that SQLite introduced in version 3.40.0 only targets web browsers but not Node.js. Other WebAssembly ports also target Node.js, most notably sql.js, but none supports persistent storage with direct file access. There also exist native bindings like better-sqlite3 or node-sqlite3. However, native bindings must be recompiled for every target platform or pre-built binaries must be shipped. This is tedious, especially for Electron deployments.

node-sqlite3-wasm supports persistent storage with direct file access by implementing an SQLite OS Interface or "VFS" that translates SQLite file access to Node.js' file system API.

node-sqlite3-wasm is currently based on SQLite 3.45.3.

Getting Started

To install node-sqlite3-wasm, run

npm install node-sqlite3-wasm

To use it, run

const { Database } = require("node-sqlite3-wasm");
const db = new Database("database.db");

Important: node-sqlite3-wasm is not fully garbage-collected. You have to manually close a database, otherwise you risk memory leaks (see Database.close()). Also, if you use prepared statements explicitly (see Database.prepare()), you have to manually finalize them. Alternatively, the Database class provides the convenience methods

These convenience methods use a prepared statement internally and take care of finalizing it.

Note: Foreign key support is enabled by default.

Example

const { Database } = require("node-sqlite3-wasm");
const db = new Database("database.db");

db.exec(
  "DROP TABLE IF EXISTS employees; " +
    "CREATE TABLE IF NOT EXISTS employees (name TEXT, salary INTEGER)"
);

db.run("INSERT INTO employees VALUES (:n, :s)", {
  ":n": "James",
  ":s": 50000,
});

const r = db.all("SELECT * from employees");
console.log(r);
// [ { name: 'James', salary: 50000 } ]

db.close();

API

class Database

Constructor

Methods

Properties

new Database(path, [options])

Creates a new database connection. By default, the database file is created if it doesn't exist.

Important: You have to manually close the database, otherwise you risk memory leaks (see Database.close()).

Arguments

  • path: the path to the database file
  • options (optional)
    • fileMustExist (default: false): if the database file does not exist it will not be created. Instead an SQLite3Error will be thrown. This option is ignored if readOnly is true.
    • readOnly (default: false): opens the database in read-only mode
const db = new Database("database.db");
const db = new Database("database.db", { fileMustExist: true });

Database.all(sql, [values, options]) -> rows

Creates a prepared statement, executes it with the given values and returns the resulting rows as an array of objects. The prepared statement is finalized automatically.

Arguments

  • sql: string containing the SQL statement
  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
  • options (optional)
    • expand (default: false): if true, each returned row is a nested object with keys corresponding to tables in the query. If a result column is an expression or subquery, it will be returned under the key $.
db.all("SELECT * FROM book");
db.all("SELECT * FROM book WHERE title = ?", "The Little Prince");
db.all("SELECT * FROM book WHERE title = :t", { ":t": "The Little Prince" });
db.all("SELECT * FROM book WHERE title IN (?, ?)", [
  "The Little Prince",
  "The Hobbit",
]);

Database.close()

Closes the database.

Important: You have to manually close the database, otherwise you risk memory leaks.

Important: Closing the database with Database.close() does not automatically finalize pending prepared statements.

db.close();

Database.exec(sql)

Executes the given SQL string. The SQL string may contain several semicolon-separated statements.

db.exec(
  "DROP TABLE IF EXISTS book; CREATE TABLE book (id INTEGER PRIMARY KEY, title TEXT)"
);

Database.function(name, func, [options]) -> this

Registers a user-defined function.

Arguments

  • name: the name of the function
  • func: the implementation of the function
  • options (optional)
    • deterministic (default: false): if true, the function is considered deterministic
db.function("regexp", (y, x) => new RegExp(y, "i").test(x), {
  deterministic: true,
});
db.all("SELECT * FROM book WHERE title REGEXP ?", ".*little.*");

Database.get(sql, [values, options]) -> row

Creates a prepared statement, executes it with the given values and returns the first resulting row as an object. The prepared statement is finalized automatically.

Arguments

  • sql: string containing the SQL statement
  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
  • options (optional)
    • expand (default: false): if true, the returned row is a nested object with keys corresponding to tables in the query. If a result column is an expression or subquery, it will be returned under the key $.
db.get("SELECT * FROM book WHERE id = ?", 7);
db.get("SELECT * FROM book WHERE id = $id", { $id: 7 });
db.get("SELECT * FROM book WHERE id = ? AND title = ?", [
  3,
  "The Little Prince",
]);

Database.prepare(sql) -> Statement

Creates a prepared statement from the given SQL string.

Important: You have to manually finalize a statement, otherwise you risk memory leaks. See Statement and, in particular, Statement.finalize().

const stmt = db.prepare("INSERT INTO book (title) VALUES (?)");
try {
  // do something with the statement here
} finally {
  stmt.finalize();
}

The Database class provides the convenience methods

These convenience methods use a prepared statement internally and take care of finalizing it.

Database.run(sql, [values]) -> info

Creates a prepared statement, executes it with the given values and returns an object with the properties changes and lastInsertRowid describing the number of modified rows and the id of the last row inserted. lastInsertRowid is a BigInt if its value exceeds Number.MAX_SAFE_INTEGER. The prepared statement is finalized automatically.

Arguments

  • sql: string containing the SQL statement
  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
db.run("INSERT INTO book (title) VALUES (?)", "The Little Prince");
db.run("INSERT INTO book VALUES (?, ?)", [10, "The Little Prince"]);
db.run("INSERT INTO book VALUES (@id, :title)", {
  "@id": 10,
  ":title": "The Little Prince",
});

Database.inTransaction

Property determining whether the database is currently in a transaction.

const stmt = db.prepare("INSERT INTO book (title) VALUES (?)");
try {
  db.exec("BEGIN TRANSACTION");
  stmt.run("The Little Prince");
  stmt.run("The Hobbit");
  db.exec("COMMIT");
} catch (err) {
  if (db.inTransaction) db.exec("ROLLBACK");
  console.log(err);
} finally {
  stmt.finalize();
}

Database.isOpen

Property determining whether the database is currently open.

class Statement

Methods

Properties

Important: You have to manually finalize a statement, otherwise you risk memory leaks (see Statement.finalize()).

const stmt = db.prepare("SELECT * FROM book WHERE id = ?");
try {
  // do something with the statement here
} finally {
  stmt.finalize();
}

As an alternative, the Database class provides the convenience methods

These convenience methods use a prepared statement internally and take care of finalizing it.

Statement.all([values, options]) -> rows

Executes the prepared statement with the given values and returns the resulting rows as an array of objects.

Arguments

  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
  • options (optional)
    • expand (default: false): if true, each returned row is a nested object with keys corresponding to tables in the query. If a result column is an expression or subquery, it will be returned under the key $.

See also Database.all()

Statement.finalize()

Finalizes the statement and frees all allocated memory. Once a statement has been finalized, it cannot be used anymore.

Important: You have to manually finalize a statement, otherwise you risk memory leaks.

Important: Closing the database with Database.close() does not automatically finalize pending prepared statements.

Statement.get([values, options]) -> row

Executes the prepared statement with the given values and returns the first resulting row as an object.

Arguments

  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
  • options (optional)
    • expand (default: false): if true, the returned row is a nested object with keys corresponding to tables in the query. If a result column is an expression or subquery, it will be returned under the key $.

See also Database.get()

Statement.iterate([values, options]) -> IterableIterator<row>

Executes the prepared statement with the given values and returns the resulting rows as an iterator of objects.

Arguments

  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.
  • options (optional)
    • expand (default: false): if true, each returned row is a nested object with keys corresponding to tables in the query. If a result column is an expression or subquery, it will be returned under the key $.

Statement.run([values]) -> info

Executes the prepared statement with the given values and returns an object with the properties changes and lastInsertRowid describing the number of modified rows and the id of the last row inserted. lastInsertRowid is a BigInt if its value exceeds Number.MAX_SAFE_INTEGER.

Arguments

  • values (optional): values to bind to the statement's parameters. Either a single value, an array, or an object in case of named parameters.

See also Database.run()

Statement.database

The Database object that instantiated this statement.

Statement.isFinalized

Property determining whether the statement has been finalized using Statement.finalize(). A finalized statement must not be used anymore.

class SQLite3Error

node-sqlite3-wasm throws an SQLite3Error whenever an error in SQLite or in the API occurs. SQLite3Error is a subclass of Error.

Notes About Types

Numbers

JavaScript's Number type is a double-precision 64-bit binary format IEEE 754 value. Integers can only be represented without loss of precision in the range -253 + 1 to 253 - 1, inclusive. SQLite3 works with 8-byte signed integers with a range of -263 to 263 - 1, inclusive. Since this range exceeds the range of safe integers in JavaScript, node-sqlite3-wasm automatically converts integers outside this safe range to BigInt. It is your responsibility to ensure that you handle the returned values, whether Number or BigInt, correctly. node-sqlite3-wasm also allows you to input BigInt values as query parameters, or arguments or return values of user-defined functions.

Binary Large Objects (BLOBs)

An SQLite Binary Large Object (BLOB) is represented by a Uint8Array in JavaScript.

Building

Docker and npm are required for building. Mocha is required to run tests.

To build node-sqlite3-wasm, simply run

npm run build

This will download the emscripten Docker image and the SQLite source files. Then it will compile the project source files and generate dist/node-sqlite3-wasm.js and dist/node-sqlite3-wasm.wasm.

License

node-sqlite3-wasm is MIT licensed.

Parts of the code are from sql.js, which is also MIT licensed. SQLite is in the public domain.

node-sqlite3-wasm's People

Contributors

github-actions[bot] avatar tndrle avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

node-sqlite3-wasm's Issues

[feature request] add support for `returing`

Thanks for your great work!

I'm writing a dialect for Kysely, that needs get lastInsertRowid, changes and rows when the sql have returing, but in my current(v0.8.0) try:

import sqlite from "node-sqlite3-wasm";

const db = new sqlite.Database(':memory:')

db.run('create table test1(a INTEGER, b TEXT)')

console.log(db.run('insert into test1(a, b) values (1, \'test\') returning *'))
// { changes: 0, lastInsertRowid: 1 }

console.log(db.exec('insert into test1(a, b) values (1, \'test\') returning *'))
// undefined

console.log(db.prepare('insert into test1(a, b) values (1, \'test\') returning *').all())
// [{ a: 1, b: 'test' }], but no changes and lastInsertRowid

maybe expose a new API or just expose sqlite.changes()?

Issue with Dropbox sync?

Hi,

We have got another report about issue which didn't happen while using better-sqlite3 native module in our Electron app.

User reports that he gets often the following error:

A JavaScript error occurred in the main process
Uncaught Exception:
SQlite3Error: database is locked
at Database._handleError

It disappears when Dropbox synchronization is disabled.

It is possible that your library is more sensitive for files locked by Dropbox sync process than native SQLite3 module?

failed to compile wasm module: RangeError: WebAssembly.Compile is disallowed on the main thread

I'm creating an electron App then i want to use node-sqlite3-wasm
I was using better-sqlite3 before, to switch to node-sqlite3-wasm, I just changed the lines:

const Database = require('better-sqlite3');
const db = new Database('foobar.db', { verbose: console.log });

to

const { Database } = require("node-sqlite3-wasm");
const db = new Database('breach.db', { fileMustExist: true });

of course I also changed the request methods like db.prepare() to equivalents...

but when i run my app, i get this error :

failed to compile wasm module: RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.
instantiateSync @ VM167 node-sqlite3-wasm.js:8
VM161 renderer_init:2 Unable to load preload script: /Path/to/my-app/src/preload.js
(anonymous) @ VM161 renderer_init:2
VM161 renderer_init:2 RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.
    at instantiateSync (VM167 node-sqlite3-wasm.js:8:18906)
    at createWasm (VM167 node-sqlite3-wasm.js:8:19965)
    at VM167 node-sqlite3-wasm.js:8:33718
    at Object.<anonymous> (VM167 node-sqlite3-wasm.js:15:5)
    at Object.<anonymous> (VM167 node-sqlite3-wasm.js:23:3)
    at Module._compile (VM126 loader:1269:14)
    at Module._extensions..js (VM126 loader:1324:10)
    at Module.load (VM126 loader:1124:32)
    at Module._load (VM126 loader:965:12)
    at f._load (VM158 asar_bundle:2:13330)

i included the module from preload.js and serve it to index.html via contextBridge.exposeInMainWorld

can sameone tell me how to solve this problem please ?

Issue with Electron and Windows 11

Hi,

First of all thank you for this library - we had serious issues with better-sqlite3 on older linux machines and we have decided to use this one in Publii.

Unfortunately we have discovered one important issue on Windows 11 - it seems that your library doesn't properly close the database connection. In result our app is unable to remove website folder while restoring website from backup - the db.sqlite file cannot be removed (even by rimraf). The EPERM error is returned. File is removed after closing the app so definitely it is connected with some unclosed handles to the db.sqlite file.

The problem occurs only on Windows - we didn't observed it on mac OS or Linux. We have even tried to test it outside protected directories (on desktop) and with EV-signed app version, but the problem still occurs, so most probably it is not caused by Windows itself. Also app based on better-sqlite3 properly closes the DB connection and there is no issue with removing db.sqlite file.

I have even tried to create a custom build and use close() instead of close_v2() as I have seen that better-sqlite3 uses close(), but without success, so there must be some additional operations to do during closing the connection with DB.

Table index is out of bounds error

It seems that large-ish blobs (bigger than ~64kb) can't be put into a table, I'm not sure where the error is, but this should reproduce the error:

import * as Sqlite3Wasm from 'node-sqlite3-wasm';
const Database = Sqlite3Wasm.default.Database;

const SAMPLE = new TextEncoder ().encode ( 'a'.repeat ( 100_000 ) );

const wdb = new Database ( ':memory:' );

await wdb.exec ( `CREATE TABLE example ( id INTEGER PRIMARY KEY, data BLOB )` );
await wdb.prepare ( `INSERT INTO example VALUES( ?, ? )` ).run ([ 1, SAMPLE ]);

const read = await wdb.all ( `SELECT data FROM example WHERE id=1` );

await wdb.exec ( `DELETE FROM example WHERE id=1` );

And error is this:

wasm://wasm/0039d32e:1


RuntimeError: table index is out of bounds
    at wasm://wasm/0039d32e:wasm-function[285]:0x5341e
    at wasm://wasm/0039d32e:wasm-function[411]:0x79401
    at wasm://wasm/0039d32e:wasm-function[784]:0xc1ddc
    at ccall (/Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:29808)
    at /Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:30193
    at Statement._bindValue (/Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:12466)
    at Statement._bindArray (/Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:11500)
    at Statement._bind (/Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:10208)
    at Statement.run (/Users/fabio/Code/fabiospampinato/tiny-sqlite3/node_modules/node-sqlite3-wasm/dist/node-sqlite3-wasm.js:8:9263)
    at file:///Users/fabio/Code/fabiospampinato/tiny-sqlite3/tasks/blob.js:61:60

Node.js v18.12.0

Could this be fixed?

knex support?

Hi, that pretty comfortable to use knex as query builder for sql databases
seems default sqlite3 driver is not compatible with node-sqlite3-wasm
is there any way to connect node-sqlite3-wasm with knex?

node-sqlite3-wasm.d.ts missing

It would be nice to have d.ts file (maybe adapted sqlite3.d.ts) to omit warnings in VSC.
Or maybe @types/node-sqlite3-wasm ?

Drizzle support

Thank you! Amazing work!
No one has done this until now, and using SQLite with node was a headache.

I was wondering if you could add official support for the Drizzle ORM.

Drizzle is a new, popular yet simple ORM
https://orm.drizzle.team/

Your API is very similar to better-sqlite3, so it may be already fully supported.
I am sure that if you mention that many people will know and support your work.

Good Luck :)

Include FTS5 extension

It would be great if the SQLite build could include the FTS5 extension to support use cases that need fulltext search support.
It should be enough include -DSQLITE_ENABLE_FTS5 in SQLITE_FLAGS

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.