Giter VIP home page Giter VIP logo

wa-sqlite's Introduction

wa-sqlite CI

wa-sqlite

This is a WebAssembly build of SQLite with experimental support for writing SQLite virtual filesystems and virtual table modules completely in Javascript. This allows alternative browser storage options such as IndexedDB and File System Access. Applications can opt to use either a synchronous or asynchronous (using Asyncify) SQLite library build (an asynchronous build is required for asynchronous extensions).

IndexedDB and Origin Private File System virtual file systems and a virtual table module that accesses Javascript arrays are among the examples provided as proof of concept.

Try the demo or run benchmarks with a modern desktop web browser. More information is available in the FAQ, discussion forums, and API reference.

Build

The primary motivation for this project is to enable additions to SQLite with only Javascript. Most developers should be able to use the pre-built artifacts in ./dist. Note that earlier versions of the project only provided pre-built artifacts in the "buildless" branch; that branch will no longer be maintained.

Minor build customization (e.g. changing build defines or flags) can be done with make arguments, and the helper project sqwab can be used to build without a local build environment.

If you do want to build yourself, here are the prerequisites:

  • Building on Debian Linux is known to work, compatibility with other platforms is unknown.
  • yarn - If you use a different package manager (e.g. npm) then file paths in the demo will need adjustment.
  • Emscripten SDK 3.1.25+.
  • curl, make, openssl, sed, tclsh, unzip

Here are the build steps:

  • Make sure emcc works.
  • git clone [email protected]:rhashimoto/wa-sqlite.git
  • cd wa-sqlite
  • yarn install
  • make

The default build produces ES6 modules + WASM, synchronous and asynchronous (using Asyncify) in dist/.

API

Javascript wrappers for core SQLITE C API functions (and some others) are provided. Some convenience functions are also provided to reduce boilerplate. Here's sample code to load the library and call the API:

  import SQLiteESMFactory from 'wa-sqlite/dist/wa-sqlite.mjs';
  import * as SQLite from 'wa-sqlite';

  async function hello() {
    const module = await SQLiteESMFactory();
    const sqlite3 = SQLite.Factory(module);
    const db = await sqlite3.open_v2('myDB');
    await sqlite3.exec(db, `SELECT 'Hello, world!'`, (row, columns) => {
      console.log(row);
    });
    await sqlite3.close(db);
  }

  hello();

The implementation of sqlite3.exec may be of interest to anyone wanting more fine-grained use of SQLite statement objects (e.g. for binding parameters, explicit column datatypes, etc.).

API reference

Demo

To serve the demo directly from the source tree:

The demo page provides access to databases on multiple VFS implementations, including IndexedDB (which is the only one persistent across page loads and multiple tabs). In addition, in each database there is a SQLite module named "array" that provides some historical stock data from a common Javascript array - use it for virtual tables in SQL like this:

CREATE VIRTUAL TABLE IF NOT EXISTS goog USING array;

-- Use it directly out of the Javascript array:
SELECT * FROM goog LIMIT 5;

-- Copy into a native table (on the current VFS):
CREATE TABLE IF NOT EXISTS tbl AS SELECT * FROM goog;

For convenience, if any text region is selected in the editor, only that region will be executed. In addition, the editor contents are restored across page reloads using browser localStorage.

License

MIT License as of February 10, 2023, changed by generous sponsors Fleet Device Management and Reflect. Existing licensees may continue under the GPLv3 or switch to the new license.

wa-sqlite's People

Contributors

dependabot[bot] avatar joeyates avatar rhashimoto avatar shoestringresearch avatar tantaman avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wa-sqlite's Issues

Publish as npm module

Very curious in your approach, wanted it to be directly from npm... especially given the GPL licensing. In the mean time I'll try building locally.

accessing current time on VFS database throws bad function error

On the demo, accessing the current time, e.g. with:

SELECT datetime('now');

works on the default filesystem but fails with any custom VFS with:

RuntimeError: null function or function signature mismatch
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.wasm:wasm-function[659]:0x38ed2
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.wasm:wasm-function[1176]:0x57047
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.wasm:wasm-function[503]:0x26f89
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.wasm:wasm-function[1177]:0x5709b
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.wasm:wasm-function[530]:0x2b26c
    at G (https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.mjs:17:311)
    at https://rhashimoto.github.io/wa-sqlite/dist/wa-sqlite.mjs:110:235
    at Object.step (https://rhashimoto.github.io/wa-sqlite/src/sqlite-api.js:562:28)
    at https://rhashimoto.github.io/wa-sqlite/src/examples/tag.js:37:28
    at async HTMLButtonElement. (https://rhashimoto.github.io/wa-sqlite/demo/index.js:121:23)

A custom VFS is supposed to borrow the default time method here but this is apparently broken.

Slow Query Execution for Tables with Diverse Data

I am experiencing a perplexing issue with the wa-sqlite regarding the execution speed of queries on tables with varying data. To provide context, when I run a query on a table containing 17284 rows, each with identical data, the query performs exceptionally fast. However, when the data in these rows differs from one another, the query execution becomes noticeably slow

For Test table that have 17284 rows, simple query Select Count(*) got 53ms
query done for SELECT Count(*) FROM Test took: 53 ms
But for Test2 table that have 15000 rows, but the values on each row are differs from each other
query done for SELECT Count(*) FROM Test2 took: 309 ms
example row for Test2:
id: 16,
name: "Student name 1704788636264",
email: "Student email 1704788636264",
phone "Student phone 1704788636264",
age: "Student age 1704788636264"

Error: no such module: rtree

I'm trying to use the rtree module following the sqlite docs however this statement fails:

CREATE VIRTUAL TABLE demo_index USING rtree(
   id,              -- Integer primary key
   minX, maxX,      -- Minimum and maximum X coordinate
   minY, maxY       -- Minimum and maximum Y coordinate
);

with this error:

Error: no such module: rtree
    at check (https://rhashimoto.github.io/wa-sqlite/src/sqlite-api.js:857:11)
    at Object.step (https://rhashimoto.github.io/wa-sqlite/src/sqlite-api.js:690:14)
    at async execute (https://rhashimoto.github.io/wa-sqlite/src/examples/tag.js:50:16)

I've checked by doing a local build that -DSQLITE_ENABLE_RTREE is indeed passed as a compilation option, so I would expect this to work. Unless I'm missing something?

Cheers,

Jun

How do I load the module in my application?

Applications should import the Emscripten module factory and the SQLite API:

// Emscripten module factory (most applications use only one)
import SQLiteESMFactory from "wa-sqlite/dist/wa-sqlite.mjs";
import SQLiteAsyncESMFactory from "wa-sqlite/dist/wa-sqlite-async.mjs";

// SQLite API constants and functions
import * as SQLite from 'wa-sqlite';

Most applications will only use one of the Emscripten builds, either the synchronous build or the asynchronous build. See this question for an explanation of the difference.

The Emscripten module factory is a function that asynchronously loads, compiles, and initializes the corresponding WebAssembly file. Both the module (.mjs) and WebAssembly (.wasm) files are located in the dist/ subdirectory of the wa-sqlite package. If the Emscripten module factory is called with no arguments, then the .wasm file will be loaded from the same directory path as the .mjs file, e.g.:

const module = await SQLiteESMFactory();

That works fine if you're serving the Emscripten module as-is to the browser but bundling can complicate matters (especially when targeting a Worker). If you need the ability to explicitly specify the .wasm URL, do that by passing a configuration argument to the Emscripten module factory:

const module = await SQLiteESMFactory({
  // When provided a filename, return the URL for that filename.
  // This example uses the same directory as the document.
  locateFile(file) {
    return `./${file}`;
  }
});

Once you have an Emscripten module instance, you can create an API instance:

const sqlite3 = SQLite.Factory(module);

// And you're off and running...
const db = await sqlite3.open_v2('myDB');
...

See also the reference docs.

mjs and masm files ( /dist) are missing

Hi @rhashimoto 😊 Thank you for your work 🙏🏻 I am hoping that I will be able to get the demo running. However, seems like a couple of files missing . Specifically these 2 files are missing:

~/dist/wa-sqlite-async.mjs
~/dist/wa-sqlite.mjs
~/dist/wa-sqlite-async.wasm
~/dist/wa-sqlite-async.wasm

From where can I get those files ?
In the meantime, I just copied them from : https://rhashimoto.github.io/wa-sqlite/dist/

Also it seems to need CSP settings. wasm-unsafe-eval

Typescript typings are incomplete / not a module

You can't import the main api via

import * as SQLite from "wa-sqlite";

because it throws this error:

File '.../node_modules/wa-sqlite/src/types/sqlite-api.d.ts' is not a module.

The dist file etc are also not typed, import SQLiteAsyncESMFactory from "wa-sqlite/dist/wa-sqlite-async.mjs"; throws a type error, as does import * as VFS from "wa-sqlite/src/VFS";

Call stack size exceeded of wasm asyncify

Hey! I found one weird bug, I have RangeErrors for some queries when I use asyncified version of wa-sqlite. SQLite just fails to prepare this query:

VALUES(0) UNION VALUES(1) UNION VALUES(2) UNION VALUES(3) UNION VALUES(4) UNION VALUES(5) UNION VALUES(6) UNION VALUES(7) UNION VALUES(8) UNION VALUES(9) UNION VALUES(10) 
UNION VALUES(11) UNION VALUES(12) UNION VALUES(13) UNION VALUES(14) UNION VALUES(15) UNION VALUES(16) UNION VALUES(17) UNION VALUES(18) UNION VALUES(19) UNION VALUES(20) 
UNION VALUES(21) UNION VALUES(22) UNION VALUES(23) UNION VALUES(24) UNION VALUES(25) UNION VALUES(26) UNION VALUES(27) UNION VALUES(28) UNION VALUES(29) UNION VALUES(30) 
UNION VALUES(31) UNION VALUES(32) UNION VALUES(33) UNION VALUES(34) UNION VALUES(35) UNION VALUES(36) UNION VALUES(37) UNION VALUES(38) UNION VALUES(39) UNION VALUES(40) 
UNION VALUES(41) UNION VALUES(42) UNION VALUES(43) UNION VALUES(44) UNION VALUES(45) UNION VALUES(46) UNION VALUES(47) UNION VALUES(48) UNION VALUES(49) UNION VALUES(50) 
UNION VALUES(51) UNION VALUES(52) UNION VALUES(53) UNION VALUES(54) UNION VALUES(55) UNION VALUES(56) UNION VALUES(57) UNION VALUES(58) UNION VALUES(59) UNION VALUES(60) 
UNION VALUES(61) UNION VALUES(62) UNION VALUES(63) UNION VALUES(64) UNION VALUES(65) UNION VALUES(66) UNION VALUES(67) UNION VALUES(68) UNION VALUES(69) UNION VALUES(70) 
UNION VALUES(71) UNION VALUES(72) UNION VALUES(73) UNION VALUES(74) UNION VALUES(75) UNION VALUES(76) UNION VALUES(77) UNION VALUES(78) UNION VALUES(79) UNION VALUES(80) 
UNION VALUES(81) UNION VALUES(82) UNION VALUES(83) UNION VALUES(84) UNION VALUES(85) UNION VALUES(86) UNION VALUES(87) UNION VALUES(88) UNION VALUES(89) UNION VALUES(90) 
UNION VALUES(91) UNION VALUES(92) UNION VALUES(93) UNION VALUES(94) UNION VALUES(95) UNION VALUES(96) UNION VALUES(97) UNION VALUES(98) UNION VALUES(99) UNION VALUES(100) 
UNION VALUES(101) UNION VALUES(102) UNION VALUES(103) UNION VALUES(104) UNION VALUES(105) UNION VALUES(106) UNION VALUES(107) UNION VALUES(108) UNION VALUES(109) UNION VALUES(110) 
UNION VALUES(111) UNION VALUES(112) UNION VALUES(113) UNION VALUES(114) UNION VALUES(115) UNION VALUES(116) UNION VALUES(117) UNION VALUES(118) UNION VALUES(119) UNION VALUES(120) 
UNION VALUES(121) UNION VALUES(122) UNION VALUES(123) UNION VALUES(124) UNION VALUES(125) UNION VALUES(126) UNION VALUES(127) UNION VALUES(128) UNION VALUES(129) UNION VALUES(130) 
UNION VALUES(131) UNION VALUES(132) UNION VALUES(133) UNION VALUES(134) UNION VALUES(135) UNION VALUES(136) UNION VALUES(137) UNION VALUES(138) UNION VALUES(139) UNION VALUES(140) 
UNION VALUES(141) UNION VALUES(142) UNION VALUES(143) UNION VALUES(144) UNION VALUES(145) UNION VALUES(146) UNION VALUES(147) UNION VALUES(148) UNION VALUES(149) UNION VALUES(150) 
UNION VALUES(151) UNION VALUES(152) UNION VALUES(153) UNION VALUES(154) UNION VALUES(155) UNION VALUES(156) UNION VALUES(157) UNION VALUES(158) UNION VALUES(159) UNION VALUES(160) 
UNION VALUES(161) UNION VALUES(162) UNION VALUES(163) UNION VALUES(164) UNION VALUES(165) UNION VALUES(166) UNION VALUES(167) UNION VALUES(168) UNION VALUES(169) UNION VALUES(170) 
UNION VALUES(171) UNION VALUES(172) UNION VALUES(173) UNION VALUES(174) UNION VALUES(175) UNION VALUES(176) UNION VALUES(177) UNION VALUES(178) UNION VALUES(179) UNION VALUES(180) 
UNION VALUES(181) UNION VALUES(182) UNION VALUES(183) UNION VALUES(184) UNION VALUES(185) UNION VALUES(186) UNION VALUES(187) UNION VALUES(188) UNION VALUES(189) UNION VALUES(190) 
UNION VALUES(191) UNION VALUES(192) UNION VALUES(193) UNION VALUES(194) UNION VALUES(195) UNION VALUES(196) UNION VALUES(197) UNION VALUES(198)

image

The interesting thing is that:

  1. It runs well when DevTools is opened, and keeps working well after devtool is closed (after closing current tab & opening new it becomes reproducible again)
  2. It runs well on the first attempt. After it fails

I tested it on m1 MacBook with Chrome

112.0.5615.137 (Official Build) (arm64)

The query run ok with any wasm standard building under any condition.

Possible Web Locks deadlock

There is a potential deadlock in src/examples/WebLocks.js as described here. xLock() is supposed to timeout to prevent this but the current implementation does not.

What about replacing Asyncify with Atomics and SharedArrayBuffer?

UPDATE 5/7/23: AccessHandlePoolVFS uses OPFS for storage without Asyncify or Atomics/SharedArrayBuffer. It requires recent browser versions (Chrome 108+, Safari 16.4+, Firefox 111+) that support the latest OPFS access handle spec.

There are significant time and space penalties for using WebAssembly SQLite built with Asyncify, which allows SQLite to callback to asynchronous Javascript functions. This is why wa-sqlite provides both synchronous (without Asyncify) and asynchronous builds, so developers can avoid those penalties if they don't need to use asynchronous callbacks.

Asynchronous callbacks are especially useful for calling asynchronous APIs like IndexedDB or fetch. But @phiresky (sql.js-httpvfs) and @jlongster (absurd-sql) independently discovered a clever way to make synchronous calls to asynchronous APIs using Atomics and SharedArrayBuffer. It's pretty cool!

The main catch with these APIs (and why it took so long to get support in Safari again) is they are potentially exploitable by Spectre malware attacks, so they are only enabled on pages served with special COOP/COEP HTTP headers which also heavily restrict including content from a different origin. This may prevent integrating certain third-party features like federated sign-in (e.g. OAuth2) and ads right now.

Implementing a SQLite VFS or module using Atomics and SharedArrayBuffer based on the described methods should be straightforward with the wa-sqlite synchronous build and I would be excited to see it happen. I don't plan to do that myself, however - i.e. I don't plan to add a synchronous IndexedDB (or File System Access) VFS to this repo - at least until it can be made to work with federated sign-in.

The size of the WebAssembly for wa-sqlite is about 507 KB for the synchronous build and 1,032 KB for the Asyncify build, so roughly double the size. The speed penalty is harder to characterize because although the cost of unwinding and rewinding the stack for asynchronous calls is quite high, it may still be relatively small compared to the asynchronous function itself. For example, if you're making a 1500 ms asynchronous call to WebTorrent then you may not care if Asyncify adds another 5 ms (I made these numbers up just for illustration). In general, the slower your wa-sqlite storage medium, the less you should care about Asyncify performance...though you still might want to tune your page size to reduce both Asyncify and non-Asyncify overhead.

Those interested in synchronous WebAssembly filesystems should also track Emscripten's WasmFS which is in development.

Full text support

We're evaluating wa-sqlite for a project. Would it be im/possible to set up full text support (fts5)? If possible, roughly how so? I see there's a call to create_module to add ARRAY. Or would we need to customize the Makefile? I read that "FTS5 is included as part of the SQLite amalgamation" but the Makefile already mentions "amalgamation" so I'm not sure. NOTE: I'm using the buildless branch right now.

Add a way to clear an orphaned lock on IndexedDB VFS

If the demo page crashes or is closed during a transaction on the IndexedDB VFS, e.g. if the user executes a BEGIN TRANSACTION and no subsequent COMMIT or ROLLBACK, then the database will have an orphaned lock and all future locking attempts will fail with SQLITE_BUSY.

If the journal is not also written to IndexedDB (e.g. PRAGMA journal_mode = MEMORY or OFF) then the database may be unrecoverably corrupted and it should be deleted (but see below). If the journal is on IndexedDB then forcibly clearing the lock (by resetting isLocked in the metadata object) will allow the database to be automatically repaired if necessary on the next usage.

These operations can be done directly on the IndexedDB store, but there should be methods on the IndexedDB VFS for both.

As an aside, the database can be corrupted when the journal is lost because transaction changes are only partially written. With the current implementation this will happen when the maximum number of cached blocks is reached and subsequent writes cause dirty blocks to be incrementally flushed. If the VFS did not cap the number of cached blocks, they would all be written atomically upon sync and the database would never be left in a corrupted state even with no journal. The downside of that is unlimited memory usage for large transactions, but it's a trade-off worth considering for many (probably most) applications.

Fix IndexedDB xTruncate cache interaction

The current implementation of xTruncate for IndexedDB pre-dates the addition of caching. In addition to updating the size in metadata, it deletes IndexedDB block objects past the EOF. If dirty blocks past EOF are in the cache, they will be written again on xSync.

The proper fix here is probably:

  • In xTruncate:
    • Update size in metadata (as before).
    • Remove blocks past EOF from the cache, not from IndexedDB.
  • In xSync:
    • After flushing the cache, delete IndexedDB block objects past EOF.

Which VFS is the best?

As usual...it depends.

First of all, the primary reason wa-sqlite was created was to experiment with different APIs and ideas for SQLite storage. The example VFS classes were written to try a variety of approaches to see what works, what doesn't, and where to go next. Definitely not all but also perhaps not any may be well-suited for your application. You're welcome to use one but modifying an example VFS or writing your own to fit your requirements may be better, and that's really the spirit of the project.

For example, all the example classes that implement the SQLite locking model use only exclusive Web Locks, so they won't allow concurrent read transactions from separate connections. If that is a feature you want, you can change it (quite easily in this case).

Of the example VFS classes, IDBBatchAtomicVFS is the most well-exercised and the most general of the IndexedDB classes. The new Origin Private File System API is just now getting support across the major browsers, and AccessHandlePoolVFS implements the most promising OPFS approach. AccessHandlePoolVFS doesn't support multiple connections because OPFS access handles are exclusive, but sharing a connection across multiple contexts can be done at the application level.

There doesn't appear to be a definitive winner between IDBBatchAtomicVFS and AccessHandlePoolVFS, though IDBBatchAtomicVFS has the edge right now in maturity and browser support. The better choice will depend on the application usage and priorities. Here are some of the differences:

  • IDBBatchAtomicVFS
    • Pros
      • Lower write transaction overhead.
      • Easy to modify to allow concurrent read transactions.
      • Can trade durability for performance.
    • Cons
      • Requires asynchronous API workarounds.
      • Slower at raw I/O.
  • AccessHandlePoolVFS
    • Pros
      • No asynchronous API workarounds.
      • Fast I/O, should have faster read transactions.
    • Cons
      • Just getting wide browser support now.
      • No read concurrency.
      • Performs extra syncs because of conservative filesystem assumptions.

Strange error stack trace: both "SQL error" and "journal file not found"

Hi. I encountered these strange stack trace on using wa-sqlite in my project:

image

Upon reproducing/figuring out how does the error happen, it looks like at that moment, there seems to be two place "simultaneously" call wa-sqlite:

// in one file
  onMount(async () => {
    await migrate(localDb, defaultMigrationQueryMap, defaultQueriesStringMap)
  })

// in another
export const localSnippetsStore = await createLocalSnippetStoreV2(localDb)

Where localDb is a wrapper of Drizzle around SQLiteAPI that uses .statements:

// in one file
export async function createLocalDb(executor: QueryExecutor) {
  return drizzle(async (queryString, params, method) => {
    const result = await executor.execute(queryString, ...params)
    if (method === 'get' && result.length > 0) {
      return {rows: result[0]}
    }
    return {rows: result}
  })
}

// in another
    ...
    async execute(query: string, ...params: SQLiteCompatibleType[]): Promise<SQLiteCompatibleType[][]> {
      const rows = []
      for await (const stmt of sqlite3.statements(db, query)) {
        params.forEach((param, index) => sqlite3.bind(stmt, index + 1, param))
        while (await sqlite3.step(stmt) === SQLite.SQLITE_ROW) {
          rows.push(sqlite3.row(stmt))
        }
      }
      return rows
    },
    ...

If I comment out the await migrate and only run localSnippetStore = ... (or vice versa), then the code works as expected. Therefore, I feel like my way of wrapping might be the problem, but cannot be sure how to process further. Do you have any suggestion?

Many thanks!

Please read this (click here) before opening a new issue 😀

Please take the time to check the FAQ and documentation before opening a new issue.

Although many GitHub projects use Issues as a help desk, this project reserves Issues for bugs in the project itself. Asking for debugging help with your own code is discouraged unless it indicates a project bug.

Topics other than bug reports should usually take place in Discussions instead. Also, this repo generally does not accept pull requests so please start a conversation first.

Thanks!

How to bundle / specify wasm URL?

Using webpack, it tries to load the WASM from a file:/// url and the wasm file is not bundled as an asset. Is it possible to specify the wasm URL or make it work with webpack in some other way?

Error: database disk image is malformed

I'm getting error database disk image is malformed when setting cache_size. When I removed cache_size config, the error is gone
Logs: executeQuery result for SELECT id FROM tyCOmBsBQlXDRsoxskoM WHERE _status =? is empty Error: database disk image is malformed at check (sqlite-api.js:1156:11) at Object.eval [as step] (sqlite-api.js:983:14)
VFS: IDBBatchAtomicVFS

here is my code for database initialization

    const module = await SQLESMFactory();
    const sqlite3: SQLiteAPI = SQLite.Factory(module);
    let db: number = -1
    const vfs = new IDBBatchAtomicVFS('starion-db')
    await vfs.isReady;
    await sqlite3.vfs_register(vfs, true);
    db = await sqlite3.open_v2(fileName);
    await sqlite3.exec(db, `PRAGMA cache_size=${32 * 1024};`)
    await sqlite3.exec(db, `PRAGMA page_size=8192;`)
    sqlite3.exec(db, `PRAGMA journal_mode=MEMORY;`)
    await sqlite3.exec(db, `PRAGMA temp_store=MEMORY;`);

OriginPrivateFileSystemVFS SQL_IOERR with journal_mode TRUNCATE or PERSIST

If multiple connections are using the OriginPrivateFileSystemVFS with PRAGMA journal_mode TRUNCATE or PERSIST, an SQL_IOERR is likely to result. The problem is that in these modes multiple connections can open the journal file and OPFS access handles are exclusive. A workaround is not to use these modes with multiple connections.

This is not a high priority to fix right now. IDBBatchAtomicVFS is better in pretty much every way that matters to application writers - ease of integration, performance, maturity - so the OPFS VFS is mostly a proof of concept.

How do I implement file locking in a VFS?

The example VFS classes that access shareable storage (IndexedDB and OPFS) use the Web Locks API to implement locking. As of December 2021, this API is supported on all the evergreen browsers (Safari recently fixed this Worker bug which should ship in 15.6, but didn't so hopefully the next one).

Although the SQLite locking model allows shared read locks - i.e. multiple database connections can read the same database at the same time to enhance performance - the example classes do not support this so both read and write transactions take an exclusive lock. There is a drop-in replacement locking class, WebLocksShared, which does support the full locking model including shared reads. To use it, you will need to modify the VFS code by changing the import:

import { WebLocksExclusive as WebLocks } from './WebLocks.js';

to this:

import { WebLocksShared as WebLocks } from './WebLocks.js';

If you use a locking implementation that returns SQLITE_BUSY, like WebLocksShared, you must also handle this result in your application code (wa-sqlite throws an Error with code property SQLITE_BUSY and message property "database is locked"). From the SQLite docs for sqlite3_step:

SQLITE_BUSY means that the database engine was unable to acquire the database locks it needs to do its job. If the statement is a COMMIT or occurs outside of an explicit transaction, then you can retry the statement. If the statement is not a COMMIT and occurs within an explicit transaction then you should rollback the transaction before continuing.

Failure to do this will risk deadlock if you have concurrent access (e.g. multiple browser tabs). This added responsibility for applications is the reason why WebLocksShared is not the default implementation in the examples.

The alternative mechanisms to support locking without the Web Locks API are not great. The best of these probably is using a service worker as a central resource to manage locks. You can look at the edit history of this entry for this and other ideas. One important detail is that implementing the full SQLite locking model (i.e. including shared reads) requires upgradeable locks, i.e. from shared to exclusive, and if the upgrade fails the lock state should remain shared.

sqlite3.exec throws "RuntimeError: memory access out of bounds"

We have a Chrome extension using wa-sqlite (latest versions), and some Chromebooks are seeing:

image

The error typically takes ~2 days to show up, where the database is used for ~10 queries per minute. Once this error happens, it continues happening for each subsequent exec call.

I'm not familiar with WASM. Some thoughts/questions:

  1. Could there be a memory leak? If so, how to check?
  2. Could enabling Chrome extensions dev mode be causing this?
  3. Can I work around this by catching this error and re-initializing the DB? Or will the issue persist since it still uses the same WASM memory?
  4. Can this be fixed by compiling with more memory, like TOTAL_STACK=2mb

Can I use a sample VFS in my application?

Anyone is welcome to use wa-sqlite, including any of the sample classes like IDBBatchAtomicVFS, in their application provided they accept the terms of the license. However, there are some additional important considerations I'll outline here.

The code under examples/ and demo/ is meant to be a starting point to see how to use and extend the library. They are not necessarily suitable for use in your production application.

There are now three sample VFS classes for IndexedDB: IDBMinimalVFS, IDBVersionedVFS (deprecated), and IDBBatchAtomicVFS. They each have strengths and weaknesses. IDBBatchAtomicVFS balances performance and generality, and is now the featured IndexedDB VFS in the online demo and benchmarks. IDBMinimalVFS is the shortest and simplest, so it may be the best place to get started with understanding how they work.

There are two sample VFS classes for the Origin Private File System API: OriginPrivateFileSystemVFS and AccessHandlePoolVFS. Browser support for this API is increasing but is not as universal as IndexedDB, particularly with the features used by AccessHandlePoolVFS. Because AccessHandlePoolVFS appears to be the more promising approach of the two, development on OriginPrivateFileSystemVFS will likely be discontinued.

Note that all the sample VFS classes that use Web Locks for locking always use an exclusive lock so they don't support concurrent read transactions. If you want to allow concurrent reads, see the FAQ item on locks.

BigInt support

Passing BigInt to the bind APIs currently converts them to null as their type is not handled in the switch statement.

switch (typeof value) {
case 'number':
if (value === (value | 0)) {
return sqlite3.bind_int(stmt, i, value);
} else {
return sqlite3.bind_double(stmt, i, value);
}
case 'string':
return sqlite3.bind_text(stmt, i, value);
default:
if (value instanceof Int8Array || Array.isArray(value)) {
return sqlite3.bind_blob(stmt, i, value);
} else if (value === null) {
return sqlite3.bind_null(stmt, i);
} else if (value === undefined) {
// Existing binding (or NULL) will be used.
return SQLite.SQLITE_NOTICE;
} else {
console.warn('unknown binding converted to null', value);
return sqlite3.bind_null(stmt, i);

Looking at the official WASM build -- looks like it is possible to just expose bind_int64 from the C api and pass a BigInt to it.

WebLocks doesn't support multiple connections to the same database in the same context

WebLocks (and WebLocksShared) associate lock state with the filename. That means it can't distinguish multiple connections to the same database using the same WebLocks instance. The connections would overwrite each other's state.

An easy fix would be to give every open file its own instance of WebLocks, but...

Concurrent access to the same database in the same SQLite context where one access is a writer doesn't work anyway because Asyncify-ed WebAssembly is not re-entrant during an asynchronous callback. It isn't possible for one database call to block and then be able to make another database call, so fixing the bug won't enable any new usages.

Note that you can access the same database in the same SQLite context as long each call into SQLite completes before the next call is made.

OOM crash on Chrome Android

Both the demo and benchmarks pages crash Chrome Android on Google Pixel 2 and Lenovo Moto G6 Plus. Works correctly on the Android SDK emulator and every other browser/platform combination, so it may be specific to ARM devices.

Filed a Chromium bug and indications are an OOM crash and excessive register allocation time in the WebAssembly compiler, possibly (presumably?) related.

blobs should be uint8array not int8array?

I notice that blobs, when read out of wa-sqlite, are being returned as int8array rather than uint8array.

Also when inserting blobs -- you expect int8array.

I believe these should be Uint8Arrays instead.

SQLite constant definitions

As noted in #21, sqlite-api.js and VFS.js each contain their own overlapping constant definitions, violating DRY.

Also add some missing useful constants, like those here.

column_blob returns incorrect values

Any time a blob column is returned, the wrong values are returned for that column.

The pointer to the sqlite3_colum_blob is the same for every row in the result set even when the blobs are different.

I've created a small repository that reproduces the exact issue here:

https://github.com/tantaman/sandbox/tree/main


The interesting bits are in src/main.ts:

await sql`DROP TABLE IF EXISTS foo`;
await sql`CREATE TABLE IF NOT EXISTS foo (a)`

console.log('INSERTING 010203');
await sql`INSERT INTO foo VALUES (X'010203')`

console.log('SELECTING');
let result = await sql`SELECT * FROM foo`;
printBytesResult(result);

console.log('INSERTING 01020304');
await sql`INSERT INTO foo VALUES (X'01020304')`

console.log('SELECTING');
result = await sql`SELECT * FROM foo`;
printBytesResult(result);

console.log('INSERTING 01020304');
await sql`INSERT INTO foo VALUES (X'01020304')`

console.log('SELECTING');
result = await sql`SELECT * FROM foo`;
printBytesResult(result);

cosonle.log('QUOTE SELECTING');
result = await sql`SELECT quote(a) FROM foo`;
printHexResult(result);

Instead of getting a byte array of values 010203 or 01020304 we get:

INSERTING 010203
main.ts:73 SELECTING
main.ts:97 raw bytes Int8Array(3) [-16, -81, 81, buffer: ArrayBuffer(16777216), byteLength: 3, byteOffset: 5353200, length: 3, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af51
main.ts:77 INSERTING 01020304
main.ts:80 SELECTING
main.ts:97 raw bytes Int8Array(3) [-16, -81, 81, buffer: ArrayBuffer(16777216), byteLength: 3, byteOffset: 5350640, length: 3, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af51
main.ts:97 raw bytes Int8Array(4) [-16, -81, 81, 0, buffer: ArrayBuffer(16777216), byteLength: 4, byteOffset: 5350640, length: 4, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af5100
main.ts:84 INSERTING 01020304
main.ts:87 SELECTING
main.ts:97 raw bytes Int8Array(3) [-16, -81, 81, buffer: ArrayBuffer(16777216), byteLength: 3, byteOffset: 5352176, length: 3, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af51
main.ts:97 raw bytes Int8Array(4) [-16, -81, 81, 0, buffer: ArrayBuffer(16777216), byteLength: 4, byteOffset: 5352176, length: 4, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af5100
main.ts:97 raw bytes Int8Array(4) [-16, -81, 81, 0, buffer: ArrayBuffer(16777216), byteLength: 4, byteOffset: 5352176, length: 4, Symbol(Symbol.toStringTag): 'Int8Array']
main.ts:98 hex f0af5100
main.ts:91 QUOTE SELECTING
main.ts:104 raw bytes (3) [1, 2, 3]
main.ts:105 hex X'010203'
main.ts:104 raw bytes (4) [1, 2, 3, 4]
main.ts:105 hex X'01020304'
main.ts:104 raw bytes (4) [1, 2, 3, 4]
main.ts:105 hex X'01020304'

Notice the byteOffest parameter for each row

E.g., byteOffset: 5352176

Each blob has the same byte offset which is certainly incorrect.

Doing a sqlSELECT quote(a) FROM foo; then casting a from hex to bytes returns the correct result (see console.log('QUOTE SELECTING') section). So for some strange reason sqlite3_column_text is not impacted by this bug.

IDBBatchAtomicVFS error: `TypeError: Cannot read properties of null (reading 'fileSize')`

Hi!

Thanks for the awesome library. I'm toying around with it to make a "full-stack" SQLite web application (SQLite persistence in browser and SQLite in the server). However, I encountered a strange error, trying to make IDBBatchAtomicVFS works. My code is something like this:

import * as IDBBatchAtomicModule from 'wa-sqlite/src/examples/IDBBatchAtomicVFS.js'

export async function hello() {
  const module = await SQLiteESMFactory()
  const sqlite3 = SQLite.Factory(module)
  const vfs = new IDBBatchAtomicModule['IDBBatchAtomicVFS']([])
  await vfs.isReady
  sqlite3.vfs_register(vfs, true)
  const db = await sqlite3.open_v2('mydb')
  const queryContent = (await import('../../../queries/01__snippet_creation.sql?raw')).default
  await sqlite3.exec(
    db,
    queryContent,
  )
  await sqlite3.close(db);
}

Where queryContent is something like:

CREATE TABLE IF NOT EXISTS snippets (
    id TEXT PRIMARY KEY,
    "name" TEXT,
    "language" TEXT,
    "text" TEXT,
    "encrypted" BOOLEAN,
    position REAL,
    updated_at DATETIME,
    created_at DATETIME
);

The error is like this:

TypeError: Cannot read properties of null (reading 'fileSize')
    at #xWriteHelper (IDBBatchAtomicVFS.js:247:40)
    at IDBBatchAtomicVFS.xWrite (IDBBatchAtomicVFS.js:232:17)
    at Ic (wa-sqlite.mjs?url:70:51)
    at wa-sqlite.wasm:0x70b37
    at wa-sqlite.wasm:0x57d2c
    at wa-sqlite.wasm:0x57ecf
    at wa-sqlite.wasm:0x2329
    at wa-sqlite.wasm:0x3f6f0
    at wa-sqlite.wasm:0x15241
    at wa-sqlite.wasm:0x5f3f7

Do you have any idea how does this error happen and what can I do to fix it? I'm using Vite, installing the library from GitHub with yarn install rhashimoto/wa-sqlite, but still serving the .mjs and .wasm files as static files.

Many thanks!

Trigger on multiple tabs

I'm attempting to utilize the demo, and it works fine on a single tab. I'm trying to create a trigger that fires upon insertion. However, in the case of multiple tabs, the trigger only fires on the tab where the insert operation is executed. I'm aware that I could use a SharedWorker to share the database connection across tabs. But in my case, I need to support many version of browser like safari 15 ( that doesn't support Sharedworker)

IndexedDB VFS xClose doesn't flush the cache

Right now the IndexedDB VFS cache only writes dirty blocks in xSync, not in xClose. xSync should always be called for database files, but I'm not certain whether that is true for journal files - I think it might depend on xDeviceCharacteristics flags (maybe SQLITE_IOCAP_SEQUENTIAL?).

working around dyncall issues

I ran into this issue:

https://github.com/emscripten-core/emscripten/pull/13883/commits

I was able to workaround with some minor changes in the Makefile:

dist/wa-sqlite.mjs: $(BITCODE_FILES) $(LIBRARY_FILES) $(EXPORTED_FUNCTIONS) $(EXTRA_EXPORTED_RUNTIME_METHODS)
        mkdir -p dist
        EMCC_CLOSURE_ARGS="--externs my_externs.js" $(EMCC) $(EMFLAGS) $(EMFLAGS_DIST) \
          $(EMFLAGS_INTERFACES) \
          $(EMFLAGS_LIBRARIES) \
          $(BITCODE_FILES) -o $@

dist/wa-sqlite-async.mjs: $(BITCODE_FILES) $(LIBRARY_FILES) $(EXPORTED_FUNCTIONS) $(EXTRA_EXPORTED_RUNTIME_METHODS) $(ASYNCIFY_IMPORTS)
        mkdir -p dist
        EMCC_CLOSURE_ARGS="--externs my_externs.js" $(EMCC) $(EMFLAGS) $(EMFLAGS_DIST) \
          $(EMFLAGS_INTERFACES) \
          $(EMFLAGS_LIBRARIES) \
          $(EMFLAGS_ASYNCIFY_DIST) \
          $(BITCODE_FILES) -o $@

And this tiny side file:

-- my_externs.js
// @externs
var dynCall_v;

Virtual table missing on demo page with IndexedDB VFS

The demo page provides access to four separate databases. The sample "array" module is installed in each one to provide access to some stock data directly from a Javascript array using a virtual table:

CREATE VIRTUAL TABLE IF NOT EXISTS goog USING array;

However, this is not working on the database using IndexedDbVFS. CREATE VIRTUAL TABLE doesn't return an error but attempting to access the table, e.g. with...

SELECT * FROM goog LIMIT 5;

...returns a "no such table: goog" error.

Support concurrent access to multiple databases

Hello! ✋
In our use case, we are managing several databases from our application concurrently.
We are taking into consideration the different VFSs of wa-sqlite, but after some testing, we realized that when we access concurrently different databases (no concurrent access to the same database), we encounter all sorts of possible errors (some listed below).

  • Is concurrent access to multiple databases supported?
  • If yes, what is the proper way to achieve it? Should we create a VFS for each DB?

I put together a tiny demo that helps us (and maybe someone else) replicate the concurrency problems, currently available here. It is similar to the original demo (access it at demo/concurrency.html), with the difference that it can be run on several databases concurrently and also protect them with a mutex. Interestingly, by accessing the different databases exclusively with a mutex, there are no errors.

My suspect is that asyncify + concurrency use breaks the sqlite implementation. In this case, the only options are either using a sync implementation (e.g. sync file handle pool) or prevent any concurrent access with a mutex (with possible hit on performance)

Here are some errors happening when running SQLite concurrently:

Uncaught (in promise) RuntimeError: memory access out of bounds
    at wa-sqlite-async.wasm:0xf0993
wa-sqlite.wasm:0x3fffa Uncaught (in promise) RuntimeError: null function or function signature mismatch
    at wa-sqlite.wasm:0x3fffa
OriginPrivateFileSystemVFS.js:116 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'filename')
    at OriginPrivateFileSystemVFS.js:116:30

...

Example VFS locking can have name collisions across filesystems

The example IndexedDB and OPFS VFS classes use WebLocks to implement locking. The current implementation locks a database file by acquiring a WebLock with the filename. This naming scheme is not specific enough because there may be database files with the same name across different filesystems in the same origin.

For example, there can be two IndexedDB databases, "idb1" and "idb2", with each containing an SQLite database named "foo.db". These SQLite databases will use the same named WebLock(s). This could cause unnecessary degraded performance.

The fix needs to made in each IndexedDB and OPFS VFS that uses WebLocks (so not AccessHandlePoolVFS, which doesn't need locks because it doesn't support concurrency) to make the lock name unique across filesystems and filenames. A workaround is to use unique SQLite database names if your application uses multiple filesystems.

PRAGMA synchronous=0; causes `database disk image is malformed` for `IDBBatchAtomic` VFS

When responding to #23 (comment) it made me wonder if setting PRAGMA schema.synchronous = 0 would have any impact on OPFS perf. IIUC, this would be similar behavior to relaxed durability that we have with IDB but with OPFS.

Running the wa-sqlite benchmarks with:

-- Pre-run setup
PRAGMA journal_mode=delete;
PRAGMA synchronous=0;

causes Test 6: Creating an index to always fail with

Error: database disk image is malformed
    at check (https://rhashimoto.github.io/wa-sqlite/src/sqlite-api.js:857:11)
    at Object.step (https://rhashimoto.github.io/wa-sqlite/src/sqlite-api.js:690:14)
    at async execute (https://rhashimoto.github.io/wa-sqlite/src/examples/tag.js:50:16)

Why are there synchronous and asynchronous builds?

WebAssembly currently doesn't have native support for coroutines or continuations. That makes it difficult to connect SQLite with browser resources that are inherently asynchronous, like IndexedDB.

Asyncify is a clever workaround for that restriction. It postprocesses the .wasm file and adds some library support to effectively suspend some WebAssembly calls, run some asynchronous Javascript, and resume executing the WebAssembly. That can be used to bridge C's customarily blocking i/o, which SQLite uses, and the browser's asynchronous APIs (though not the only way).

Asyncify is amazing and awesome, but it isn't free. It makes the WebAssembly substantially larger and slower, particularly with SQLite which is something of a worst case. Developers can decide whether the costs are acceptable for their application, but the asynchronous build should only be used if it is actually needed.

exclusive locking_mode in two tabs

I just tested my app with two tabs open, and on the second tab, DB access is always somehow locked. When I don't use exclusive locking, my app works as expected. Is it possible I am doing something wrong?

const asyncModule = await SQLiteAsyncESMFactory();
const sqlite3 = SQLite.Factory(asyncModule);
sqlite3.vfs_register(
  new IDBBatchAtomicVFS("evolu", { durability: "relaxed" })
);
const connection = await sqlite3.open_v2("app", undefined, "evolu");
// await sqlite3.exec(connection, "PRAGMA locking_mode=exclusive");

RuntimeError: memory access out of bounds - IDBBatchAtomicVFS

I am using the IDBBatchAtomicVFS in a chrome extension. I can install a quite large DB, and if I only send a few queries to the DB, they return fine. If I use IDBBatchAtomicVFS in a worker on the main website (so not in the chrome ext), then it is much slower than the OPFS sync and if I push it to hard, then it sometimes hangs, but I am not getting errors. However, in the chrome extension I can very reliably get these memory errors when I push it. It will start returning queries, then bork with this.

wa-sqlite-async-0af0ef34.wasm:0xf0993 Uncaught (in promise) RuntimeError: memory access out of bounds
    at wa-sqlite-async-0af0ef34.wasm:0xf0993
    at Oh.c.<computed> (wa-sqlite-async.mjs:48:321)
    at wa-sqlite-async.mjs:50:183

or

content.tsx-b9501ab3.js:64 Uncaught (in promise) RuntimeError: memory access out of bounds
    at chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/wa-sqlite-async-0af0ef34.wasm
    at Oh.c.<computed> [as _sqlite3_malloc] (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:109:12452)
    at W1.t.str_new (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:29712)
    at chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:29251
    at AsyncGenerator.next (<anonymous>)
    at n (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:32134)
    at chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:32408
    at Object.th [as getDefinitions] (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:50:94)
    at x (adapter-78aeae99.js:5:1236)
    at c (adapter-78aeae99.js:5:14906)

And sometimes this:

content.tsx-b9501ab3.js:64 Uncaught (in promise) Error: database disk image is malformed
    at S (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:31769)
    at Object.step (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:29575)
    at async n (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:107:32241)
    at async Object.th [as getDefinitions] (chrome-extension://mliddjnlkinmciggplpkdoiffahabapa/assets/dbs-2d7a9f74.js:50:88)

This method (so the query that gets executed against sqlite) has already been called successfully a number of times at this stage but borks after a few.

Alas, there is no sync in chrome extensions, so I'm not sure whether there are any other options I could try. I attempted to use the other IDB VFSes but I couldn't get any of them to get as far as installing my 300MB db. Any pointers much appreciated!

IndexedDbVFS rollback does not work unless cache_size=0

In the demo, the following SQL inserts two rows into a table, then inserts another row within a transaction that is rolled back:

PRAGMA journal_mode=delete;
PRAGMA cache_size=1024;
DROP TABLE IF EXISTS tbl;
CREATE TABLE IF NOT EXISTS tbl (x PRIMARY KEY, y);
REPLACE INTO tbl VALUES ('foo', 6), ('bar', 7);

BEGIN;
INSERT INTO tbl VALUES ('not', 'stored');
ROLLBACK;
SELECT * FROM tbl;

There should be only two rows in the table, but instead there are three. If cache_size is set to 0 then it works correctly.

Asyncify and sqlite3_interrupt

This is a pretty cool implementation of sqlite wasm. I am looking to abort long running queries. I saw your Faq page about asyncify and wondered if I use it and implement sqlite3_interrupt should I be able to achieve aborting long queries? Thanks for any pointers and heads up.

How do I use wa-sqlite in Node.js?

wa-sqlite is built as a set of ES6 modules. As ES6 modules generated by Emscripten
do not support Node.js very well, we have to employ a couple of workarounds for
the SQLiteESMFactory as documented in emscripten-core/emscripten#11792:

  • Set globals for __dirname and require()
  • Provide a locateFile() in the config object to the factory

The code snippet below illustrates how this is done:

  var SQLite = await import('wa-sqlite')
  var { default: SQLiteESMFactory } = await import('wa-sqlite/dist/wa-sqlite-async.mjs')

  // Remember the original globals
  const currentDirName = globalThis.__dirname
  const originalRequire = globalThis.require

  // Replace this with actual path to wa-sqlite/dist
  const sqliteDistPath = 'node_modules/wa-sqlite/dist' 

  // If we are using node, replace the globals below
  // with values that our module factory will use
  if (typeof process === 'object') {
    const { createRequire } = await import('module')
    globalThis.__dirname = sqliteDistPath
    globalThis.require = createRequire(__dirname + sqliteDistPath + '/wa-sqlite-async.mjs')
  }

  // Invoke the ES6 module factory (with a custom `locateFile` method for node)
  // to create the SQLite Emscripten module. This will fetch and compile the 
  // .wasm file.
  const options = typeof process === 'object'
    ? { locateFile: path => sqliteDistPath + '/' + path }
    : {}
  const module = await SQLiteESMFactory(options)

  // Reinstate the globals after creating the module
  globalThis.require = originalRequire
  globalThis.__dirname = currentDirName

  // Use the module to build the API instance.
  const sqlite3 = SQLite.Factory(module)

How can I use an ES6 module in a worker?

The wa-sqlite build produces ES6 modules, which is nice for development. But most applications should put time-consuming operations in a Worker, and support for ES6 modules in a Worker isn't universal yet. Worker code generally needs to be bundled.

Emscripten modules need to load their associated .wasm file, and the generated code for ES6 uses import.meta.url as its default location. import.meta.url is not valid outside an ES6 module, however, so the bundler has to translate this to something else. When the target context is a Worker, that something else needs to be valid in a Worker, so something like document.currentScript.src, which is the substitution my bundler chooses, causes a runtime exception.

The fix I use is to configure a bundler string replacement plug-in (@rollup/plugin-replace in my case) to replace import.meta.url with an empty string. There may be better approaches out there but this is sufficient for me so far.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.