Comments (6)
@Azarattum Just skimming it, it looks reasonable. You might not need locking if you don't have multiple connections accessing the database concurrently. I write very little for node so I don't know anything about locking or bundling there.
This is getting off-topic for this issue. If you have other questions I'd suggest moving to a new Discussion so people subscribed here don't get notifications.
from wa-sqlite.
Added to FAQ. Thanks!
from wa-sqlite.
Is there a VFS adapter that supports native filesystem with node:fs
?
from wa-sqlite.
Is there a VFS adapter that supports native filesystem with
node:fs
?
@Azarattum I have not written one. My interests are mainly in the browser, and I know there is already a way to use a native SQLite database from node (though I have never tried it myself) so it's not something I care much about.
It probably wouldn't be difficult for you to write one, but my guess is that a node extension calling a native SQLite library is going to be measurably faster than a WASM SQLite calling back to Javascript calling node:fs
.
from wa-sqlite.
@rhashimoto I've actually written one in the meantime. What do you think, does it look OKish? I wasn't sure how to implement locks though... I wanted to use wa-sqlite
over native bindings to achieve cross-platform portability. Ideally i would want my app to compile into a single .cjs
file that can be ran be Node. Do you have any idea on how that might be done? I'm using vite to bundle my project. However, it doesn't play with webassembly quite well...
node-vfs.ts
// @ts-ignore
import { access, FileHandle, open, rm } from "node:fs/promises";
import {
SQLITE_OPEN_DELETEONCLOSE,
SQLITE_IOERR_SHORT_READ,
SQLITE_OPEN_CREATE,
SQLITE_CANTOPEN,
SQLITE_IOERR,
SQLITE_OK,
Base,
// @ts-ignore
} from "@vlcn.io/wa-sqlite/src/VFS";
// @ts-ignore
import { resolve } from "node:path";
type File = { handle: FileHandle; flags: number; path: string };
type Result<T = number> = { set: (value: T) => void };
type Data = { size: number; value: Int8Array };
export class NodeVFS extends Base {
files = new Map<number, File>();
name = "node";
xOpen(path: string | null, id: number, flags: number, outFlags: Result) {
return this.handle(null, async () => {
path =
path ||
Math.floor(Math.random() * Number.MAX_SAFE_INTEGER).toString(36);
path = resolve(path) as string;
let mode = "r+";
try {
await access(path);
} catch (error) {
if (flags & SQLITE_OPEN_CREATE) mode = "w+";
else throw error;
}
try {
const handle = await open(path, mode);
this.files.set(id, { handle, flags, path });
outFlags.set(flags);
} catch {
return SQLITE_CANTOPEN;
}
});
}
xClose(id: number) {
return this.handle(id, async ({ handle, flags, path }) => {
this.files.delete(id);
await handle.close();
if (flags & SQLITE_OPEN_DELETEONCLOSE) await rm(path);
});
}
xRead(id: number, data: Data, offset: number) {
return this.handle(id, async ({ handle }) => {
const size = (await handle.stat()).size;
const start = Math.min(offset, size);
const end = Math.min(offset + data.size, size);
const length = end - start;
if (length) {
// @ts-ignore
const buffer = Buffer.alloc(length);
await handle.read(buffer, 0, length, start);
data.value.set(buffer);
}
if (length < data.size) {
data.value.fill(0, end);
return SQLITE_IOERR_SHORT_READ;
}
});
}
xWrite(id: number, data: Data, offset: number) {
return this.handle(id, async ({ handle }) => {
await handle.write(new Uint8Array(data.value), 0, data.size, offset);
});
}
xTruncate(id: number, size: number) {
return this.handle(id, async ({ handle }) => {
await handle.truncate(size);
});
}
xFileSize(id: number, outSize: Result) {
return this.handle(id, async ({ handle }) => {
outSize.set((await handle.stat()).size);
});
}
xDelete(path: string) {
return this.handle(null, async () => {
await rm(path);
});
}
xAccess(path: string, flags: number, accessOut: Result) {
return this.handle(null, async () => {
try {
await access(path, flags);
accessOut.set(1);
} catch {
accessOut.set(0);
}
});
}
handle<T extends number | null>(
id: T,
fn: (file: T extends number ? File : undefined) => Promise<void | number>
) {
return (this as any).handleAsync(() => {
const file = id ? this.files.get(id) : undefined;
if (id && !file) return Promise.resolve(SQLITE_IOERR);
return fn(file as any)
.then((code) => code || SQLITE_OK)
.catch(() => SQLITE_IOERR);
});
}
}
from wa-sqlite.
I changed the emscripten ENVIRONMENT option to "web,worker" to address reported issues with webpack. This may unfortunately cause the .mjs files not to load properly in node. If that happens the workaround should be to use a custom build with "node" added to ENVIRONMENT.
I apologize if this does break things for node users but there are more webpack users.
from wa-sqlite.
Related Issues (20)
- Support concurrent access to multiple databases HOT 3
- RuntimeError: memory access out of bounds - IDBBatchAtomicVFS HOT 5
- Racy IDBBatchAtomicVFS behavior when calling methods directly HOT 6
- PRAGMA synchronous=0; causes `database disk image is malformed` for `IDBBatchAtomic` VFS HOT 4
- Replace sqlite with libsql
- Corruption in vlcn fork with FTS5 HOT 8
- IDBBatchAtomicVFS error: `TypeError: Cannot read properties of null (reading 'fileSize')` HOT 4
- Strange error stack trace: both "SQL error" and "journal file not found" HOT 7
- Trigger on multiple tabs HOT 3
- Slow Query Execution for Tables with Diverse Data HOT 1
- Error: database disk image is malformed HOT 10
- Error: no such module: rtree HOT 2
- Feature request: add last_insert_rowid method
- sqlite3.exec throws "RuntimeError: memory access out of bounds" HOT 11
- Yarn install `Packing the package failed exit code 1` HOT 2
- Empty blobs as bind parameters being returned as NULLs HOT 2
- Full text support HOT 2
- Which VFS is the best?
- Call stack size exceeded of wasm asyncify HOT 7
- Example VFS locking can have name collisions across filesystems HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wa-sqlite.