tursodatabase / libsql-js Goto Github PK

The following should print false / true / false, but instead it always prints false:

    console.log(`db.inTransaction = ${db.inTransaction}`)
    db.transaction(() => {
        console.log(`db.inTransaction = ${db.inTransaction}`)
    })()
    console.log(`db.inTransaction = ${db.inTransaction}`)

If you attempt to run integration tests, you will see:

  beforeEach hook for Open in-memory database

  Rejected promise returned by test. Reason:

  Error {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
      '/Users/penberg/src/tursodatabase/libsql-js/index.js',
    ],
    message: `Cannot find module '@libsql/darwin-arm64'␊
    Require stack:␊
    - /Users/penberg/src/tursodatabase/libsql-js/index.js`,
  }

  › - /Users/penberg/src/tursodatabase/libsql-js/index.js
  › Object.<anonymous> (/Users/penberg/src/tursodatabase/libsql-js/index.js:42:5)

The issue is that we broke the tests in a bundler fix 3ba82a3.

We can fix the issue with the following, but that will break bunding again:

diff --git a/index.js b/index.js
index f92abb8..df0db81 100644
--- a/index.js
+++ b/index.js
@@ -39,7 +39,7 @@ const {
   statementColumns,
   statementSafeIntegers,
   rowsNext,
-} = require(`@libsql/${target}`);
+} = load(__dirname) || require(`@libsql/${target}`);

 const SqliteError = require("./sqlite-error");

diff --git a/promise.js b/promise.js
index 503cc61..ba0a26b 100644
--- a/promise.js
+++ b/promise.js
@@ -26,7 +26,7 @@ const {
   statementColumns,
   statementSafeIntegers,
   rowsNext,
-} = require(`@libsql/${currentTarget()}`);
+} = load(__dirname) || require(`@libsql/${currentTarget()}`);

 /**
  * Database represents a connection that can prepare and execute SQL statements.
  * ```

With the following knex dialect that runs db.sync() when a connection is opened:

const BaseClient = require("knex/lib/dialects/better-sqlite3");

class Client_Libsql extends BaseClient {
    static dialect = "libsql";

    _driver() {
        return require("libsql");
    }

    async acquireRawConnection() {
      const options = this.connectionSettings.options || {};

      console.log("Connecting with Turso")

      const db = new this.driver(this.connectionSettings.filename, options);
      db.sync();

      return db;
    }
}

Object.assign(Client_Libsql.prototype, {
    dialect: "libsql",
    driverName: "libsql",
});

module.exports = Client_Libsql;

An user reports that they get the following error when no database exists:

/tmp/testing/node_modules/libsql/index.js:90
    databaseSyncSync.call(this.db);
                     ^

Error: replication error: Injector error: SQLite error: database disk image is malformed
    at Database.sync (/tmp/testing/node_modules/libsql/index.js:90:22)
    at Client_Libsql.acquireRawConnection (/tmp/testing/src/db/turso-knex-client.js:17:10)
    at create (/tmp/testing/node_modules/knex/lib/client.js:262:39) {
  code: ''
}

But if they run the app again after the failure, it starts working fine.

The libSQL crate now supports BEGIN READ ONLY (tursodatabase/libsql@91f4780), let's wire it up in the transaction API.

Synchronization often suggests a two-way sync between two copies of data. What seems to be implemented here is a one-way "pull" of data missing in the embedded replica. Clients periodically "pull" new data into the database (similar to a git pull), and there is no concept of local changes to be pushed up to the service, since those are always delegated to the service.

Hello everyone,
I'm currently in the process of building an Expo app utilizing the Expo Router v3 API routes. Previously, I had been using Planetscale without any issues. However, upon attempting to migrate to Turso, I've encountered a bundling error:

Metro error: Unable to resolve module ./.targets from /Users/fefranco/code/libsql-expo/node_modules/libsql/index.js: 

None of these files exist:
  * node_modules/libsql/.targets(.web.ts|.ts|.web.tsx|.tsx|.web.js|.js|.web.jsx|.jsx|.web.json|.json|.web.cjs|.cjs|.web.mjs|.mjs|.web.scss|.scss|.web.sass|.sass|.web.css|.css)
  * node_modules/libsql/.targets/index(.web.ts|.ts|.web.tsx|.tsx|.web.js|.js|.web.jsx|.jsx|.web.json|.json|.web.cjs|.cjs|.web.mjs|.mjs|.web.scss|.scss|.web.sass|.sass|.web.css|.css)
   6 | // Static requires for bundlers.
   7 | if (0) {
>  8 |   require("./.targets");
     |            ^
   9 | }
  10 |
  11 | let target = currentTarget();


Call Stack
  Object.requireFileContentsWithMetro (node_modules/@expo/cli/src/start/server/getStaticRenderFunctions.ts:190:13)
  processTicksAndRejections (node:internal/process/task_queues)
  bundleAsync (node_modules/@expo/cli/src/start/server/metro/bundleApiRoutes.ts:39:26)
  getApiRoute (node_modules/@expo/cli/src/start/server/metro/createServerRouteMiddleware.ts:138:36)
  handler (node_modules/@expo/server/src/index.ts:178:20)
  <unknown> (node_modules/@expo/server/src/vendor/http.ts:36:24)
TypeError: Cannot read properties of null (reading 'GET')
    at handler (/Users/fefranco/code/libsql-expo/node_modules/@expo/server/src/index.ts:184:32)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /Users/fefranco/code/libsql-expo/node_modules/@expo/server/src/vendor/http.ts:36:24
Unable to resolve "./.targets" from "node_modules/libsql/index.js"

I've created a minimal reproduction of the issue in CodeSandbox

Although better-sqlite3 is great, the synchronous API makes it hard to make it work when we enable networking in libSQL. Therefore, let's switch to https://www.npmjs.com/package/sqlite-async API instead, which looks like better-sqlite3 but is async.

I use fastify, turso and drizzle-orm.
It works fine locally on my windows machine.

I'm trying to deploy my app to koyeb.com (they have AMD EPYC CPU).

Dockerfile

FROM node:18-alpine AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable
RUN corepack prepare pnpm@latest --activate
RUN pnpm i --frozen-lockfile

FROM node:18-alpine AS builder
WORKDIR /app
COPY . .
COPY --from=deps /app/node_modules ./node_modules
RUN npm run build

FROM node:18-alpine AS runner
WORKDIR /usr/app
COPY --from=builder /app/dist ./dist
COPY package.json ./
RUN npm i --omit=dev
USER node
ENV NODE_ENV="production"
CMD ["npm", "start"]

tsconfig.json
https://github.com/fastify/tsconfig/blob/master/tsconfig.json

After running npm start I get this error:

Error: Error relocating /usr/app/node_modules/@libsql/linux-x64-musl/index.node: fcntl64: symbol not found
    at Module._extensions..node (node:internal/modules/cjs/loader:1452:18)
    at Module.load (node:internal/modules/cjs/loader:1197:32)
    at Module._load (node:internal/modules/cjs/loader:1013:12)
    at Module.require (node:internal/modules/cjs/loader:1225:19)
    at require (node:internal/modules/helpers:177:18)
    at Object.<anonymous> (/usr/app/node_modules/libsql/index.js:42:5)
    at Module._compile (node:internal/modules/cjs/loader:1356:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)
    at Module.load (node:internal/modules/cjs/loader:1197:32)
    at Module._load (node:internal/modules/cjs/loader:1013:12) {
  code: 'ERR_DLOPEN_FAILED'
}

Same error with 18-alpine, 20-alpine, 21-alpine and npm/pnpm

UPD. Downgrading @libsql/client to 0.4.3 worked for me
Related issue: https://discord.com/channels/933071162680958986/1214203500939186186

We currently only support local database with Node, but with a WebAssembly build of libSQL can unlock that in other environments such as the browser. IOW, we need a libsql package similar to https://www.npmjs.com/package/@sqlite.org/sqlite-wasm and write that up to the client SDK.

Not sure if this is a bun or a libsql issue, but in earlier versions 0.3.10 this issue did not appear. A workaround is to lock the libsql version to 0.3.10 in your package.json.

Looks like missing execute_batch() in the libSQL remote protocol implementation:

penberg@vonneumann libsql-js % node examples/remote/example.js
thread '<unnamed>' panicked at 'not yet implemented', /Users/penberg/.cargo/git/checkouts/libsql-00c15cfa15b9f13b/c032b9f/libsql/src/hrana/mod.rs:277:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
zsh: abort      node examples/remote/example.js

The following fails due to an attempt to split statements on ";", even though it's valid SQL :

    db.exec("INSERT INTO t (value) VALUES ('x;y')`)

Similarly, SQL that attempts to create a trigger will fail because the trigger statements each need to end with a semicolon that's part of the greater SQL statement that creates it.

create trigger tr
after insert on tbl
for each row
begin
    update some stuff;  // this semicolon causes problems
end;

This failure is similar to the way the libsql shell previously failed on creating triggers when blindly splitting on semis: tursodatabase/libsql-shell-go#131

See functions js_exec_sync and js_exec_async, which both do the same thing.

Likely the only way to correctly handle semis in these situations is to use a tokenizing SQL parser that better understands SQLite syntax. It would be even better if this was a part of the common libsql core so all future client SDKs could use it without having to bring their own parser.

While things work OK for JavaScript, now TypeScript types are missing.

import Libsql from 'libsql-experimental'

Error:

error TS7016: Could not find a declaration file for module 'libsql-experimental'. 'path/to/node_modules/libsql-experimental/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/libsql-experimental` if it exists or add a new declaration (.d.ts) file containing `declare module 'libsql-experimental';`

Something changed with the addition of the promise API.

Gustavo asked the following

What to do about scripts (IOW "multiple statements"), such as when you want to load some DB dump?

LLRT is still in very very early stages, so it's not surprising that it's not working with turso/libsql.

It seems like it should be possible, and the main hurdle seems to be somehow bundling the correct binaries. I keep running into Cannot find module '@libsql/linux-arm64-gnu' and some minor syntax issues like awslabs/llrt#296

Installing libsql-experimental fails with Bun for some reason:

penberg@vonneumann foo % bun install libsql-experimental
bun add v0.7.0 (aa1ad7f0)
  🔍 @libsql-experimental/darwin-arm64 [1/2]
error: package "@libsql-experimental/darwin-arm64" not found registry.npmjs.org/@libsql-experimental/darwin-arm64 404
  🔍 @libsql-experimental/linux-x64-gnu [2/2]
error: package "@libsql-experimental/linux-x64-gnu" not found registry.npmjs.org/@libsql-experimental/linux-x64-gnu 404
error: @libsql-experimental/[email protected] failed to resolve
error: @libsql-experimental/[email protected] failed to resolve

I used the package in a lambda built via sst & esbuild. It seems the dependency is not included inside the output bundle.

Error: Cannot find module '@libsql/darwin-arm64'

I see that there is a special line to force bundlers to include the dependency, but the file ./targets seems to be missing, maybe it's related?

The build breaks with:

> cargo build --message-format=json | npm exec neon dist

   Compiling libsql-js v0.3.12 (/home/haaawk/libsql-js)
error: No artifacts were generated for crate libsqlibsql-js                                  
error: Broken pipe (os error 32)
warning: build failed, waiting for other jobs to finish..

This is due to the fact that 1.79.0-nightly (80d5b607d 2024-04-19) version of cargo does not have the following lines in JSON that were present in previous versions of cargo:

{"reason":"compiler-message","package_id":"path+file:///Users/haaawk/work/libsql-js-perf#[email protected]","manifest_path":"/Users/haaawk/work/libsql-js-perf/Cargo.toml","target":{"kind":["cdylib"],"crate_types":["cdylib"],"name":"libsql-js","src_path":"/Users/haaawk/work/libsql-js-perf/src/lib.rs","edition":"2021","doc":true,"doctest":false,"test":true},"message":{"rendered":"warning: unused variable:cx\n --> src/database.rs:290:24\n |\n290 | ...&self, cx: &mut FunctionContext) -> Option<Arc<Mutex<libsql::Connection>>> {\n | ^^ help: if this is intentional, prefix it with an underscore: _cx\n |\n = note: #[warn(unused_variables)] on by default\n\n","$message_type":"diagnostic","children":[{"children":[],"code":null,"level":"note","message":"#[warn(unused_variables)]on by default","rendered":null,"spans":[]},{"children":[],"code":null,"level":"help","message":"if this is intentional, prefix it with an underscore","rendered":null,"spans":[{"byte_end":11024,"byte_start":11022,"column_end":26,"column_start":24,"expansion":null,"file_name":"src/database.rs","is_primary":true,"label":null,"line_end":290,"line_start":290,"suggested_replacement":"_cx","suggestion_applicability":"MaybeIncorrect","text":[{"highlight_end":26,"highlight_start":24,"text":" fn get_conn(&self, cx: &mut FunctionContext) -> Option<Arc<Mutex<libsql::Connection>>> {"}]}]}],"code":{"code":"unused_variables","explanation":null},"level":"warning","message":"unused variable:cx","spans":[{"byte_end":11024,"byte_start":11022,"column_end":26,"column_start":24,"expansion":null,"file_name":"src/database.rs","is_primary":true,"label":null,"line_end":290,"line_start":290,"suggested_replacement":null,"suggestion_applicability":null,"text":[{"highlight_end":26,"highlight_start":24,"text":" fn get_conn(&self, cx: &mut FunctionContext) -> Option<Arc<Mutex<libsql::Connection>>> {"}]}]}}

{"reason":"compiler-message","package_id":"path+file:///Users/haaawk/work/libsql-js-perf#[email protected]","manifest_path":"/Users/haaawk/work/libsql-js-perf/Cargo.toml","target":{"kind":["cdylib"],"crate_types":["cdylib"],"name":"libsql-js","src_path":"/Users/haaawk/work/libsql-js-perf/src/lib.rs","edition":"2021","doc":true,"doctest":false,"test":true},"message":{"rendered":"warning: 2 warnings emitted\n\n","$message_type":"diagnostic","children":[],"code":null,"level":"warning","message":"2 warnings emitted","spans":[]}}

{"reason":"compiler-artifact","package_id":"path+file:///Users/haaawk/work/libsql-js-perf#[email protected]","manifest_path":"/Users/haaawk/work/libsql-js-perf/Cargo.toml","target":{"kind":["cdylib"],"crate_types":["cdylib"],"name":"libsql-js","src_path":"/Users/haaawk/work/libsql-js-perf/src/lib.rs","edition":"2021","doc":true,"doctest":false,"test":true},"profile":{"opt_level":"z","debuginfo":0,"debug_assertions":false,"overflow_checks":false,"test":false},"features":[],"filenames":["/Users/haaawk/work/libsql-js-perf/target/release/liblibsql_js.dylib"],"executable":null,"fresh":true}

Currently, it only executes the first statement. e.g. db.exec("create table foo (x); create table bar (x);") only creates the foo table and ignores everything after the first semicolon.

I'm using raspberry Pi with Bunjs, try to do import libsql but it throws error

error: Cannot find module "@libsql/linux-arm64-musl" from "/root/dev/bunjs/node_modules/libsql/index.js"

I'm getting the following errors when using with knex.

Im using the database export from the pormise.js file

1. statement.run is undefined. i noticed this when running knex.raw()
2. types are not exported of promise.js so in a ts file import Database from 'libsql/promise'

Hi, I am new to JS so it is possible I am missing something.

• I am using the RANDOM ROWID feature with libsql. When this is enabled I cannot retrieve items using the lastInsertRowid property from stmt.run() commands after insertion. The lastInsertRowid is slightly off from the actual rowid. However using stmt.all() after a select * ... statement I can get the correct rowids.
• Looking at the tests they do not seem to cover the BigInt/SafeIntegers use case.

Here is example code to reproduce the issue:

import Database from 'libsql';

const db = new Database(':memory:');
db.defaultSafeIntegers(true);
db.exec(`
  DROP TABLE IF EXISTS users;
  CREATE TABLE users (rowid INTEGER PRIMARY KEY, name TEXT) RANDOM ROWID;`);

const stmt_insert = db.prepare("INSERT INTO users(name) VALUES (?)");
const info_insert = stmt_insert.run('alma');
console.log([typeof info_insert.lastInsertRowid, info_insert.lastInsertRowid]);
const stmt_list = db.prepare("SELECT * from users");
const info_list = stmt_list.all()
console.log([typeof info_list[0].rowid ,info_list]);

and sample output:

$ node ex.js
[ 'number', 4504423795271622000 ]
[ 'bigint', [ { rowid: 4504423795271622254n, name: 'alma' } ] ]

Running new Database with a configuration that wants to use a remote sqld will cause the program to block on disk or network I/O for some amount of time, potentially for very long amounts of time. It can also cause the constructor to fail. There are two problems with this off the top of my head:

• It's highly unexpected for a constructor to block in JS, especially for setups that want to eagerly create objects for later use, or expect that their DI never blocks or fails, and always provides an object.
• It prevents the program from working at all if something is wrong establishing a connection to the server at the time the constructor is called (unless the requirement is that the caller should catch and retry errors with the constructor, which I don't like).

It would be more clean if the constructor only failed on configuration or programming errors (or I suppose whatever might make the underlying better-sqlite3 constructor fail, but it might be reasonable to delay that as well). This would allow the program to work if initially offline or without connection, and make error recovery easier to implement. As an alternative:

• The request to pull new data could lazily contact the server and yield an error message at that point.
• The SDK could eagerly contact the server off the main thread so that it's quickly ready for the first sync.

The types promise in

https://github.com/libsql/libsql-js/blob/8c56620c00550609129b6bd62bdd6651bc97e842/types/index.d.ts#L24

that each statement knows if it returns data. But in reality the prop does not exist and is always undefined.

Add support for the readonly option that's passed to the Database constructor. Needed by, for example, Knex.

We need to close the underying connection.

Example:

import Database from "libsql";
const opts = {
    authToken: process.env.TURSO_AUTH_TOKEN || ''
} as any;

const db = new Database(process.env.TURSO_DATABASE_URL || '', opts)
console.log(db.prepare('select invalidcolumn')); //doesn't throw error;
console.log(db.prepare('select * as')); //throw error

Using the 'better-sqlite' driver it throws an error in both situations.

With #54, the sync example fails as follows:

penberg@vonneumann sync % node example.js
2023-11-01T09:20:21.706640Z  INFO libsql_replication::replicator: Attempting to perform handshake with primary.
2023-11-01T09:20:21.706790Z  INFO libsql::replication::remote_client: Attempting to perform handshake with primary.
2023-11-01T09:20:21.972033Z ERROR libsql_replication::replicator: error connecting to primary. retrying. error: Replicator client error: invalid length: expected length 32 for simple format, found 0

Hello,
I am using @sveltejs/kit with @sveltejs/adapter-node and that kinda forces me to go ESM. And ESM does not like __dirname and similar stuff.

This line causes me troubles - https://github.com/libsql/libsql-js/blob/main/index.js#L33

What is to solution to this? Create ESM index.js?
I might be also very wrong and this issues is completely unrelated.

I've been stuck with this error for some time and it feels like I am getting out of options. Any help and ideas appreciated.
Thank you 🙏

We have bunch of unwrap()calls in the code, let's fix them up...

There's preliminary support for promise API, but we still need to do asyncify the following:

[ ] Database.open()
[x] Database.sync()
[ ] Statement.run()
[ ] Make Statement.iterate() return an async iterator?

SQLite supports a variety of styles for bound parameters:

• ?
• ?NNN
• :VVV
• @ VVV
• $VVV

With 0.18, this library only supports the first (positional) and fourth (:) styles, while better-sqlite3 supports them all.

One thing that took me by surprise is the way the second (positional/indexed) style works with better-sqlite3. You pass the values in an object as if they were named, like this:

const insert = db.prepare('INSERT INTO users VALUES (?3, ?2, ?1)')
insert.run({
    2: 'Alice',
    1: '[email protected]',
    3: 2,
})

Specifying them as varargs to run() or an array just doesn't work at all.

We currently keep the libsql types in-tree, but let's migrate them to a @types/libsql package instead.

import Database from 'libsql-experimental'
const db = new Database(":memory:")
console.log(db.memory)

The above prints "false" when it should instead print "true" as it does with better-sqlite3.

With better-sqlite3, the readonly option behaves as such:

• Constructor fails when combined with an in-memory database

TypeError: In-memory/temporary databases cannot be readonly

• Queries that mutate fail

SqliteError: attempt to write a readonly database

• db.readonly set to true

This library does none of the above yet.