Giter VIP home page Giter VIP logo

knarch.db's Introduction

KNArch.db

Move Along!

This project is archived. All Sqlite work on Kotlin/Native is happening in SQLiter.

For historical purposes, read all about KNarch.db below.

What is this?

This library is primarily an implementation of sqlite for iOS and MacOS, (roughly) compatible with Android, implemented with Kotlin/Native, to faciliate Kotlin multiplatform development.

KNArch?

K(otlin)N(ative)Arch(itecture)

Sounds like narc.

Build

The CI build is run on MS App Center. Currently it runs the Kotlin/Native iOS tests only, but that's the bulk of the framework. C++ builds are still done locally. Will be added to CI when K/N releases slow down a bit.

Looking for help to sort out multiplatform testing

Usage

To see a self-contained example, look at sample-notepad.

For a more robust example, including Sqldelight, see the Droidcon App.

The dependency mechanism of Kotlin Multiplatform is somewhat in flux, so refer to the previous 2 examples for current usage. You'll need to use Kotlin/Native 0.9.1, as 0.9.2 has incompatible dependency resolution. This is temporary until the new Multiplatform plugin is stable.

Status

There's a lot that's different about Kotlin on native platforms from Kotlin on the JVM. The original goal was to simply port AOSP logic, similar to Doppl. However, there are 2 major issues with that approach.

  1. Threading in K/N is very different. In demo-size projects, this isn't always obvious, but when you start thinking about what needs to exist for a production level app, this becomes a big issue.

  2. How architecture will evolve for multiplatform from "Android" isn't necessarily clear. But it should evolve. Simply copying artifacts is probably not the best choice.

The core database functionality works. On top of that, a core suite of tests from Android's CTS suite has been ported and applied. This is to verify that what the classes are supposed to do is what they actually do. The current goal is to create something that can be used in real, production apps by leveraging both AOSP-based logic and tests, to jumpstart the library.

With that in place, we will work on some next-stage goals simultaneously.

  1. Use the library in production application development to get feedback and refine the test suite and ensure a reliable core exists on which to base multiplatform mobile development.

  2. Refine the design. Changing the interface without a good reason isn't necessarily useful, but retaining everything from AOSP doesn't make sense either.

TL;DR Yes, put this in your app today. It works. There's a lot that's going to change, but if you're looking at Kotlin Multiplatform for your iOS implementation, I assume you have a high tolerance for change. Our goal is to have solid functionality testing to verify that things work as expected, and starting with the AOSP base gives us that.

Multiplatform

The MP portion of the framework is very much in flux. After getting iOS to function, the common code was minimally implemented to support testing in apps and providing a driver for SQLDelight. The common code is going to get a rethink, mostly around initialization, in the near future.

Differences

Due to the KN differences, primarily threading but others as well, the internals of the implementation have had fairly significant changes from the original code.

Threads

Threading is by far the most significant difference. KN is designed in such a way that you can't really share data between threads unless it is completely frozen. Android's SQLite stack has an inherently multithreaded design, which includes connection pools and explicit synchronization. Out of the box, this would be impossible with K/N.

For the initial release, all database access is single threaded. You can share the sqlite instance between threads, but when called from different threads, actual database calls are serialized.

FYI, this was Android's sqlite model until about 2.2/2.3. See my largely outdated answer on SO. Modern Android Sqlite allows concurrent reads, but writes are serialized, and only if you enable WAL. If you're not familiar with what I'm talking about, you almost certainly don't need to worry about serialized DB access and our implementation is plenty fine.

As this implementation stabilizes, pooling may be added back. We're going to push that decision off and let the framework and K/N itself mature a bit.

Transactions

To simplify the implementation, transactions block. That means if you neglect to call "endTransaction", you'll deadlock other threads that try to start transactions. If you neglect to call "endTransaction" in any db app you'll probably have issues, but you'll definitely have them here. This is a deliberate limitation, and will be reviewed when we revisit connection pools and WAL support.

This isn't a huge deal, but you definitely want to make sure you're not reading from the main thread if you're doing big transactions.

Transaction listeners

Transaction listeners are currently frozen, but we're going to make them thread local. Again, K/N and threads are not the same, and there are some rules to follow. You can't interact with a transaction from another thread currently, so making the listener thread local shouldn't be an issue. If you're using transaction listeners, it may be something to consider.

Attached databases

This functionality was removed during porting because of threading issues and how relatively uncommon this is. Will be added back.

Custom Functions

Not available. This is a relatively complex feature that will have C++ and K/N repercussions (threading, etc). Something to look into down the road if desired.

Design

Sqlite support in AOSP is implemented by Java classes that talk to C++ through JNI. We're using the same basic architecture. The C++ code from AOSP has been ported to use equivalent K/N types. See the cpp folder for that code.

In kotlin/kotlin-ios you'll find the public interface code. The api structure is roughly similar to the AOSP code, but there are differences and omissions because various things don't apply and/or aren't supported.

There will be updates to the design docs in the near future. Follow the Touchlab blog, kpgalligan on medium, or the Touchlab Twitter for info.

knarch.db's People

Contributors

kpgalligan avatar jasonjmcghee avatar

Stargazers

Ibn Mesbah avatar Josh Skeen avatar mengrong.yang avatar  avatar Hiroshi Kikuchi avatar Maxim avatar Daniel Eke avatar Filipe Uva avatar Eric avatar Nikita avatar Tim Rijckaert avatar rrodovalho avatar Yasuhiro SHIMIZU avatar Quang Luong avatar Pranav Lathigara avatar Vinayagasundar avatar Amine Bezzarga avatar Luca Spinazzola avatar Osama Ahmed Elsayed avatar Leonel Mendez Jimenez avatar rayworks avatar Jamolkhon Khakimov avatar Asif Patel avatar 鲁大师 avatar Spiros Economakis avatar Hannes Dorfmann avatar Pedro Veloso avatar Samuel Urbanowicz avatar  avatar Andreas Mattsson avatar 0x0 Khalid AlSubhi avatar Nuh Koca avatar Harshit Dwivedi avatar Murad avatar Piotr Wittchen avatar Aidan Follestad avatar Zach Klippenstein avatar Ersin Ertan avatar Hivana Alice avatar Eva Tatarka avatar Kirill Biakov avatar sivaprakash avatar Zachary Smith avatar Drew Carlson avatar Jay Newstrom avatar Andrei Mukamolau avatar Michal Havryluk avatar  avatar Dandan Meng avatar Peter Chislett avatar Kachi avatar Nelmer De La Cruz avatar Kevin Schildhorn avatar Sam Hill avatar Omniyyah avatar Takahiro Menju avatar Aleksey Mikhailov avatar Prince Chen avatar Jorge Coca avatar  avatar  avatar XiaoYiXiao369 avatar  avatar  avatar Gilberto Olimpio avatar  avatar Yossi Elkrief avatar tianyu.she avatar panpf avatar  avatar Pierre Haufe avatar César Díez Sánchez avatar Amr Abd El Wahab El Desouky avatar Mohan C M avatar 脉脉不得语 avatar Matthew Herod avatar Khairil Ushan avatar Kelvin Bulwinkel avatar Arpan Sarkar avatar Tarek Belkahia avatar  avatar Eduardo Pool avatar Gérard Paligot avatar Dayan Ruben avatar Philip K. Han avatar Justin Mancinelli avatar Yahor Berdnikau avatar  avatar Jeff Namnum avatar Ildar Karimov avatar

Watchers

 avatar Justin Mancinelli avatar Ildar Karimov avatar Peter Chislett avatar Jeff Namnum avatar Jay Newstrom avatar James Cloos avatar Adrian Opyrchał avatar Alec Kazakova avatar Eduardo Pool avatar Ben Whitley avatar Nick Todd avatar Jitty Andiyan avatar

knarch.db's Issues

Review SQLiteDatabase for synchronized

We share SQLiteDatabase among threads, but it's possible we're causing threading issues. Review synchronized blocks in java SQLiteDatabase and ensure we don't need mutexes. See SQLiteOpenHelper for an example

Fix dbconfig reset

Right now updating the db config closes and reopens the db connect, but that will fail because we're removing the dbconfig. Re-enable testEnableAndDisableForeignKeys and add some tests related to this process.

Update source licenses

Review all source files and make sure we have the appropriate licenses applied. In particular, anything that came out of AOSP. We also took some code from lrucache in C++ from an open source project, which isn't Apache 2, so lets make sure we have all the licensing straight.

Multithreading tests

We have some basic multithreading tests but could really use more comprehensive testing around multiple threads and K/N.

DbStats in connection

We've disabled the db stats functionality in connection. That should either be reenabled or removed.

Kotlin/Native errors can be pretty useless

There's a validation phase that can fail and give zero feedback. In this case, a @SymbolName annotation had the wrong name, which was duplicated. Should figure out if the KN team wants individual reports of this or if its not really something that can be easily parsed out.

> Task :knarch-ios:compileKonanKnarchIos_arm64
error: compilation failed: Assertion failed

 * Source files: ContentValues.kt, Cursor.kt, DatabaseErrorHandler.kt, Functions.kt, PlatformSQLiteOpenHelper.kt, SQLiteCursorDriver.kt, SQLiteDatabase.kt, SQLiteException.kt, SQLiteOpenHelper.kt, SQLiteProgram.kt, SQLiteQuery.kt, SQLiteStatement.kt, DefaultSystemContext.kt, Log.kt, SystemContext.kt, AbstractCursor.kt, AbstractWindowedCursor.kt, ContentValues.kt, CrossProcessCursor.kt, Cursor.kt, CursorIndexOutOfBoundsException.kt, CursorWindow.kt, DatabaseErrorHandler.kt, DatabaseUtils.kt, DefaultDatabaseErrorHandler.kt, Functions.kt, SQLException.kt, StaleDataException.kt, CppCursorWindow.kt, SQLiteClosable.kt, SQLiteConnection.kt, SQLiteCursor.kt, SQLiteCursorDriver.kt, SQLiteDatabase.kt, SQLiteDatabaseConfiguration.kt, SQLiteDatabaseCorruptException.kt, SQLiteDatabaseProvider.kt, SQLiteDebug.kt, SQLiteDirectCursorDriver.kt, SQLiteException.kt, SQLiteGlobal.kt, SQLiteOpenHelper.kt, SQLiteProgram.kt, SQLiteQuery.kt, SQLiteQueryBuilder.kt, SQLiteSession.kt, SQLiteStatement.kt, SQLiteStatementInfo.kt, SQLiteTransactionListener.kt, File.kt, IOException.kt, Printer.kt, StdPrinter.kt, TypeAliases.kt, Functions.kt, PlatformSQLiteOpenHelper.kt, TypeAliases.kt
 * Compiler version info: Konan: 0.8-dev / Kotlin: 1.2.60
 * Output kind: LIBRARY

exception: java.lang.AssertionError: Assertion failed
        at org.jetbrains.kotlin.backend.konan.llvm.Llvm.externalFunction$backend_native_compiler(ContextUtils.kt:299)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitFunction(LlvmDeclarations.kt:399)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoid$DefaultImpls.visitSimpleFunction(IrElementVisitorVoid.kt:52)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitSimpleFunction(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoid$DefaultImpls.visitSimpleFunction(IrElementVisitorVoid.kt:53)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitSimpleFunction(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitSimpleFunction(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.ir.declarations.impl.IrFunctionImpl.accept(IrFunctionImpl.kt:81)
        at org.jetbrains.kotlin.ir.declarations.impl.IrFileImpl.acceptChildren(IrFileImpl.kt:71)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoidKt.acceptChildrenVoid(IrElementVisitorVoid.kt:245)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitElement(LlvmDeclarations.kt:202)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoid$DefaultImpls.visitPackageFragment(IrElementVisitorVoid.kt:30)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitPackageFragment(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoid$DefaultImpls.visitFile(IrElementVisitorVoid.kt:37)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitFile(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoid$DefaultImpls.visitFile(IrElementVisitorVoid.kt:38)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitFile(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.backend.konan.llvm.DeclarationsGeneratorVisitor.visitFile(LlvmDeclarations.kt:147)
        at org.jetbrains.kotlin.ir.declarations.impl.IrFileImpl.accept(IrFileImpl.kt:68)
        at org.jetbrains.kotlin.ir.declarations.impl.IrModuleFragmentImpl.acceptChildren(IrModuleFragmentImpl.kt:49)
        at org.jetbrains.kotlin.ir.visitors.IrElementVisitorVoidKt.acceptChildrenVoid(IrElementVisitorVoid.kt:245)
        at org.jetbrains.kotlin.backend.konan.llvm.LlvmDeclarationsKt.createLlvmDeclarations(LlvmDeclarations.kt:33)
        at org.jetbrains.kotlin.backend.konan.llvm.IrToBitcodeKt.emitLLVM(IrToBitcode.kt:65)
        at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$4$2.invoke(KonanDriver.kt:97)
        at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$4$2.invoke(KonanDriver.kt)
        at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:139)
        at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:118)
        at org.jetbrains.kotlin.konan.util.UtilKt.profileIf(Util.kt:34)
        at org.jetbrains.kotlin.backend.konan.PhaseManager.phase$backend_native_compiler(KonanPhases.kt:138)
        at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$4.invoke(KonanDriver.kt:96)
        at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$4.invoke(KonanDriver.kt)
        at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:139)
        at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:118)
        at org.jetbrains.kotlin.konan.util.UtilKt.profileIf(Util.kt:34)
        at org.jetbrains.kotlin.backend.konan.PhaseManager.phase$backend_native_compiler(KonanPhases.kt:138)
        at org.jetbrains.kotlin.backend.konan.KonanDriverKt.runTopLevelPhases(KonanDriver.kt:90)
        at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:75)
        at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:42)
        at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:95)
        at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:50)
        at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:91)
        at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:69)
        at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:37)
        at org.jetbrains.kotlin.cli.common.CLITool$Companion.doMainNoExit(CLITool.kt:180)
        at org.jetbrains.kotlin.cli.common.CLITool$Companion.doMain(CLITool.kt:171)
        at org.jetbrains.kotlin.cli.bc.K2Native$Companion$main$1.invoke(K2Native.kt:197)
        at org.jetbrains.kotlin.cli.bc.K2Native$Companion$main$1.invoke(K2Native.kt:188)
        at org.jetbrains.kotlin.konan.util.UtilKt.profileIf(Util.kt:34)
        at org.jetbrains.kotlin.konan.util.UtilKt.profile(Util.kt:29)
        at org.jetbrains.kotlin.cli.bc.K2Native$Companion.main(K2Native.kt:190)
        at org.jetbrains.kotlin.cli.bc.K2NativeKt.main(K2Native.kt:202)
        at org.jetbrains.kotlin.cli.utilities.MainKt.main(main.kt:27)


> Task :knarch-ios:compileKonanKnarchIos_arm64 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':knarch-ios:compileKonanKnarchIos_arm64'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/bin/java'' finished with non-zero exit value 2

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 38s
5 actionable tasks: 2 executed, 3 up-to-date
Kevins-MacBook-Pro:knarch kgalligan$ ./buildall.sh build
~/temp4/knarch/cpp ~/temp4/knarch

SQLiteDatabaseConfiguration email regex

SQLiteDatabaseConfiguration replaces email addresses in log statements. During conversion that was just commented out, but should either go back and replace or pull it out fully. Not hard, but should test that it actually does what we want.

Multiplatform test coverage

Need to sort out multiplatform tests with K/N, then possibly add some tests for features added by the multiplatform, and some general sanity checks. Detailed sqlite testing is already provided by platform specific testing.

connection reconfigure

There's a 'reconfigure' method in the original connection code. We should review that all of that relevant logic is captured in other places.

SQLiteDatabase WriteAheadLogging

We're serializing connections to the database, so WAL isn't useful. Should either be removed completely or we should have multithreading enabled again.

Review CursorWindow data scheme

In Android and the c++ code, the data for the window is explicitly created and destroyed. In cases where 'close' isn't called explicitly, the finalize method will eventually free this memory. KN does not have a finalize. To achieve the same effect, we create a ByteArray in the Kotlin code, then give the pointer address of the start of that array, and when we want to clear that ByteArray, set it to null. It should free the memory immediately, but we should do some checking that this actually happens when expected.

Implement connection pool

The current implementation should work across threads in KN, but is very simplistic. We actually create a mutex lock on the session for transactions, so if you start a transaction, you really need to make sure you end it. Either this process will need very serious testing to ensure the app doesn't result in an eventual deadlock, or pooling needs to be implemented (or something else).

It is definitely a stopgap until we come up with a more robust solution.

Pack bitcode file into klib

The klib format is aware of native artifacts, but I can't figure out how to get them wrapped into klib by default. Either there should be a flag, or we'll need to manually modify the klibs, but once they're in there it'll be simpler to config downstream clients with "standard" gradle.

SQLiteGlobal defaultPageSize

We're just hard coding this, which isn't even correct according to defaults of sqlite. Should review if this is a value we want the system to be able to change at all. Current android code does a statfs to figure out block size, which would make sense.

Check that multiple threads have separate top level state

Using workers, all top level vars and objects have their own copy of state in different workers. This also appears to be true with threads that aren't started by a worker, but we should make sure that's true (and that the C++ layer state is global). This will impact things like ThreadLocal as more threading constructs are created.

Review and possibly refactor the statement cache

The db connection uses a statement cache with logic similar to the original, but the way we're storing and routing things is pretty different. It seems pretty possible that there are error flows that will leave statements hanging out in the database, which will ultimately cause issues when trying to disconnect. This isn't a super high priority, because if you're getting exceptions while creating statements you have other issues, but still. See acquirePreparedStatement

OperationLog in connection

The db connection object has 'OperationLog', which logs what the code is running. We removed it during the multithreading burndown effort. We should decide if this is still interesting to keep around at all, and implement in the C++ space.

Cancel is disabled

Cancellation signal was removed because it was explicitly multithreaded. We can probably figure out a new implementation of that, but for now we have some dead code floating around.

Allow attached databases

Attaching databases was removed, but now that we can store data across threads without issue, there's no real reason to have this functionality removed.

Review default param values

Removed default params on SQLiteDatabase to support multiplatform with KN compiler. Review if we want to support some of these defaults and either write some kind of wrapper or file bugs.

SQLiteCursor has no finalize

There's no destructor in kotlin native, and no finalizer, so cursor window doesn't get called if not called explicitly. You should always close your cursor, but there's no solution if it's not closed. either document or come up with an alternate solution.

"Dump" logic

There are a number of debugging "dump" methods in the sqlite stack, as well as the OperationLog in the connection object. These will be fairly useful for debugging, but aren't super critical for right now.

SQLiteQueryBuilder review

This is a very java thing. I don't think anybody is really going to use this as-is. Should update or remove.

sqldelight gradle plugin v1.0.0-kn0.9-a7 with kotlin-native-gradle-plugin:0.9 fails to build with "> org/jetbrains/kotlin/gradle/plugin/KotlinSourceSet"

When applying:
apply plugin: 'com.squareup.sqldelight'

while using kotlin-native-gradle-plugin:0.9 with your sqldelight build gradle-plugin:1.0.0-kn0.9-a7

I get the following error

A problem occurred evaluating project ':common'.
> org/jetbrains/kotlin/gradle/plugin/KotlinSourceSet

If you update sample-notepad-sqldelight versions:

    ext.versions = [
        'supportLibrary'  : '27.1.0',
        'kotlin'          : '1.3-M2',
        'kotlin_native_version' : '0.9',
        'kotlinCoroutines': '0.26.0-eap13',
        'knarchDb'      : '0.7.2-kn0.9-a2',
        'knarchThreads'      : '0.3.2-kn0.9-a2',
        'sqldelight'      : '1.0.0-kn0.9-a7'
]

you will also encounter the bug

This is releated to kotlin 1.3-M2's bug

this is fixed in kotlin native 9.1 ... but I am wondering if you have a work-around for building with kotlin native 9.0? (Just because all dependencies in a project have to be compiled with the same kotlin native version and I am using coroutines compiled with native 9.0 in my project)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.