jank-lang / jank Goto Github PK
View Code? Open in Web Editor NEWA Clojure dialect hosted on LLVM with native C++ interop
Home Page: https://jank-lang.org
License: Mozilla Public License 2.0
A Clojure dialect hosted on LLVM with native C++ interop
Home Page: https://jank-lang.org
License: Mozilla Public License 2.0
We use immer for persistent, immutable data structures in C++ and we use vcpkg to fetch our C++ packages at build time. Unfortunately, the latest vcpkg version for immer doesn't have some new features he's released, like map transients. Updating this would allow us to use those.
This is used by the clojure.core/repeat
fn. Should be a straightforward runtime object to cook up, as well as implementing the core fn itself.
When running
./bin/configure -GNinja -DCMAKE_BUILD_TYPE=Debug -Djank_cling_build_dir=build/cling-build
-- Running vcpkg install
I get the following error:
-- Running vcpkg install - failed
CMake Error at third-party/vcpkg/scripts/buildsystems/vcpkg.cmake:893 (message):
vcpkg install failed. See logs for more information:
/home/mohv/src/jank/build/vcpkg-manifest-install.log
Call Stack (most recent call first):
/usr/share/cmake-3.22/Modules/CMakeDetermineSystem.cmake:124 (include)
CMakeLists.txt:28 (project)
here is the full log
Detecting compiler hash for triplet x64-linux...
Detecting compiler hash for triplet x64-clang-static...
The following packages will be built and installed:
folly[core,zlib]:x64-clang-static -> 2023.05.22.00#1
immer:x64-clang-static -> 0.8.0#1
libguarded:x64-clang-static -> 2019-08-27#3
magic-enum:x64-clang-static -> 0.9.2
Restored 0 package(s) from /home/mohv/.cache/vcpkg/archives in 8.82 us. Use --debug to see more details.
Installing 1/4 folly:x64-clang-static...
Building folly[core,zlib]:x64-clang-static...
-- [OVERLAY] Loading triplet configuration from: /home/mohv/src/jank/vcpkg-triplet/x64-clang-static.cmake
-- Using cached facebook-folly-v2023.05.22.00.tar.gz.
-- Cleaning sources at /home/mohv/src/jank/third-party/vcpkg/buildtrees/folly/src/3.05.22.00-f3a2ac5bcf.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/mohv/src/jank/third-party/vcpkg/downloads/facebook-folly-v2023.05.22.00.tar.gz
-- Applying patch reorder-glog-gflags.patch
-- Applying patch disable-non-underscore-posix-names.patch
-- Applying patch boost-1.70.patch
-- Applying patch fix-windows-minmax.patch
-- Applying patch fix-deps.patch
-- Applying patch openssl.patch
-- Applying patch strong-symbols.patch
-- Using source at /home/mohv/src/jank/third-party/vcpkg/buildtrees/folly/src/3.05.22.00-f3a2ac5bcf.clean
-- Configuring x64-clang-static
-- Building x64-clang-static-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:134 (message):
Command failed: /home/mohv/src/jank/third-party/vcpkg/downloads/tools/cmake-3.25.1-linux/cmake-3.25.1-linux-x86_64/bin/cmake --build . --config Debug --target install -- -v -j3
Working Directory: /home/mohv/src/jank/third-party/vcpkg/buildtrees/folly/x64-clang-static-dbg
See logs for more information:
/home/mohv/src/jank/third-party/vcpkg/buildtrees/folly/install-x64-clang-static-dbg-out.log
Call Stack (most recent call first):
/home/mohv/src/jank/build/vcpkg_installed/x64-linux/share/vcpkg-cmake/vcpkg_cmake_build.cmake:74 (vcpkg_execute_build_process)
/home/mohv/src/jank/build/vcpkg_installed/x64-linux/share/vcpkg-cmake/vcpkg_cmake_install.cmake:16 (vcpkg_cmake_build)
ports/folly/portfile.cmake:71 (vcpkg_cmake_install)
scripts/ports.cmake:147 (include)
error: building folly:x64-clang-static failed with: BUILD_FAILED
Elapsed time to handle folly:x64-clang-static: 3.8 min
Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+folly
You can submit a new issue at:
https://github.com/microsoft/vcpkg/issues/new?title=[folly]+Build+error&body=Copy+issue+body+from+%2Fhome%2Fmohv%2Fsrc%2Fjank%2Fbuild%2Fvcpkg_installed%2Fvcpkg%2Fissue_body.md
-- Running vcpkg install - failed
CMake Error at third-party/vcpkg/scripts/buildsystems/vcpkg.cmake:893 (message):
vcpkg install failed. See logs for more information:
/home/mohv/src/jank/build/vcpkg-manifest-install.log
Call Stack (most recent call first):
/usr/share/cmake-3.22/Modules/CMakeDetermineSystem.cmake:124 (include)
CMakeLists.txt:28 (project)
CMake Error: CMake was unable to find a build program corresponding to "Ninja". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
-- Configuring incomplete, errors occurred!
Right now, jank reads single-file applications and it compiles clojure.core
prior to the user code every time. Adding support for require
will also entail the rest of the details around module loading and caching. This sets us up for being used within leiningen.
require
alias
supportWe won't be able to add the full ns
macro until syntax quoting is done.
The hack I put in to make ∀ highlight is not very good. It highlights ∀ even if it's in the middle of another symbol. It also doesn't play nicely with word boundaries, which makes the typical case of (∀)
pretty annoying.
Vim doesn't allow adding non-ASCII characters to the iskeyword set, so this will either take a change in vim or a bunch of manual regexing. And that's just for the coloring. I'm not sure how to get past the word boundary issue...
I use submodules in my .vim dir to manage my plugins which doesn't really work well with the current jank setup. And since I use vim in different environments, it's not feasible to hard code a symlink or mess with environment variables.
I followed the compiling instructions and got a "hello world" working on my Mac! Hurray!
One thing: I had to type
./bin/configure -Djank_cling_build_dir=build/cling-build
I needed three attempts to figure this out, the info in the Readme led me to
plain ./bin/configure
and then ./bin/configure -Djank_cling_build_dir=cling-build
.
Maybe it makes sense to clarify this in the Readme for Clojurians (like me) who in the last decade had used make
and C++
maybe once and now building Jank with "fingers crossed".
Reference: https://clojure.org/reference/transients
We have transient data structures, just not runtime objects.
Then we need the necessary core fns.
transient
persistent!
assoc!
dissoc!
conj!
pop!
disj!
As mentioned in #24, CLJS uses ~{}
and there's no good reason for jank not to as well. There's some remaining work right now to improve the parsing of interpolated expressions, so that will likely be included along with this ticket.
interesting project.
suggestion:
r.e. static typing & templates
have you considered trying to retrofit type annotations in a way which fit the syntax of an existing lisp
e.g.
start out with untyped code that looks like clojure, generating C++ templates:
(defn foo [x y z]...)
compiles to template<typename X,typename Y, typename Z> auto foo(X x,Y y, Z z){...}
i.e. an argument without any annotation gets an anonymous type parameter.
Then come up with a retrofitted syntax to add annotations for named type-parameters and actual types (perhaps a parenthesised pair in the argument list could specify (argname type)
e.g. (defn foo [(a int)(b int) c]...)
would compile to template<typename C> auto foo(int a,int b, C c){...}
clojures' syntax is pretty interesting IMO, the special use of [] and {} does give your eye something specific to latch onto whilst still being easy to parse and not too far from other lisps'.
another possible inspiration for a type annotation syntax is http://adambard.com/blog/core-typed-vs-haskell/
I do realise 'jank' is NOT designed to be compatible with any existing lisp, being a whole new language with unique goals, but maybe if you more closely follow an existing syntax you're more likely to find other people can intuitively read it - 'principle of least surprise'.
I notice ':' having use in your syntax for types, but in lisps that gets used for keywords. Perhaps you could retain lisp-like keywords and a clojure-like syntax where {:x 0 :y 1 :z 2} is like an 'anonymous struct constructor', initializing named fields, and keyword arguments (which seem like a really nice feature in lisps).
Recently, I set up automatic releases, based on the last successful commit: https://github.com/jank-lang/jank/releases
These are just a tarball of the result of using cmake to install all necessary files. In order to make this easy to actually install on another machine, we'd really benefit from continuous packages for various distros and OSs. I'd say start first with whatever you use, if you want to do this.
jank currently uses folly's string, but they're quite slow to construct and don't support custom allocators (meaning they leak like crazy right now). I think we'll need a custom string, but it can follow what folly is doing, if that doesn't impact allocation speed.
Hi!
Looking forward to this project!
I'd like to discuss if you already started working on anything related to LSP support, I'm maintainer of clojure-lsp and have a lot of experience with LSP protocol and editors using it so if you agree I can try to help with that :)
Some questions:
This may already exist somewhere, but jank could really use a fully clojure test suite for every clojure.core function. This would be beneficial for all clojure dialects, so perhaps we keep it in a separate repo. Work on this can start prior to jank actually supporting all of those functions. Usage of clojure.test makes sense, as long as there's no interop.
If this doesn't exist, it'd be quite the undertaking, due to the size of clojure.core. However, it'd also be an excellent sweat bed for jank to run continuously. Worth the effort.
Clojure’s object model is intense. In fact, representing it 1:1 from Java to C++ is impossible, since C++ has stricter rules are duplicate base classes and bases with same-name fields and different types. Furthermore, C++, even with the Boehm GC, is much slower at creating objects than the JVM is. It’s bread and butter work for the JVM. Also, while everything is an Object in the JVM, C++ doesn’t have the same notion. If every jank class were to inherit from the same Object class, it would have very serious performance implications when it comes to allocations.
So jank has two key problems here:
So far, I’ve worked around the first one by optimizing other things, so jank can be faster than Clojure in a given program for example, but when measuring just the object creations, Clojure is still around twice as fast. I want to fix this.
This task would entail implementing and benchmarking a few different solutions, all of which move jank’s object model away from C++ inheritance and toward something more dataoriented. This gets us around C++’s virtual inheritance allocation costs, but it can also allow every jank type to be treated as an object, which will not only simplify jank’s runtime code, but will itself be a key optimization.
Right now, I have two key design ideas:
So far, I have prototyped the first approach and found object creation is nearly twice as fast. This funding would allow me to implement both of these solutions fully, benchmark them, and research further ways to improve them. Finally, I will integrate the fastest solution into jank and reap the benefits.
Here's what I got:
/home/lyh/Documents/CS/jank/jank/build/llvm/include/llvm/Support/Signals.h:119:24: error: unknown type name 'uintptr_t'; did you mean '__intptr_t'?
/home/lyh/Documents/CS/jank/jank/build/llvm/lib/Support/Unix/Signals.inc:348:11: error: out-of-line definition of 'CleanupOnSignal' does not match any declaration in namespace 'llvm::sys'
folly, a dependency coming in through vcpkg, fails to build on macOS in CI. I don't have a mac for easy testing to figure out what's going on, so I've disabled macOS builds in CI for now. An example log of the failure is here: https://github.com/jank-lang/jank/actions/runs/4029054748/jobs/6926556586
Help would be appreciated, by anyone with a mac and some interest or knowledge of vcpkg, folly, and C++ compilation.
Automated releases for macOS are blocked on this, naturally.
can you point me to http client / server examples please?
I would love to have automatic formatting on save enabled; it would make accepting C++ PRs much easier, too. However, I haven't been able to get clang-format to format C++ in a way that matches jank's current style. It may require changes to clang-format in order to do this. Or perhaps some other tool can do it better.
Clojure supports unboxed signatures for fns with up to 4 params. jank can do the same thing, but it comes in two steps:
Right now, jank tracks boxing info, but not type info. This may be required, if we support unboxed things which aren't numbers. I'm not yet sure if we will.
Alex Miller covers an example of this here: https://youtu.be/s_xjnXB994w?t=2529
Now that #27 is done, we can add a hash map with transients into jank.
hash-map
function to clojure.core
map?
This is blocked on #37, which is changing the whole object model. Makes sense to only do this afterward.
Haven't dived in yet but really love the concept and look forward to seeing it develop. Not crazy about the name but I appreciate the ambition either way! Hope you don't mind but wanted to show some support.
As title, thanks.
I'm not sure how this will work yet, if jank will have a lein plugin or if something else is required. But jank will be adding support for require
shortly, so we should be able to start hooking in deps, generating compilation artifacts, etc.
For now, blocked on #31.
run-main
command to jankproject.clj
which get passed to jank as command-line flags (see its currently supported flags and expose all of those)NOTE: Keep in mind that jank will load .jank
or .cljc
files, but not .clj
files. Just like ClojureScript.
run-main
commandThe jank CLI options are defined here: https://github.com/jank-lang/jank/blob/main/include/cpp/jank/util/cli.hpp
The source to process them is here: https://github.com/jank-lang/jank/blob/main/src/cpp/jank/util/cli.cpp
Our run-main
command will be very similar to the run
command, but will load a module, rather than a file. We're using a new command, rather than a flag, since there's no point in loading a specific file from the lein project; we want to load a module, based on the class path we have.
I'd duplicate the run
function we have into a run_main
and use the rt_ctx
to load the module. https://github.com/jank-lang/jank/blob/main/src/cpp/main.cpp#L29
We'll need a new transient string target_module
option to go along with the command.
With out module loaded, we'll need to find the var for -main
within that class and invoke it. The rt_ctx
can be used to find the var. You can use dynamic_call
on that to invoke it (don't forget to deref
the var to get its root).
Passing in the command-line arguments will be more difficult, since we'll need to add CLI support for a --
flag, which then consumes everything after it. jank uses CL11 for flag handling, so you'll need to check out how to do --
there: https://github.com/CLIUtils/CLI11
The final command for lein jank run foo bar spam
should look like this:
jank run-main my.program --class-path <...> -- foo bar spam
Hi,
I got jank built in Arch on WSL, but got the following error when running the tests:
===============================================================================
/home/chris/repos/jank/test/cpp/jank/jit/processor.cpp:33:
TEST CASE: Files
/home/chris/repos/jank/test/cpp/jank/jit/processor.cpp:33: FATAL ERROR: test case CRASHED: SIGABRT - Abort (abnormal termination) signal
===============================================================================
[doctest] test cases: 1 | 0 passed | 1 failed | 31 skipped
[doctest] assertions: 0 | 0 passed | 0 failed |
[doctest] Status: FAILURE!
==27405==
==27405== Process terminating with default action of signal 6 (SIGABRT)
==27405== at 0x9EF98EC: __pthread_kill_implementation (pthread_kill.c:44)
==27405== by 0x9EAAEA7: raise (raise.c:26)
==27405== by 0x9E9453C: abort (abort.c:79)
==27405== by 0x9BCA832: __gnu_cxx::__verbose_terminate_handler() [clone .cold] (vterminate.cc:95)
==27405== by 0x9BD6D0B: __cxxabiv1::__terminate(void (*)()) (eh_terminate.cc:48)
==27405== by 0x9BD6D78: std::terminate() (eh_terminate.cc:58)
==27405== by 0x9BD7AB6: __cxa_pure_virtual (pure.cc:50)
==27405== by 0x5A34525: clang::CodeGen::CodeGenTBAA::getBaseTypeInfoHelper(clang::Type const*) (in /home/chris/repos/jank/build/libjankcling.so)
==27405== by 0x5A337BC: clang::CodeGen::CodeGenTBAA::getBaseTypeInfo(clang::QualType) (in /home/chris/repos/jank/build/libjankcling.so)
==27405== by 0x5A342D2: clang::CodeGen::CodeGenTBAA::getBaseTypeInfoHelper(clang::Type const*) (in /home/chris/repos/jank/build/libjankcling.so)
==27405== by 0x5A337BC: clang::CodeGen::CodeGenTBAA::getBaseTypeInfo(clang::QualType) (in /home/chris/repos/jank/build/libjankcling.so)
==27405== by 0x5A342D2: clang::CodeGen::CodeGenTBAA::getBaseTypeInfoHelper(clang::Type const*) (in /home/chris/repos/jank/build/libjankcling.so)
Got any ideas what might be wrong? I could provide more info if you think it would help 👍
Docs: https://clojure.org/guides/weird_characters#_character_literal
This will need support in a few areas:
We currently run the in-source tests after compiling, but it would be helpful to smoke test the packaged distributions to ensure all necessary files are in there, in the right place, and that the basic functionality works.
Right now, we just have a compiler suite. The runtime suite should use a jank implementation of clojure.test
to test the whole of clojure.core
and other namespaces shipped with jank. The runtime suite is blocked on #31.
Hi!
i am trying to build jank on guix
just run bin/configure and got this
-- Running vcpkg install
-- Running vcpkg install - failed
CMake Error at third-party/vcpkg/scripts/buildsystems/vcpkg.cmake:863 (message):
vcpkg install failed. See logs for more information:
/home/moon/code/jank/build/vcpkg-manifest-install.log
Call Stack (most recent call first):
/gnu/store/5p4fnymnslnydibk7qcisal3443w09pm-cmake-minimal-3.21.4/share/cmake-3.21/Modules/CMakeDetermineSystem.cmake:124 (include)
CMakeLists.txt:17 (project)
CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
jank/build/vcpkg-manifest-install.log
is empty
Thanks!
Wondering if it's time to request an "official" reader conditional key and file extension.
https://clojure.org/guides/reader_conditionals
#?(:clj (Clojure expression)
:cljs (ClojureScript expression)
:cljr (Clojure CLR expression)
:default (fallthrough expression))
Maybe :jank
sounds too project specific, maybe :cljl
? Kinda hard to read.
jank has support for a new special form, called native/raw
. It works in place of Clojure's interop syntax and allows for inline C++. But it also support interpolating jank expressions into that C++. Docs on the rationale and final solution are here: https://github.com/jank-lang/jank/blob/main/DESIGN.md#interop
Right now, all of this gets highlighted as a string, in vim/emacs/vscode, but the interpolated forms are Clojure code and should be highlighted accordingly. Ideally normal code completion, repl behavior, etc can work from within those forms, but we can take this one step at a time.
We may be able to make the syntax highlighting changes here in https://tree-sitter.github.io/tree-sitter/ and call it a day. But we might also consider getting into the vim/emacs/vscode configurations for Clojure and then forking them for jank to add this support. Would be your call on how to tackle this. I use vim and would love it to have this working, but we'll want great tooling for everyone, so might as well start with whatever you use.
Most lisp-like languages have one indentation scheme (no puns), which is not up for debate. jank is the same way, and it follows clojure's indentation almost to the letter.
Vim's clojure files are here https://github.com/guns/vim-clojure-static but the indentation logic is ridiculous, so I haven't bothered yet.
In short, every first new line of a form indents 2 spaces and subsequent new lines match the previous. An exception is if one argument is passed, then the next new lines match that argument. Certain "special forms" (function, lambda, if, etc) always indent 2 spaces.
So, examples:
(print!
1
2)
(; Subsequent lines match the first argument. ;)
(print!
1 2
3 4)
(; If one is supplied on the first line, the rest match up. ;)
(print! 1
2)
(; Lambdas have the signature on the first line, but the body doesn't indent to match it. ;)
(lambda () ()
(print! "meow"))
(; If expressions take a condition, but the then/else bodies don't match up to it. ;)
(if (empty? foo)
(print! "empty")
(print! "not empty"))
jank uses a tool called Cling to JIT compile C++ at runtime. Cling is a tool built on Clang/LLVM which uses Clang's C++ compilation abilities, but exposes them in a way that works JIT.
Very recently, Cling released a new version which supports LLVM 13. They were previously on LLVM 9. However, that newest version has issues when I enable optimizations, so I can't actually use it. Fixing this would require digging into the C++ internals of what's going on in Cling and why this is failing. I could also put you into contact with the developer of Cling, whom I know, and let him know you're helping out on jank. He should be able to guide you some.
I got this when trying to run this on my Macbook Pro M1.
[nix-shell:~/Projects/jank]$ jank-configure
The Meson build system
Version: 0.61.2
Source dir: /Users/pez/Projects/jank
Build dir: /Users/pez/Projects/jank/build
Build type: native build
Project name: jank
Project version: snapshot
C++ compiler for the host machine: clang++ (clang 11.1.0 "clang version 11.1.0")
C++ linker for the host machine: clang++ ld64 530
Host machine cpu family: aarch64
Host machine cpu: arm64
meson.build:8:0: ERROR: Include dir lib/magic_enum/include does not exist.
A full log can be found at /Users/pez/Projects/jank/build/meson-logs/meson-log.txt
Please let me know if I can provide any further detail. Or if I can make any experiments that would help you understand what's going on (or not).
Clojure has sorted-map
, which builds a PersistentTreeMap
, and sorted-set
, which builds a PersistentTreeSet
. immer doesn't support these, but there's some good info here: arximboldi/immer#105
jank will need its own version, but hopefully we can benefit from the stdlib.
Overall, since we have hash maps, array maps, and hash sets, I think the sorted variants are low priority.
jank has support for a new special form, called native/raw
. It works in place of Clojure's interop syntax and allows for inline C++. But it also support interpolating jank expressions into that C++. Docs on the rationale and final solution are here: https://github.com/jank-lang/jank/blob/main/DESIGN.md#interop
Similar to #24, clj-kondo doesn't know about native/raw
and so it causes linting failures. It also doesn't see that the vars referenced within the native/raw interpolations are referenced, so we often end up with unused param warnings which are incorrect. Addressing this will require changes to clj-kondo, which is all Clojure work, I think.
We need to figure out what goes into making editor-friendly REPL servers and cook one up. I think it makes the most sense to write it in jank itself, to dogfood sufficiently. I don't think existing Clojure REPL servers can work, but this should also be investigated.
As I intend for jank to be a monorepo of official tools, this project should be merged into main eventually.
This is blocked on #31 and work with native boxes (sockets, mainly).
bdwgc has support for type descriptors, to describe the layout of objects to minimize scanning time. The header is here: https://github.com/ivmai/bdwgc/blob/master/include/gc/gc_typed.h
We should take full advantage of this until we switch to mmtk.
I'm having trouble building. When I run jank-configure
I get an error that lib/magic_enum/include does not exist
. Which is true, but I don't know how to make it exist. lib/magic_enum
exists but is empty.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.