dfinity / vessel Goto Github PK

This is a followup on the issues discussed here dfinity/motoko-base#397 and here dfinity/motoko-base#392.

Using master as the latest branch for motoko base lib(current commit 25831e6e03c0231a2839029f048e10f61a2dcb03) throws an error saying: Text.mo:83 type error [M0072], field textCompare does not exist in type. Reverting the changes in sources that are mentioned in this commit dfinity/motoko-base@c507105 throws another error saying .vessel/base/master/src/ExperimentalStableMemory.mo:64.5-64.9: type error [M0072], field stableVarQuery does not exist in type.

Tested on:
vessel v0.6.4 and v0.6.3
dfx version 0.10.1
system: ubuntu 20.04

I build a module with test file, so I need make test often to make sure the code is right.

But It installs the same packages everytime, waste a litte time:

make test
make -C test
[INFO] Installing 2 packages
[INFO] Installation complete.
.vessel/.bin/0.6.5/moc --package base .vessel/base/927119e172964f4038ebc7018f9cc1b688544bfa/src --package matchers .vessel/matchers/v1.2.0/src -wasi-system-api -o Test.wasm Test.mo && wasmtime Test.wasm
All tests passed.
rm -f Test.wasm

The README currently states:

1. Download a copy of the vessel binary from the release page or build one yourself
2. Run vessel init in your project root.

There are a lot of steps missing in between 1 and 2. What do I do with the binary once I've downloaded it? Etc.

On macOS I had to chmod +x it, try to run it via ./vessel-macos, open System Preferences > Security & Privacy and allow it to run, then I had to move it somewhere to it's in my path (I chose /usr/local/bin based on this answer on StackExchange).

Issue #35 in here references a vessel-install.sh script, but I don't see that anywhere in this repo.

So I tried upgrading from ubuntu 20 to ubuntu 22 but it seems that on 22 they're using the new improved openssl3.

If I try to use vessel I get this: vessel: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

Can vessel start using openssl3 or is that a dependency out of your hands?

Related:

https://stackoverflow.com/questions/72133316/ubuntu-22-04-libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-di

Trying to fetch base from branch joachim/no-word

{ name = "base"
        , repo = "https://github.com/dfinity/motoko-base"
        , version = "joachim/no-word"
        , dependencies = [] : List Text
        }

vessel install fails, maybe due to / in the branch name?

$ vessel install
[INFO] Installing 3 packages
[INFO] Downloading tar-ball: "base"
[WARN] Downloading tar-ball failed, cloning as git repo instead: "base"
Error: No such file or directory (os error 2)

I’d like to see a default.nix here that allows me to run

nix-env -iA vessel -f .

just like I do with moc.

~/dfinity/motoko-merkle-tree/gen-tests $ vessel bin
Error: Failed to parse the package set file

Caused by:
    No such file or directory (os error 2)
~/dfinity/motoko-merkle-tree/gen-tests $ ls
dist-newstyle  gen-tests.cabal  main.hs
~/dfinity/motoko-merkle-tree/gen-tests $ ls ../package-set.dhall 
../package-set.dhall

I would have expected vessel to work in a subdirectory as well.

If that is intentional, the error message could be better, e.g. say

Did not find package-set.dhall. vessel must be run in the same directory as the package-set.dhall.

A new vessel docs command could generate documentation for the local project and all its dependencies, which would allow you to browse it in your local browser.

• This makes it easier to preview your libraries documentation
• It also makes it easier to access your dependencies documentation at the exact version you're currently using
• Offline friendly, as soon as you downloaded your dependencies once no internet connection is needed to access their docs

Motivation

Currently, vessel allows dependency tracking through a commit hash, a tag, or a branch name.

The code for both tags and branch names of a dependency can be updated remotely by the package developer. When this happens, a vessel install doesn't automatically fetch the newest version of the branch/tag code from GitHub.

See this topic issue and discussion from the developer forums

Potential Solution(s)

1. Add a -f flag, which would force re-fetch all of dependencies of a project listed in the package-set.dhall, and overwrite any dependencies that already exist. This would be very much similar to npm install --force, as shown in the docs for npm install.

2. An alternative solution to this would be to include a release property that could be used in place of the version in the package-set.dhall if one wanted to specifically track a branch/tag in which the underlying code is update-able, which would signify that this package should be re-fetched on every vessel install.

@ggreif @crusso

We discussed wanting a new vessel feature to support multiple packages in one repo.

In particular, we'd like to continue to use vessel but have it support

1. moving the base library (back) into the main motoko repository.
2. splitting the base library into platform-independent parts (data structures) and IC-specific parts.

To support 1 and 2, we'd like to refine how vessel works so that a package is located not only by a repo, but instead by a pair of fields, repo and path, and the path specifies where within that repo one finds the vessel config files and the src directory for the package.

If the path field is absent, it is assumed to be the empty path, meaning that the repo has only one package, at its root (the current design).

Then, we'd have the motoko repo organized like this (simplified), assuming that the compiler and base and ic directories represent the code now in the motoko repo and base repos, but reorganized:

compiler
  src
  ...
lib
  base
    src
  ic
    src

cc @rossberg @crusso

I have tried to follow the directions provided.

After downloading macos binary from releases, I am unable to open it. I get an error:

How do I install vessel after downloading it?

Thank you.

Hi,

My name is Tiago and am on the Motoko Bootcamp. I really like everything I have seen about Dfinity and ICP.

But this package manager is really raw :( And it makes it super hard to reuse code from other community members.

The end result is the bad practice that have seen a lot, that is to literally copy paste code from github to internal projects. But that is not maintainable nor scalable. The more time passes, the bigger is the tech debt of all of us.
Please kindly consider making vessel a priority again 🙏

I tried to install vessel and Mac (rightly so) does not allow for a source file to be imported from the internet and be executed without proper verification:

Hope this can be eventually sorted out.

Thank you,
Tiago L.

At the moment all vessel commands need to be run in the project root. I'd like for vessel to search for vessel.dhall in parent directories so you can run it from within a test/ or src/ directory.

cargo and npm seem to do the same thing.

We should still download packages and binaries into a .vessel directory next to the vessel.dhall file, but output paths relative to where vessel was invoked.

#24 shows that this is an actual need :)

Motivating example

Consider this scenario:

I have two packages:

• BigTest is a testing library developed to script long-running tests with a DSL
• BigMap is a data structure library developed for simple databases that get really large

Each package has a test suite, in its test subdirectory; when master advances for this package repo, we want to run CI using this test suite.

BigTest solving a problem that the test suite of BigMap needs to solve. So, the test suite of BigMap depends on BigTest.

BigMap is a service that BigTest tests, as a regression test for its own revision management. So, the test suite of BigTest depends on BigMap.

Unless we distinguish library dependencies from test dependencies (e.g., cargo for Rust does this), we have a problem: There is a cyclic dependency between these two packages, apparently.

Today

Today, vessel packages conflate the dependency information of package's library code and that of its test suite code, in the sense that there is only one place to specify this information, even though the dependency structure we want is really more fine-grained.

This issue proposes that this information be distinguished.

Proposal

I propose we distinguish library and test packages (two kinds of packages):

• Libraries depend on other libraries.
• Test suites depend on libraries.
• By default: Libraries do not depend on test suites (because test suites are either not possible to name as packages themselves, or they are distinct with distinct names from the corresponding library package).

Initially, we need not name or import test suite packages.

But we do need to identify their dependencies, and permit them to include more than the libraries that they test.

Manifests itself as Error: Invalid cross-device link (os error 18)

Right now we create temporary directories in $TMPDIR, but if that is on a different mount point (read partition) we get failures when trying to move those.

The solution I'll attempt is to create the temporary directories in .vessel/.tmp instead, which should always be on the correct mount point.

This is not an issue (or bug), so much as a short report of my recent experience using Vessel, as a Motoko programmer trying to separate a large repo into several smaller ones.

Overall: This tool is great. It's doing exactly what I want and need.

Pros:

• Vessel fills a niche that I needed: I want to have small Motoko repos that have interpendencies
• The "work" goes into a simple, re-usable form (the JSON entries in the .vessel.json and .package-set.json files)
• In about an hour, I was able to complete my refactoring task; hurray!
• vessel verify is awesome. So calming. : )

Cons:

• Moderate confusion: vessel install does not actually overwrite (re-install); It didn't propagate a fix that I thought it would. I "worked around" this (intended?) behavior by deleting the .vessel subdir and repeating the install step again.
• Minor confusion: If one forgets to add a dependency that is present in the .package-set.json file, but absent from the dependencies list in .vessel.json, the compiler will give an unhelpful error that says that the package is unknown; vessel could help by suggesting the fix of copying the missing package name into the .vessel.json file. But Vessel is simple, and doing that would be somewhat sophisticated (as would processing any errors from the compiler).

Vessel currently does not support imports from private libraries. This is due to a limitation of the headers passed by the rust_dhall crate Nadrieril/dhall-rust#134.

Here are some feature requests that would greatly improve the functionality of the Vessel package manager (from my perspective):

• The sources command should not attempt to install packages. This can sometimes cause issues with Vessel when using it offline. It would be better if sources only fetched the package source code and did not try to install anything.

• Private repository support has been requested previously in issue #62. It would be great to have the ability to use private repositories in Vessel.

• Monorepo support would be very helpful. It would be nice if the repository could declare custom subpackages, or if users could do it themselves. This would make it easier to manage large codebases with multiple packages. (#38)

• Autodiscovery of dependencies would be a great addition to Vessel. If a dependency already has a vessel.dhall file, Vessel could automatically detect it and add it to the project's dependencies.

• The src dependency should be removed. This is a restriction that is not necessary and prevents Vessel from being used with other languages. Removing this restriction would make Vessel more flexible.

• A simple install command would be very useful. Vessel could autogenerate everything based on the downloaded repository, with or without a vessel.dhall file. This would make it much easier to install packages and get started with Vessel.

• Using json/yaml instead of dhall would improve usability for most users. While dhall is a powerful tool, it can be difficult to use for beginners. Providing options to use json/yaml as alternative configuration formats would make Vessel more accessible to a wider audience. (#59)

I made an experimental package manager to test some of these features out, which can be found here.

when I try to dfx build --check my project
It happens like that follows

Building canisters to check they build ok. Canister IDs might be hard coded.
Building canisters...
The build step failed for canister 'cs7ap-qc5fi-rjs6g-k5hmq' with an embedded error: The command '"vessel" "sources"' failed with exit status 'exit status: 101'.
Stdout:

Stderr:
thread 'main' panicked at 'called Result::unwrap() on an Err value: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("github.com")), port: None, path: "/dfinity/vessel-package-set/releases/download/mo-0.6.18-20220107/package-set.dhall", query: None, fragment: None }, source: TimedOut }', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/dhall-0.10.1/src/semantics/resolve/resolve.rs:295:36
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

When doing nix-build I ran into the following error (on Linux, but I think platform shouldn't matter):

installing
hash mismatch in fixed-output derivation '/nix/store/l6h129fiylvpky3izl1whhqwgw4qd710-vessel-1.0.0-vendor.tar.gz':
  wanted: sha256:0kykwbqhh4zm4si0ibr89m3vas7ldpg90r8kjfv0qn5mr22zw48c
  got:    sha256:07rz0gm6yqxmi34dzgia31h1kvk70xa3rb7h7ygl6pylx71grwbj
cannot build derivation '/nix/store/7rn7qpl5fwbyjgi3lndqs3ji79xjhq5f-vessel-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/1f9brm017w2k82hnsdg5ss37rzs4aqjm-all-systems-go.drv', '/nix/store/7rn7qpl5fwbyjgi3lndqs3ji79xjhq5f-vessel-1.0.0.drv', '/nix/store/kwb601s7bdvysfrlfanf1p83inwhqqf2-vessel-1.0.0.drv' failed

So I had to fix the sha256 in default.nix to proceed.

Also, when compiling rust packages I see a lot of packages being compiled twice. Is this expected?

Starting with dfinity/motoko#2570 Motoko compiler releases will be published as GH release artifacts.

We'll have to (at least for a while) conditionally download from the old or the new location based on the version number.

It's common to have library and test targets with different dependencies and I can think of more possible targets like benchmarks, or maybe even canisters/canister blueprints.

In my library template I "solve" this by referencing the main vessel.dhall from the test folder and extending it using Dhall. It's not ideal because it creates a separate .vessel directory underneath test/. And requires running test commands from within test/, but I also don't hate having the test specific configuration in the test/ directory rather than adding it to the library config.

Possible Solution 1:
spago solves this by letting the user specify a config file to use instead of the default one. So you'd create a test.dhall file and call something like vessel -x test/test.dhall sources.

Possible Solution 2:
cargo and most other package managers let the user specify their multiple targets in the main manifest file, We could come up with a schema to do the same for vessel.

Related to/Generalizes #7

Haven't gotten further than running this command as recommended in the IC punks repo'

gennaroschiano@Gennaros-MacBook-Pro ICPunks % ./scripts/vessel-install.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   620  100   620    0     0   2006      0 --:--:-- --:--:-- --:--:--  2000
/Users/gennaroschiano/bin/vessel: line 1: syntax error near unexpected token `<'
/Users/gennaroschiano/bin/vessel: line 1: `<html><body>You are being <a href="https://github-releases.githubusercontent.com/248089583/a56f5800-c9ea-11eb-84e7-742446381094?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210909%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20210909T063307Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=9f030b009d52b8f2bb88e0e691868b251bbbdc8da9f9938989d3fed751f4d607&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;key_id=0&amp;repo_id=248089583&amp;response-content-disposition=attachment%3B%20filename%3Dvessel-macos&amp;response-content-type=application%2Foctet-stream">redirected</a>.</body></html>'

Here is the script that is included to install vessel

#!/bin/sh

mkdir -p $BINDIR
BIN=$BINDIR/vessel
OS_FILENAME=linux64
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
  OS_FILENAME=linux64
elif [[ "$OSTYPE" == "darwin"* ]]; then
  OS_FILENAME=macos
elif [[ "$OSTYPE" == "win32" ]]; then
  OS_FILENAME=windows64.exe
fi

VESSEL_VERSION=v0.6.0
curl -o $BIN https://github.com/dfinity/vessel/releases/download/$VESSEL_VERSION/vessel-$OS_FILENAME && chmod +x $BIN && $BIN help```

this, TOO, doesn't work as it should:

vessel init
vessel: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

what if I have openssl-3.1.4-1 installed?

Current usage within dfx as a packtool:

vessel sources

New usage:

vessel sources [--package name path]*

Where the additional --package name/path pairs are added to those provided by vessel sources (logic to resolve duplicates, etc, is optional).

See also: https://github.com/dfinity/motoko-bigmap/pull/16

$(vessel bin)/mo-ide --canister-main example_backend

hangs indefinitely.

The command

vessel bin

does not exit already for a few minutes.

vessel.dhall:

{
  dependencies = [ "base", "btree", "stable-rbtree", "stable-buffer" ],
  compiler = Some "0.9.1"
}

$ vessel --version
vessel 0.6.5

No further repro info:

https://discordapp.com/channels/748416164832608337/748416164832608341/930766049903775754