swift-server / swiftly Goto Github PK

#9 introduces a barebones Actions configuration that only tests on Ubuntu 20.04. We should update this to cover all of the Linux platforms mentioned on https://download.swift.org.

rustup provides an env file which can be used to easily set environment variables used by rustup, including amending the PATH. We should consider providing a similar file for swiftly installations, since right now we just print a message telling the user they should do it themselves.

I got this error when trying to install swiftly on MacOS 13.5 beta (22G5038d). I'm using zsh as my default shell.

> curl -L https://swift-server.github.io/swiftly/swiftly-install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18846  100 18846    0     0   308k      0 --:--:-- --:--:-- --:--:--  347k
 --
Error: getopt must be installed from the util-linux package to run swiftly-install

It would be good to validate if the directory can be created prior downloading the toolchain

$ swiftly  install latest
Fetching the latest stable Swift release...
Installing Swift 5.9.0
                                                                                                   Downloading Swift 5.9.0
100% [=========================================================================================================================================================================================================]
Downloaded 548.6 MiB of 548.6 MiB
Extracting toolchain...
Error: ExtractError(message: Optional("Failed to create dir \'/usr/local/share/toolchains/5.9.0\'"))

I tried to migrate from swiftbox to swiftly.

1. Install it and config it.

curl -L https://swift-server.github.io/swiftly/swiftly-install.sh | bash
> 1
. $HOME/.local/share/swiftly/env.sh

2. Try to install swift toolchain and I'll always get a 403 Forbidden result.

➜  kyle swiftly        
OVERVIEW: A utility for installing and managing Swift toolchains.

USAGE: swiftly <subcommand>

OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

SUBCOMMANDS:
  install                 Install a new toolchain.
  use                     Set the active toolchain.
  uninstall               Remove an installed toolchain.
  list                    List installed toolchains.

  See 'swiftly help <subcommand>' for detailed help.
➜  kyle swiftly list
Installed release toolchains
----------------------------

Installed snapshot toolchains
-----------------------------
➜  kyle swiftly install
Error: Missing expected argument '<version>'
Help:  <version>  The version of the toolchain to install.
Usage: swiftly install <version> [--token <token>]
  See 'swiftly install --help' for more information.
➜  kyle swiftly install latest
Fetching the latest stable Swift release...
Error: received status "403 Forbidden" when reaching https://api.github.com/repos/apple/swift/releases?per_page=100&page=1: {"message":"API rate limit exceeded for xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

➜  kyle swiftly install 5.9   
Fetching the latest stable Swift 5.9 release...
Error: received status "403 Forbidden" when reaching https://api.github.com/repos/apple/swift/releases?per_page=100&page=1: {"message":"API rate limit exceeded for xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

swiftly-install.sh currently can only perform fresh installations. We should update it to support just updating the swiftly binary while leaving installed toolchains alone.

We may want to preserve the clean installation functionality via an argument.

The list-available command accepts a toolchain selector and prints the various toolchains that are available to install (including those already installed) matching that selector. If no selector is provided, all of the toolchains available to install are selected. Alternatively, we can list the newest toolchain from each major.minor combination, as well as the latest snapshot available from the main branch and any major.minor snapshot releases.

For the MVP, this will be implemented using the GitHub tags API on the apple/swift repository.

This is currently mostly implemented, but it doesn't work for snapshots.

The design and current implementation assume that swiftly, its configuration, and its toolchains will be stored in ~/.swiftly. This ignores certain platform-specific conventions, such as the XDG base directory specification, which indicates it should be stored in ~/.local/share/swiftly. We should adopt these conventions where possible.

The install command should accept an argument (e.g. --use) which will automatically use the new toolchain if installation succeeds. This already happens by default if there are no toolchains installed yet, but this would be very useful in other cases too.

There should be some instructions about how to install Swiftly itself.
A command like this will work as a quick-start:

curl -L https://raw.githubusercontent.com/swift-server/swiftly/main/install/swiftly-install.sh | bash

Some versions of Swift are not available on certain platforms. For example, Ubuntu 22.04 only has downloads available for 5.7+. Right now, swiftly prints something like "Swift 5.6.3 doesn't exist", which isn't the most helpful message. Instead, it should be more clear about the fact that the given version does exist, but is just not available for the user's platform.

Right now, the installation script doesn't do a good job of detecting when a download fails. For example, the script will treat a 404 error response as if an executable was downloaded and only fail when actually trying to execute an HTML document. The script should be updated to adequately handle these cases and print a useful error message.

When running swiftly use in a non-interactive environment, in my case a Heroku buildpack, the following section in the Linux implementation of use gets immediately cancelled: Linux.swift#L69-L86, and the symlinks are never overwritten.

My use case would greatly benefit from a --disable-confirmation flag, like the swiftly install script does, which would suppress the file list (it's noise) and would silently overwrite the symlinks.

To enable users to use swiftly on platforms that aren't explicitly supported by Swift, we could provide the option during swiftly installation to use the toolchains intended for another supported platform.

e.g. If installing on debian, a user could select Ubuntu 20.04 as the platform they'd like their toolchains to download from, and it's likely they would work ok (I haven't tried this though).

See https://www.swift.org/download/#linux

swiftly currently produces progress output when downloading to give the user an idea of how long an install will take. This is ideal for humans using it via the CLI, but it can be pretty noisy when included in the logs of a CI test run. We should provide a --quiet argument that suppresses this output for such uses cases. Error and possibly success output would still be produced.

We probably would want to enable this for swiftly's tests too.

There are too many places in the code base at the moment where errors are not caught and dealt with. The use function would throw an error and then that was it. We should catch this and create the folder.

Running a global system install like:

sudo SWIFTLY_HOME_DIR=/usr/local/share SWIFTLY_BIN_DIR=/usr/local/bin ./swiftly-install.sh  --disable-confirmation

Will result the profile being added for the root user, but can include this globally for all users by creating a file in /etc/profile.d

See https://www.swift.org/download/#linux

Swift recently added downloads for Red Hat Universal Base Image 9, so swiftly should have support for it as well.

We should (optionally) install tab-completion scripts for swiftly. This will make it easier for users to remember certain commands and to select among installed toolchains.

The design document says:

The ~/.local/bin directory would include symlinks pointing to the bin directory of the "active" toolchain, if any.
This is all very similar to how rustup does things, but I figure there's no need to reinvent the wheel here.

https://github.com/swift-server/swiftly/blob/4d718c0621bdfd5835507565896ef9d28b390659/DESIGN.md#installation-of-a-swift-toolchain

But it looks like rustup does more things, known as Proxies, to select the active toolchain based on rust-toolchain.toml or cargo +nightly style at the actual command invocation time.

Of course, I think it makes a lot of sense to start by limiting the toolchain selection only globally by creating symlinks at installation time (as swiftly does now).
But I'd like to raise up the future direction discussion of shims just for the record. (Sorry if it's already discussed somewhere)

The test suite currently uses some global variables for things like the mocked home directory and capturing output. This has the unfortunate side effect that the tests must be run sequentially. We should modify such variable usage to be associated with an instance of swiftly such that the tests can be run in parallel. Much of the time in the tests is spent on downloading toolchains, so this may result in a significant speedup.

swiftly currently can parse toolchain selectors of the form a.b-DEVELOPMENT-SNAPSHOT-YYYY-mm-dd-a, but tags at https://github.com/apple/swift/tags have a swift- prefix, which swiftly cannot parse. We should update the parsing to accept that so that users can easily install a toolchain based on a git tag.

For the MVP, we'll need a bootstrapping script that can detect the platform it's being run from (for starters, just Ubuntu flavors) and then download the appropriate swiftly executable. This is inspired by rustup-init.sh.

Once all the core commands are implemented, we'll need to set up automated testing to ensure everything is working as expected. We can use GitHub actions for this.

In general root users do not have ~/.local/bin in their PATH. If the user is installing as root perhaps we should use a different PATH. I think we should also warn the user when installing as root.

swiftly should have a simple landing page similar to http://rustup.rs that clearly displays the bash one-liner that can install swiftly. It should have a button that copies the line to the user's clipboard. GitHub pages can be used for hosting.

Most commands accept a toolchain selector argument to identify the toolchain(s) the command is intended to operate over. Selectors for stable releases can currently be of the form a, a.b, or a.b.c. The install command requires that the user specify at least a minor version (e.g. a.b or a.b.c). We should update it to also accept just the major version (e.g. swiftly install 5 would install the latest Swift 5 release).

It seems weird that executing swiftly with no input, has no output:

user@vps:~# swiftly 

user@vps:~# swiftly install
Error: Missing expected argument '<version>'
Help:  <version>  The version of the toolchain to install.
Usage: swiftly install <version> [--token <token>]
  See 'swiftly install --help' for more information.

user@vps:~# swiftly install latest
Fetching the latest stable Swift release...
Installing Swift 5.8.0

. . .

As you can see, user@vps:~# swiftly has no output although other commands work fine.
Executing swiftly with no input should result in something like what it shows when you use "swiftly help".

This is a tracking issue for macOS support. More details will be added as that work is scoped out.

Once Linux has reached MVP status, this will be addressed.

The current implementation reloads config.json from disk multiple times. We should instead cache it for the duration of a swiftly command invocation.

The use command accepts a toolchain selector and marks the newest matching toolchain as "in use". On Linux, what this means is that the symlink in ~/.swiftly/bin will be updated to point to the bin directory of the toolchain in question. As part of this, we'll need to update the install command to invoke the use command post-install if no toolchain is already in use.

In case application has been built by swift container, global library path needs to be deployed in order to find the Swift runtime in order to run the application not inside a container.

For example the RPATH will be set to /usr/lib/swift/linux, and if the toolchain is going to be deployed by Swiftly under /usr/local, it is not possible to execute binaries which are built by the container:

$ chrpath -l MySwiftBinary 
MySwiftBinary: RUNPATH=/usr/lib/swift/linux:$ORIGIN

One either needs to have a global entry like this (to avoid setting LD_LIBRARY_PATH for each application):

$ cat /etc/ld.so.conf.d/swift.conf 
/usr/local/share/toolchains/5.9.0/usr/lib/swift/linux

It would be nice that swiftly use would handle this use case.

shellcheck is a static analysis tool for bash which can catch a number of common bash issues. We should run it on swiftly-install.sh and include it in the soundness check CI task.

If no other versions of swift are downloaded then swiftly install should also call use on the downloaded version of swift

swiftly should have an exec command which allows a user to run a specific swift command without having to switch the active toolchain.

e.g.

swiftly exec 5.5 build

Would run swift build using the latest 5.5 toolchain.

We currently just download Swift and extract/install it manually for our CI runs. We should use prior releases of swiftly to do this instead.

To properly enable community contributions, swiftly needs to have a proper license and guidelines for submitting contributions. Additionally, these are minimal requirements for participating in the SSWG incubation process: https://github.com/swift-server/sswg/blob/master/process/incubation.md#minimal-requirements

We can borrow much of this from vscode-swift

This command accepts a toolchain selector and prints the installed toolchains that match the selector, if any.

The update command accepts a selector and updates the newest matching toolchain to the newest available matching toolchain, if any. This happens in a few steps:

• List installed toolchains to see if any match the selector
  • If no selector is provided, match the currently in-use toolchain
• If one is found, list available toolchains, and see if any matches the selector and is newer than the found toolchain
• Uninstall the found toolchain
• Install the newer toolchain
  • If the updated toolchain was in use, mark the new one as in use

Updating depends on the functionality from a bunch of other commands, so they will need to be done first (#2, #4).

On a lark this weekend I was curious to see if it'd be possible to get swiftly ported to Windows. When I pulled the project down I noticed that a bunch of the NIO stuff doesn't seem to work out of the box on Windows (this is likely fixable, but it prompted me to look at what it was being used for).

It seems like it's primarily being used to build an HTTP client and I was wondering if it's feasible/desirable to switch to using URLSession from Foundation to make this a bit more portable. I don't know if there were specific limitations with that framework which prevented it from being chosen, but I figured I'd start the discussion since managing different toolchains on Windows is also quite a pain!

swiftly-install.sh is not currently compatible with the fish shell. We should update it to support it.

To ensure the toolchains installed by Swiftly are valid and authentic, we should validate their signatures. The processes for doing this on macOS and Linux are outlined here: https://www.swift.org/download/#using-downloads

When performing a clean installation via swiftly-insall.sh, any existing symlinks to swiftly-managed toolchains are not cleaned up, despite the pointed-to toolchain being deleted. This leaves a bunch of dangling symlinks around. We should ensure that these symlinks are cleaned up when a clean install is performed.

The uninstall command accepts a toolchain selector and uninstalls all matching toolchain(s) after prompting for confirmation. The skeleton code for this command is already implemented, but the following need to be completed in the Linux platform implementation:

• Platform.listToolchains for getting the list of currently installed toolchains
  • This can probably be default-implemented on the Platform protocol, since all the information is coming from config.json anyways
• Platform.uninstall: basically just deletes the toolchains/<toolchain name> directory and removes it from config.json

swiftly currently prints some basic amount of human readable output, but we should also introduce the ability to log output using swift-log. We should also support a --verbose option to control how much output is included.

There are no explanations on how to help developing Swiftly.

I personally had to:

1. Use SSWG's VSCode extension and set up a dev container to develop Swiftly on macOS, because otherwise there are some macOS-related errors, and also Swiftly doesn't yet support macOS.
2. Create a file at /root/.local/share/swiftly/config.json containing the json below, because otherwise swiftly would complain about the file and wouldn't work:

{
  "platform" : {
    "nameFull" : "ubuntu22.04",
    "name" : "ubuntu2204",
    "namePretty" : "Ubuntu 22.04.1 LTS"
  },
  "inUse" : null,
  "installedToolchains" : []
}

3. Use swift run swiftly [subcommand].

I just Swift 5.9.0 using Swiftly on a fresh Fedora 38 and it seems like swift repl isn't working because of a missing Python version.

python3 --version tells me I'm on 3.11.5 and swift repl returns the following error:

/home/marius/.local/share/swiftly/toolchains/5.9.0/usr/bin/lldb: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

I suspect installing Python 3.9 would fix the issue, but is there a way to get this out of the box with Swiftly?

To retrieve the currently in-use toolchain, users currently need to either visually parse the output of swiftly list or pipe it to grep use. To make this a bit easier, swiftly should print the in use toolchain if no arguments are provided to use.

e.g.

$ swiftly use
main-snapshot-2023-09-22 (in use)