Giter VIP home page Giter VIP logo

fastsetup's Introduction

fastsetup

Setup all the things

First, do basic ubuntu configuration, such as updating packages, and turning on auto-updates:

sudo apt update && sudo apt -y install git
git clone https://github.com/fastai/fastsetup.git
cd fastsetup
sudo ./ubuntu-initial.sh
# wait a couple of minutes for reboot, then ssh back in
# If you're using WSL (Windows) use `sudo ./ubuntu-wsl.sh` instead of the above line

Then, optionally, set up dotfiles:

source dotfiles.sh

...and set up conda:

source setup-conda.sh
. ~/.bashrc
conda install -yq mamba

To set up email:

sudo ./opensmtpd-install.sh

To test email, create a text file msg containing a message to send, then send it with:

cat msg |  mail -r "x@$(hostname -d)" -s 'subject' EMAIL_ADDR

Replace EMAIL_ADDR with an address to send to. You can get a useful testing address from mail-tester.

NVIDIA drivers

To install NVIDIA drivers, if required:

ubuntu-drivers devices
sudo apt-fast install -y nvidia-XXX{-server}
sudo modprobe nvidia
nvidia-smi

WSL-only:

Install the latest NVIDIA driver on your Windows PC running WSL Then do a special install of cuda

sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-fast install -y cuda
nvidia-smi

(WSL-only): Don't worry if nvidia-smi reports "Internal Error" under the "Processes" heading.

If it's working, you should still see part of your GPU name, and how much memory is available in the first heading.

fastsetup's People

Contributors

bittone avatar galaxai avatar gmjn-dvlp avatar jph00 avatar manisnesan avatar polyrand avatar radekosmulski avatar stillgreyfox avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

fastsetup's Issues

Add uninstall.sh script

Like the issue #34 mentioned in December of last year,

The fastsetup ./setup-conda.sh does permanent changes to the current shell's config files or .bashrc; activating the environment as soon as the shell or the terminal starts. Many of us use linux as our daily driver and don't want this. I think it would be better to add an uninstall.sh script since there's none for those who want to uninstall it, and instead of activating the environment as soon as the shell starts by default, what could be done instead is create a function; and those who want to activate will just have to type that command in their terminal and it will activate the conda environment, and an exit command to exit out of the environment.

potential issue with `dotfiles.sh`

I am not sure if this is really an issue, only reporting this as it might be something worth looking at at some point, especially if others report similar behavior.

At the source dotfiles.sh step, an error occurs if there is no .bash_profile in linked to dotfiles. Not sure if this is intentional, whether it is a good practice to have the .bash_profile.

Also, despite there being a .vimrc, the following line either doesn't get run or doesn't do it's job:
vim -T dumb -n -i NONE -es -S <(echo -e "silent! PluginInstall\nqall")

The plugins are not installed and I need to execute PluginInstall manually.

Will try to take a closer look at this when I am building a new server.

Access Mamba via Miniconda installation

@jph00 Thanks for the recent update to setup-conda.sh. I just ran the script in Windows 11 and it istalled Miniconda perfectly!

However, I don't think this installer includes Mamba by default? May I ask what your workflow is, using the new setup-conda.sh script?

Do you just use Conda now, or do you install Mamba separately? If so I'm unsure what's the best way to do this. What commands would you recommend running after setup-conda.sh has finished?

I'm pretty new to this so would love to get the steps working correctly. I'm currently writing up my experiences from the Live coding 1 video so I can share with others.

Conda setup installs unsupported version of Python

I've just tried to start up another EC2 instance following the setup guide but I ran into an issue after running ./setup-conda.sh. That setup script installed Python 3.9.5, which is unsupported when trying to install mamba:

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - mamba -> python[version='3.6.*|3.8.*|>=2.7,<2.8.0a0|3.7.*']

Your python: python=3.9

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - mamba -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - python=3.9 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Note that strict channel priority may have removed packages required for satisfiability.

I also tried just continuing the setup guide using conda instead of mamba but ran into other version mismatch errors.

The initial ./ubuntu-initial.sh script installed Python 3.8.10 so I worked around the issue by changing the version of the Miniconda setup script to what looked like the latest Python 3.8 version:

case "$OSTYPE" in
  darwin*)  DOWNLOAD=https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-MacOSX-x86_64.sh; ;;
  linux*)   DOWNLOAD=https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh; ;;
  *)        echo "unknown: $OSTYPE" ;;
esac
wget -q $DOWNLOAD
bash Miniconda3-py38*.sh -b
~/miniconda3/bin/conda init $SHELL_NAME
rm Miniconda3-py38*.sh

However I don't know how much of a permanent solution that is so I haven't opened a PR to make that change. Mostly just posting this issue in case anyone else runs into the same issue.

Issue w/ Install miniconda step

Steps to reproduce:

  1. Create new EC2 instance Ubuntu 20.04, C5.large.
  2. Follow all steps up to https://course.fast.ai/start_aws#step-6-setup-server
  3. Try command
    conda install -yq mamba

This step fails w/ error below
image
(base) ubuntu@www:~/fastsetup$ conda install -yq mamba Collecting package metadata (current_repodata.json): ...working... done Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve. Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve. Solving environment: ...working... Found conflicts! Looking for incompatible packages.

`Package python-libarchive-c conflicts for:
conda[version='>=4.10.3'] -> conda-package-handling[version='>=1.3.0'] -> python-libarchive-c
conda-package-handling -> python-libarchive-cThe following specifications were found to be incompatible with your system:

  • feature:/linux-64::__glibc==2.31=0
  • feature:|@/linux-64::__glibc==2.31=0
  • cffi -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • conda-package-handling -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • cryptography -> libgcc-ng -> __glibc[version='>=2.17']
  • libffi -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • libgcc-ng -> __glibc[version='>=2.17']
  • libstdcxx-ng -> __glibc[version='>=2.17']
  • mamba -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  • ncurses -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • openssl -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • pycosat -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • python=3.9 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • readline -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • ruamel_yaml -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • sqlite -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • tk -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  • xz -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • yaml -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  • zlib -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31
`

I'm a software engineer (microsoft stack) but the topic of Deep Learning is brand new to me, therefore I do not know workaround to this. Im therefore stuck at page 14 of Deep Learnign for Coders with FastAI and PyTorch. Please let me know if there is a workaround for this error. I would love to continue my learning journey. Thank you in advance.

I continue to terminate the EC2 instance and recreate, only to come arrive at the same blocker. Please let me know if you can reproduce.

`Package 'python' has no installation candidate` error when setting up server in aws ec2

I get error message when using script on website https://course20.fast.ai/start_aws

I copy and pasted the following commands into my ec2 ubuntu g2dn.xlarge server

sudo apt update && sudo apt -y install git
git clone https://github.com/fastai/fastsetup.git
cd fastsetup
sudo ./ubuntu-initial.sh

This is the error I get:

Working... this may take a while.
E: Package 'python' has no installation candidate
Package manager quit with exit code.

setup-conda 'Mambaforge (Discouraged as of September 2023)'

Hello there,

I was following your Live Coding Session, and the setup-conda.sh is not working anymore because there is a new release on Miniforge: "Mambaforge (Discouraged as of September 2023)". More information can be found under these links:

Mambaforge (Discouraged as of September 2023)
With the release of Miniforge3-22.3.1-0, that incorporated the changes in conda-forge/miniforge#277, the packages and configuration of Mambaforge and Miniforge3 are now identical. The only difference between the two is the name of the installer and, subsequently, the default installation directory.

Hope this helps 😉

Issue with curl

I got the curl --no-progress-meter is unknown error when I tried to run the script.
curl --help didn't display anything for the --no-progress-bar option

I was able to install the file after removing the --no-progress-meter option from the below line in the script.
curl -LO --no-progress-meter $DOWNLOAD

I am not sure if it's anything to do with the curl versions

How to uninstall everything this "script" did?

I am working on Rocky Linux 9 and I realized too late that this was made for Ubuntu...

I was following this video: Lesson 2: Practical Deep Learning for Coders 2022, which "guided" me to install everything with fastsetup and continue to install fastchan fastai nbdev jupyter with mamba.

  1. How do I boot into desktop without doing conda config --set auto_activate_base False?

First of all, after doing everything and rebooting my PC, I could not launch my kernel’s desktop with the command startx anymore. I spent an hour looking around for a “solution” and it ended up being mamba’s base environment being activated by default. I deactivated it to be able to enter my desktop.

  1. Why is this failing?

Since it’s not recommended to install any packages into base env as the video tutorial suggests, but instead create new envs, I wanted to uninstall the fastchan fastai, fastchan nbdev and fastchan jupyter from the base env, but I am unable to uninstall fastchan jupyter because it keeps tossing errors.

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/conda/_vendor/boltons/setutils.py", line 247, in remove
        didx = self.item_index_map.pop(item)
    KeyError: PackageRecord(_hash=-2416236097303072177, name='__linux', version='5.14.0', build='0', build_number=0, channel=Channel("@"), subdir='linux-64', fn='__linux', md5='12345678901234567890123456789012', package_type='virtual_system')
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/conda/exceptions.py", line 1129, in __call__
        return func(*args, **kwargs)
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 935, in exception_converter
        raise e
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 928, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 886, in _wrapped_main
        result = do_call(parsed_args, p)
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 752, in do_call
        exit_code = remove(args, parser)
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 240, in remove
        conda_transaction = to_txn(
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 436, in to_txn
        final_precs = compute_final_precs(
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 371, in compute_final_precs
        final_precs.remove(i_rec)
      File "/home/sebastijan/mambaforge/lib/python3.10/site-packages/conda/_vendor/boltons/setutils.py", line 249, in remove
        raise KeyError(item)
    KeyError: PackageRecord(_hash=-2416236097303072177, name='__linux', version='5.14.0', build='0', build_number=0, channel=Channel("@"), subdir='linux-64', fn='__linux', md5='12345678901234567890123456789012', package_type='virtual_system')

`$ /home/sebastijan/mambaforge/bin/mamba uninstall -c fastchan jupyter`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_EXE=/home/sebastijan/mambaforge/bin/conda
         CONDA_PYTHON_EXE=/home/sebastijan/mambaforge/bin/python
               CONDA_ROOT=/home/sebastijan/mambaforge
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
                     PATH=/home/sebastijan/mambaforge/condabin:/home/sebastijan/.local/bin:/home
                          /sebastijan/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr
                          /local/go/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=1

     active environment : None
            shell level : 0
       user config file : /home/sebastijan/.condarc
 populated config files : /home/sebastijan/mambaforge/.condarc
                          /home/sebastijan/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.10.6.final.0
       virtual packages : __cuda=12.0=0
                          __linux=5.14.0=0
                          __glibc=2.34=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/sebastijan/mambaforge  (writable)
      conda av data dir : /home/sebastijan/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/sebastijan/mambaforge/pkgs
                          /home/sebastijan/.conda/pkgs
       envs directories : /home/sebastijan/mambaforge/envs
                          /home/sebastijan/.conda/envs
               platform : linux-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.10.6 Linux/5.14.0-162.6.1.el9_1.0.1.x86_64 rocky/9.1 glibc/2.34
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.
  1. Start anew from fresh install?

My guess is that I would do best to do a fresh install of mamba, but how to do that? I tried to uninstall it via conda uninstall mamba but conda then wants to download each package mamba had instead...

  1. Another issue

I also have missing .bashrc.local warnings whenever I open terminal now, so there's something messed up with that too. My .bashrc looks like this:

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
if [ -d ~/.bashrc.d ]; then
	for rc in ~/.bashrc.d/*; do
		if [ -f "$rc" ]; then
			. "$rc"
		fi
	done
fi

unset rc

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/sebastijan/mambaforge/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/sebastijan/mambaforge/etc/profile.d/conda.sh" ]; then
        . "/home/sebastijan/mambaforge/etc/profile.d/conda.sh"
    else
        export PATH="/home/sebastijan/mambaforge/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "/home/sebastijan/mambaforge/etc/profile.d/mamba.sh" ]; then
    . "/home/sebastijan/mambaforge/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<

source ~/.bashrc.local

Error installing ubuntu packages because of expired key

I'm running fastsetup after watching lesson 2 and running it into a fresh ubuntu wsl install.

Running sudo ./ubuntu-wsl.sh quickly fails as follows:

niaobu@DESKTOP-VONKRQ4:~/fastsetup$ sudo ./ubuntu-wsl.sh
Executing: /tmp/apt-key-gpghome.gqbKssNdhu/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys C99B11DEB97541F0 1EE2FF37CA8DA16B
gpg: key 1EE2FF37CA8DA16B: public key "Launchpad PPA for apt-fast" imported
gpg: key C99B11DEB97541F0: public key "Nate Smith <[email protected]>" imported
gpg: Total number processed: 2
gpg:               imported: 2
Get:1 https://cli.github.com/packages focal InRelease [3915 B]
Err:1 https://cli.github.com/packages focal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
Get:2 http://ppa.launchpad.net/apt-fast/stable/ubuntu focal InRelease [17.5 kB]
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Get:7 http://ppa.launchpad.net/apt-fast/stable/ubuntu focal/main amd64 Packages [532 B]
Get:8 http://ppa.launchpad.net/apt-fast/stable/ubuntu focal/main Translation-en [272 B]
Reading package lists... Done
W: GPG error: https://cli.github.com/packages focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 23F3D4EA75716059
E: The repository 'https://cli.github.com/packages focal InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Googling lead me to cli/cli#6175 and a solution replacing the keys in the following line:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C99B11DEB97541F0 1EE2FF37CA8DA16B

The new key is 23F3D4EA75716059

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.