Giter VIP home page Giter VIP logo

rstudio-server-conda's Introduction

Running Rstudio Server in a Conda Environment

I usually rely on the conda package manager to manage my environments during development. Thanks to conda-forge and bioconda most R packages are now also available through conda. For production, I convert them to containers as these are easier to share.

Unfortunately, there seems to be no straightforward way to use conda envs in Rstudio server. This repository provides three approaches to make rstudio server work with conda envs.

Running Rstudio Server with Singularity

With this approach Rstudio Server runs in a Singularity container (based on rocker/rstudio).
The conda environment gets mounted into the container - like that there's no need to rebuild the container to add a package and install.packages can be used without issues. The container-based approach has the following benefits:

  • Authentication works (#3)
  • Several separate instances of Rstudio server can run in parallel, even without the Pro version.

Prerequisites

Usage

  1. Clone this repository

    git clone [email protected]:grst/rstudio-server-conda.git
    cd rstudio-server-conda/singularity
  2. Activate the target conda env or set the environment variable CONDA_PREFIX to point to the location of the conda env.

  3. Check the run_singularity.sh script. In particular, you may need to add additional bind mounts (e.g. a global data directory).

  4. Execute the run_singularity.sh script. It will automatically build the container if it is not available.

    PORT=8787 PASSWORD=notsafe ./run_singularity.sh
  5. Log into Rstudio

    • open rstudio server at http://localhost:8787 (or whatever port you specified)
    • login with your default username and the password you specified via the PASSWORD environment variable.

Running Rstudio Server with Podman/Docker

This approach is similar to Singularity, but uses Docker or Podman and a docker-compose.yml file instead.

Known limitations

  • No access to shared group directories (#14)

Prerequisites

Usage

  1. Clone this repository

    git clone [email protected]:grst/rstudio-server-conda.git
  2. Build the rstudio container (fetches the latest version of rocker/rstudio and adds some custom scripts)

    cd rstudio-server-conda/docker
    docker-compose build     # or podman-compose
  3. Copy the docker-compose.yml file into your project directory and adjust the paths.

    You may want to add additional volumes with your data.

    [...]
       ports:
          # port on the host : port in the container (the latter is always 8787)
          - "8889:8787"
        volumes:
          # mount conda env into exactely the same path as on the host system - some paths are hardcoded in the env.
          - /home/sturm/anaconda3/envs/R400:/home/sturm/anaconda3/envs/R400
          # Share settings between rstudio instances
          - /home/sturm/.local/share/rstudio/monitored/user-settings:/root/.local/share/rstudio/monitored/user-settings
          # mount the working directory containing your R project.
          - /home/sturm/projects:/projects
        environment:
          # password used for authentication
          - PASSWORD=notsafe
          # repeat the path of the conda environment (must be identical to the path in "volumes")
          - CONDAENV=/home/sturm/anaconda3/envs/R400
  4. Run your project-specific instance of Rstudio-server

    docker-compose up 
  5. Log into Rstudio

  • Open your server at http://localhost:8889 (or whatever port you specified)
  • Login with the user rstudio (when using Docker) or root (when using Podman) and the password you specified in the docker-compose.yml. If you are using Podman and login with rstudio you won't have permissions to access the mounted volumes.

Running Locally

With this approach a locally installed Rstudio server is ran such that it uses the conda env.

Known limitations

  • no authentication (#3). Use this approach only in a secure network!

Prerequisites

Usage

  1. Clone this repo

    git clone https://github.com/grst/rstudio-server-conda.git
    
  2. Run rstudio server in the conda env

    cd rstudio-server-conda/local
    conda activate my_project
    ./start_rstudio_server.sh 8787  # use any free port number here. 
    
  3. Connect to Rstudio

    You should now be able to connect to rstudio server on the port you specify. If an R Session has previously been running, you'll need to rstart the Rsession now.

    Obviously, if your env does not have a version of R installed, this will either not work at all, or fall back to the system-wide R installation.

How it works

  • Rstudio server, can be started in non-daemonized mode by each user individually on a custom port (similar to a jupyter notebook). This instance can then run in a conda environment:

    > conda activate my_project
    > /usr/lib/rstudio-server/bin/rserver \
       --server-daemonize=0 \
       --www-port 8787 \
       --rsession-which-r=$(which R) \
       --rsession-ld-library-path=$CONDA_PREFIX/lib
    
  • To avoid additional problems with library paths, also rsession needs to run within the conda environment. This is achieved by wrapping rsession into the rsession.sh script. The path to the wrapped rsession executable can be passed to rserver as command line argument.

    rserver # ...
        --rsession-path=rsession.sh
    
  • When using multiple users a unique secret-cookie-key has to be generated for each user. The path to the secret cookie key can be passed to rserver as a command line parameter.

    uuid > /tmp/rstudio-server/${USER}_secure-cookie-key
    rserver # ...
      --secure-cookie-key-file /tmp/rstudio-server/${USER}_secure-cookie-key
    

rstudio-server-conda's People

Contributors

grst avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rstudio-server-conda's Issues

WSL2 local install permission issue - different from sqlite

Hi,

I am trying to setup RStudio with conda locally in my WSL2 and seeing the following issue.

./start_rstudio_server.sh 8787
## Current env is >>
/home/pr/.miniconda3/envs/R
TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/rstudio/logging.conf'. Logging to '/home/pr/.local/share/rstudio/log/rserver.log'.
2022-03-08T16:27:51.234866Z [rserver] ERROR system error 13 (Permission denied) [path: /var/run/rstudio-server, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:842; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:626

The conda environment is showing up here correctly but the /var/run/rstudio-server doesn't exist.

I have modified the permission for the sqlite database as suggested in other issues.

ll /var/lib/rstudio-server
total 56
drwxr-xr-x  5 rstudio-server rstudio-server  4096 Mar  4 16:44 ./
drwxr-xr-x 36 root           root            4096 Mar  2 16:40 ../
drwxr-xr-x  2 root           root            4096 Mar  2 16:40 body/
drwxr-xr-x  2 root           root            4096 Mar  2 16:40 conf/
drwxr-xr-x  2 root           root            4096 Mar  2 16:40 proxy/
-rwxrwxrwx  1 rstudio-server rstudio-server 32768 Mar  4 16:44 rstudio-os.sqlite*
-rw-------  1 root           root              32 Mar  2 16:40 secure-cookie-key

I was able to get it to work with the local/start_rstudio_server.sh once over my home network but the R library and the binary was not picking up from conda even when the environment was active. However, the server now doesn't start, so I can't reproduce it.

Let me know if you need any additional information.

Thanks!

R is taking longer to start than usual

Hi,

I followed your nice script to bypass limitations of free rstudio server.
Actually I cannot manage to run it properly,

./start_rstudio_server.sh 8008

Current env is >>

/home/pkunderfranco/.conda/envs/Scenic_1.1.2.2

once I open in the web brower port 8008, R is not responding and keep loading,

Any suggestion?

Best
Paolo

running rserver on computational node

Hi,
I'm very new to docker but I already find your repo greatly helpful.
I'm using a HPC system that I do not have root privilege, so I installed Singularity using conda.
Now I can run rserver in the login node. But what I really want to do is to be able to start a SLURM job and run RStudio server there.
When I tried to do this, I got the error:

FATAL:   while extracting rstudio_4.0.5.sif: root filesystem extraction failed: extract command failed: ERROR  : Failed to create user namespace: user namespace disabled
: exit status 1

I have no idea what this means. I tried the solution from rocker project page, and it's the same.
Therefore I'm wondering how should I run RStudio server in a computational node?

Thanks so much!

error when trying to enter singularity started rstudio-server

step to reproduce

  1. clone this repo
  2. run PORT=11231 PASSWORD=112233 ./run_singularity.sh
  3. go to http://localhost:11231
  4. enter username and password

result

image

expected result

enter the rstudio

more information

I can enter docker started pure rocker/rstudio
I've tested this repo on my own arch installation and a centos 7 server, same result

tried rocker/rstudio:3.6.3 problem exists

How to close R studio server? And access terminal while server is running?

Bit of a newbie here, but when I run R studio server the regular way using r-base on ubuntu i can start and stop the server from terminal, also I can still access my terminal while running r studio server. But when I run it this way the terminal is not accessible and if I stop the session from R studio server menu nothing happens in command line?

error starting rserver

(renv) jerlich@pop-os:~/extrepos/rstudio-server-conda/local$ ./start_rstudio_server.sh 9876
## Current env is >>
/home/jerlich/miniconda3/envs/renv
01 Sep 2021 15:33:31 [rserver] ERROR database error 7 (sqlite3_statement_backend::loadOne: attempt to write a readonly database) [description: Could not delete expired revoked cookies from the database, description: Could not read revoked cookies from the database]; OCCURRED AT virtual rstudio::core::Error rstudio::core::database::Connection::execute(rstudio::core::database::Query&, bool*) src/cpp/core/Database.cpp:510; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:763

Support for containerized Rstudio

It would be nice to run rstudio server from a container:

  • it's easier to setup if you don't have rstudio server already installed
  • it potentially allows to run multiple instances even without rstudio server pro.

Podman version cannot access group directories

When running rstudio in podman, in had the issue that I could not access project directories on our scientific storage to
which I have access based on my secondary user group only.

This is considered a "security feature" of podman. A solution is described in this articel by RedHat. In brief

  • use the crun instead of the runc runtime (install crun and edit ~/.config/containers/containers.conf)
  • run the container with podman run --annotation run.oci.keep_original_groups=1 [...]

This appeared to work in principle (from a shell in the podman container I could access the group directory as expected),
but the rstudio process still got started with different permissions and didn't have access to the directory. I haven't yet figured out how to solve it (except by using Singularity instead (#13)).

Further issues:

  • Due to a bug in podman 3.0.x, annotations are not read from containers.conf, making it impossible to pass the annotation to a container while using podman-compose. This is allegedly fixed in 3.1, but that version is not available on Centos 7.

Extra Simple (Autostarting) `Local` Version

Hi,

Thanks for writing such a foundational useful package - I've pretty much only used R through this package!

This is only tangentially related to this package but could be useful for others.

One difficulty I found was some lab members never wanted to switch environments and preferred to have rstudio auto-start as normal, but with the (base/other) conda environment already setup. It was too easy to forget to activate the conda environment and get subtly different versions.

And I realized I could use the same rsession tricks but at the systemctl level (if the system uses systemd). So

Modify /lib/systemd/system/rstudio-server.service:

ExecStart=/usr/lib/rstudio-server/bin/rserver
-> ExecStart=/root/activateConda

(Run systemctl daemon-reload when first change)

and
Create /root/activateConda: (made executable)

#!/bin/bash
eval "$(/home/jlanglie/anaconda3/bin/conda shell.bash hook)"
source /home/jlanglie/anaconda3/etc/profile.d/conda.sh

conda activate base
/usr/lib/rstudio-server/bin/rserver \
  --rsession-which-r=$(which R)     \
  --rsession-ld-library-path=$CONDA_PREFIX/lib

The main disadvantage is unless you modify that file, you can't easily change the conda environment. But if you did modify the file then you only have to do sudo systemctl restart rstudio-server.service for it to reload.

ERROR after rstudio-server update

Thank you for the tool, it helped me a lot.
However,after I upgrade the rstudio-server, it seems there are some errors.

[rserver] ERROR Attempt to run server as user 'rstudio-server' (uid 999) from account 'aaa' (uid 1006) without privilege, which is required to run as a different uid; LOGGED FROM: virtual rstudio::core::ProgramStatus rstudio::server::Options::read(int, char* const*, std::ostream&) src/cpp/server/ServerOptions.cpp:322

Looking forward to your reply.

cannot clone

ubuntu@VM-8-7-ubuntu:~$ git clone [email protected]:grst/rstudio-server-conda.git
Cloning into 'rstudio-server-conda'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

ERROR system error 13 (Permission denied)

Hi Gregor,

Many thanks for creating this repo. It is very useful for me.

Previously I used this repo (run locally) on a Centos 7.8 system, and everything is perfectly fine. But when I killed the service today and tried to use another port, I met the following error:

[root@localhost ~]# conda activate r4.1
(r4.1) [root@localhost local]# ./start_rstudio_server.sh 8787
## Current env is >>
/data1/Software/Installed/Anaconda3/envs/r4.1
TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/rstudio/logging.conf'. Logging to '/var/log/rstudio/rstudio-server/rserver.log'.
2022-07-21T19:49:00.690672Z [rserver] ERROR system error 13 (Permission denied) [child-path: /data1/Software/PkgDownloads/rstudio-server-conda-master/local/rsession.sh]; OCCURRED AT rstudio::core::Error rstudio::core::system::runProcess(const string&, const string&, rstudio::core::system::ProcessConfig&, rstudio::core::system::ProcessConfigFilter) src/cpp/core/system/PosixSystem.cpp:2320; LOGGED FROM: rstudio::core::Error rstudio::core::system::launchChildProcess(std::string, std::string, rstudio::core::system::ProcessConfig, rstudio::core::system::ProcessConfigFilter, PidType*) src/cpp/core/system/PosixSystem.cpp:2168

The RStudio Server can still be accessed via web page, but once I log into my account, my terminal printed the above "ERROR system error 13" error, and I cannot open the R session. See the figure below:

20220722034926

Any suggestion or feedback would be appreciated!

RETICULATE_PYTHON variable not being recognized with local use

I'm running your local scrip on WSL2 ubuntu 20.04 LTS. I first needed to set global write permissions on /run which then allows the script to run. Unfortunately it looks like the RETICULARE_PYTHON environmental variable is not being correctly set (I guess in rsession.sh) because rstudio server reports the error:

Error in py_discover_config(required_module, use_environment) : 
  Python specified in RETICULATE_PYTHON (/bin/python) does not exist

I also got a
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

error the first time I ran it. I thought that this might be the problem and that the conda env might not be being activated, but rstudio-server is finding the correct R. I ran conda init again anyway and it reported no changes to anything but I am also not getting that error any more so maybe thats sorted? Still getting the reticulate error though. Any ideas?

Adding conda environment within container

Hi @grst ,

thanks for the very useful repository!

I was just wondering if it would be possible to run the rstudio server within a docker container, providing the conda environment with all the dependencies within that container as well, so it can be ported to different instances and the analysis reproduced easily.

I've tried like this:

Dockerfile:

FROM rocker/rstudio

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV CONDA_DIR /home/rstudio/conda

RUN apt-get update --fix-missing \
  && apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion \
  && apt-get clean

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /home/rstudio/miniconda.sh \
    && bash /home/rstudio/miniconda.sh -b -p /home/rstudio/conda \
    && rm -f /home/rstudio/miniconda.sh

ENV PATH=$CONDA_DIR/bin:$PATH
RUN which conda
RUN conda --version

COPY environment.yml /
RUN conda env create -f /environment.yml && conda clean -a -y

RUN conda env list
ENV PATH=$CONDA_DIR/envs/<envname>/bin:$PATH

ADD init2.sh /init2.sh

CMD ["/init2.sh"]

And docker-compose.yml:

version: "3.5"
services:
  rstudio:
    build: .
    # add the image name of your container
    image: <image-tag>
    ulimits:
      nofile: 10000
    ports:
        - "8889:8787"
    volumes:
      # mount the working directory containing your R project.
      - /mnt/volume/qsvac:/home/rstudio
      # share settings between rstudio instances
      - /home/centos/.local/share/rstudio/monitored/user-settings:/home/rstudio/.local/share/rstudio/monitored/user-settings
      # mount tempdir
      - /mnt/volume/rstudio-tmp/var/lib:/var/lib/rstudio-server
      - /mnt/volume/rstudio-tmp/var/run:/var/run/rstudio-server
      - /mnt/volume/rstudio-tmp/tmp:/tmp
    environment:
      - PASSWORD=notsafe
      # repeat the path of the conda environment
      - CONDAENV=/home/rstudio/conda/envs/<conda-env>

Also tested without sharing the rstudio settings and without mounting any tempdirs here. Unfortunately with no luck. If you happen to have an example on how to do that, that would be very helpful!

ERROR database error 7

I have installed rstudio-server on my Ubuntu 20.04 workstation few hours ago. I could run sessions locally on the default port 8787. So far so good.
While attempting to run the server from a conda environment I encountered your solution on SO so I closed all sessions, stopped the server, and followed the recommended procedure.
Could you please help me understanding the returned error? See the screenshot below.

Screenshot from 2021-10-28 11-23-51

I thank you in advance for your kind feedback.

Error: Incorrect or invalid username/password

Thanks for your repository! It helps me a lot in the past two years. I'm using the singularity part

However, after I update R 4.1 to 4.2, the old version of rserver cannot draw figure in the plots channel anymore, and I upgraded to the latest version. I did run the singularity:
image

But I cannot login in the server on explorer, I've tried many combinations: root, whoami, rstudio, and their passwords, none of them success.
image

Do you have any idea? Thanks!

./start_rstudio_server.sh: Does not start

I get error:

./start_rstudio_server.sh: line 14: uuid: command not found
## Current env is >>
/mnt/anaconda3/envs/r_env
unrecognised option '--secure-cookie-key-file'

No Authentication

This is awesome and exactly what I need!
However, when running RStudio Server as a non root user, it seems to bypass the login page.
This is a big security risk for multi users systems.
Is there a way to have authentication functional by still running RStudio Server as a non root user?

Support for containerized R

Can we, as an alternative to conda, also specify a docker/singularity container with R and some libraries preinstalled?

Error when launching RStudio Server: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

With the latest version of RStudio:

wget https://download2.rstudio.org/server/centos6/x86_64/rstudio-server-rhel-1.2.5033-x86_64.rpm
sudo yum install rstudio-server-rhel-1.2.5033-x86_64.rpm

I am getting the following error when trying to launch RStudio Server:

error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

Any Solution? It might be due to a " libssl.so.1.1" installation issue probably.

Alternative to `conda activate`

Hi,

I was interested in using this but got consistent errors that CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'., tracing back to the conda activate line in rsession.sh. After some debugging and following conda/conda#7980, I found replacing
conda activate ${CONDA_ENV}
with
source ~/anaconda3/bin/activate ${CONDA_ENV}
fixed it for me. I'm happy to write up a small PR. This issue might help someone else in case they also have this problem.

secure-cookie-key-file Error

When connecting to some AWS's EMR Clusters, I am getting "secure-cookie-key-file" error, as not being a recognized launch statement.

Is this a know behaviour?

By skipping such --secure-cookie-key-file option I am able to launch RStudio Server however. Is it safe to do so?

Thanks.
Juan

Permission problems with 1.4

Hi,
I have used your scripts to run Rstudio server in my docker container setup, thank you so much! But with the new 1.4 release of RStudio, they changed something when it comes to the sqlite database. I don't know why this only becomes a problem with the newest release.

After the update, Rstudio server just exited after starting and reports:

"ERROR database error 7 (sqlite3_statement_backend::loadOne: attempt to write a readonly database) [description: Could not delete expired revoked cookies from the database, description: Could not read revoked cookies from the database]; OCCURRED AT virtual rstudio::core::Error rstudio::core::database::Connection::execute(rstudio::core::database::Query&, bool*) src/cpp/core/Database.cpp:480; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:729

This seems to be related to https://community.rstudio.com/t/permissions-related-to-upgrade-to-rstudio-server-open-source-1-4/94256/3 and rocker-org/rocker-versioned2#105

The directory /var/lib/rstudio-server contains the sqlite file (/var/lib/rstudio-server/rstudio.sqlite) which needs to be writeable by the user running the server and has a default system daemon user GID UID set by the installer (at least in the deb file).

I fixed it by giving the file to my user executing rstudio server, but deleting the file and allowing other users to write in the parent directory apparently also works.

The better fix may be to give rstudio server the --database-config-file arg flag and provide a config file like this

directory=/var/lib/rstudio-server 

In the default file, all options are commented, so it is basically empty.

Hopefully this helps someone,
Lukas Jansen

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.