Giappo personal utility package
A library of useful functions to save time if you are working with Rstudio or the peregrine cluster.
| Branch | Travis |
|---|---|
master |
 |
develop |
 |
giovanni |
 |
pedro |
 |
pedro-2 |
 |
theo |
 |
How to generate an ssk key for the peregrine cluster: https://wiki.hpc.rug.nl/peregrine/getting_started/logging_in/windows/ssh_key_login
run_on_cluster matches the named elements of list args to the inout function arguments.
However, if the value of an argument is NULL, the value is lost somewhere on the way.
I have an issue with lintr::expect_lint_free() in test-style.R.
It fails when I run tests, returning the output of lintr::lint_package(), and when I run it directly on console I get Error: Reason: On CRAN
Is it supposed to check that everything is linted, or that there is no in-code lint statements?
I got this error trying to build on either develop or master on unix.
Do you have the same issue on windows?
─ installing the package to build vignettes E creating vignettes (6.5s) --- re-building ‘demo.Rmd’ using rmarkdown createTcpServer: address already in use Quitting from lines 32-40 (demo.Rmd) Error: processing vignette 'demo.Rmd' failed with diagnostics: Can't get Google credentials. Are you running googledrive in a non-interactive session? Consider: * drive_deauth()to prevent the attempt to get credentials. * Calldrive_auth()` directly with all necessary specifics.
--- failed re-building ‘demo.Rmd’
--- re-building ‘pocket_demo.Rmd’ using rmarkdown
Quitting from lines 24-55 (pocket_demo.Rmd)
Error: processing vignette 'pocket_demo.Rmd' failed with diagnostics:
there is no package called 'sls'
--- failed re-building ‘pocket_demo.Rmd’
SUMMARY: processing the following files failed:
‘demo.Rmd’ ‘pocket_demo.Rmd’
Error: Vignette re-building failed.
Execution halted
Warning message:
Disconnecting from unused ssh session. Please use ssh_disconnect()
Error: System command 'R' failed, exit status: 1, stdout + stderr:
E> * checking for file ‘/Users/theo/Github/jap/DESCRIPTION’ ... OK
E> * preparing ‘jap’:
E> * checking DESCRIPTION meta-information ... OK
E> * installing the package to build vignettes
E> * creating vignettes ... ERROR
E> --- re-building ‘demo.Rmd’ using rmarkdown
E> createTcpServer: address already in use
E> Quitting from lines 32-40 (demo.Rmd)
E> Error: processing vignette 'demo.Rmd' failed with diagnostics:
E> Can't get Google credentials.
E> Are you running googledrive in a non-interactive session? Consider:
E> * drive_deauth() to prevent the attempt to get credentials.
E> * Call drive_auth() directly with all necessary specifics.
E>
E> --- failed re-building ‘demo.Rmd’
E>
E> --- re-building ‘pocket_demo.Rmd’ using rmarkdown
E> Quitting from lines 24-55 (pocket_demo.Rmd)
E> Error: processing vignette 'pocket_demo.Rmd' failed with diagnostics:
E> there is no package called 'sls'
E> --- failed re-building ‘pocket_demo.Rmd’
E>
E> SUMMARY: processing the following files failed:
E> ‘demo.Rmd’ ‘pocket_demo.Rmd’
E>
E> Error: Vignette re-building failed.
E> Execution halted
E> Warning message:
E> Disconnecting from unused ssh session. Please use ssh_disconnect()
E>
Stack trace:
x System command 'R' failed, exit status: 1, stdout + stderr:
E> * checking for file ‘/Users/theo/Github/jap/DESCRIPTION’ ... OK
E> * preparing ‘jap’:
E> * checking DESCRIPTION meta-information ... OK
E> * installing the package to build vignettes
E> * creating vignettes ... ERROR
E> --- re-building ‘demo.Rmd’ using rmarkdown
E> createTcpServer: address already in use
E> Quitting from lines 32-40 (demo.Rmd)
E> Error: processing vignette 'demo.Rmd' failed with diagnostics:
E> Can't get Google credentials.
E> Are you running googledrive in a non-interactive session? Consider:
E> * drive_deauth() to prevent the attempt to get credentials.
E> * Call drive_auth() directly with all necessary specifics.
E>
E> --- failed re-building ‘demo.Rmd’
E>
E> --- re-building ‘pocket_demo.Rmd’ using rmarkdown
E> Quitting from lines 24-55 (pocket_demo.Rmd)
E> Error: processing vignette 'pocket_demo.Rmd' failed with diagnostics:
E> there is no package called 'sls'
E> --- failed re-building ‘pocket_demo.Rmd’
E>
E> SUMMARY: processing the following files failed:
E> ‘demo.Rmd’ ‘pocket_demo.Rmd’
E>
E> Error: Vignette re-building failed.
E> Execution halted
E> Warning message:
E> Disconnecting from unused ssh session. Please use ssh_disconnect()
E>
Execution halted
Exited with status 1.
`
ctrl + E leads to utter sadness. Vignettes take centuries to build. Probably it cannot deal with the jobs trick
I get that functions are grouped in files by theme, but I find it cumbersome to find a specific function among them. I need to hit Ctrl+Alt+F every time to find where a function is defined.
Could we move each or most functions to a separate .R file?
At the moment the functions I run on the cluster are not running, due to this error, found in the log file:
Error in loadNamespace(name) : there is no package called ‘jap’
Calls: <Anonymous> ... loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution haltedAs far as I can tell, this arises from the call of run_functions or another similar function call, such as here: 2175425#diff-dfac0e28e7132865889d96bcaed539dbR267
I see here that the namespace is called explicitly from this commit onward, when it wasn't before. I can't say for sure, but I suspect that this is what's causing the issue. I haven't changed it as to test it out I will need to push this to master. Is there any reason to call jap:: here? @Giappo, are you able to submit jobs that run at the moment?
On the trial runs I'm doing now I notice that upon saving the output it's being placed in the jap_scripts folder, most likely because that becomes the R working directory. I've checked, and in sls this is done by a save function that has the same folder structure built in. This seems redundant, as the point of the folder handling in jap should take care of this.
Perhaps we could address this by changing the wd while the pipelines run to be /home/$USER/ or /data/$USER/ or something similar, depending on user choice. That way a save() call in the pipeline would have access to the ~/Projects/$PROJECT/results/ path easily.
What do you think? Maybe we can do this with just a simple call to cd on run_on_cluster.bash.
Adding this here so I don't forget, but it's a good idea to find a way to dynamically change the email for the SLURM report on run_on_cluster, something like:
#!/bin/bash
#SBATCH --time=00:04:58 --partition=short
my_email=$3
chosen_partition=gelifesI'm happy to try doing this 👍 Perhaps we could store the email in the Rprofile?
GitHub as a host for parameter files, job scripts and R scripts is not very reliable. It's frequent that overnight the connection to GitHub fails which causes pocket_experiment to break without recovery.
Perhaps we can consider an alternative to downloading these files so often, or, alternatively, have a fallback for when the connection isn't successful.
The function find_github_folder is supposed to be a core function for many other functions. For this reason it should be fast. At the current stage, however, it is too slow.
This is due to the fact that it has to scan your entire disk(s) to look for a folder with a given name (default: "Githubs"). Probably there are tricks to improve the speed of the search.
Currently all main jobs are submitted to the gelifes partition. I would be nice to be able to select other partitions, such as regular to maximize each user's priority. I will do so 👍
So that us lowly non-mac users can check if the package builds also on gentrified neighborhoods, it would be nice to have a travis build for linux and mac.
I took the liberty of adding a .yml on my branch which can easily be merged to develop and master. Could you please enable the travis build on your account and merge (or I can merge myself once enabled).
Running jap::edit_profile() fails under Windows for R version 4.0.0.
The cause is the call to rappdirs::app_dir()$os
rappdirs::app_dir()$os
#> [1] "win"The package checks for "windows". @Giappo, can you please confirm if you get "windows" or "win" on your side? I'm guessing this changed when going from 3.6.3 -> 4.0.0, if you are running 3.6.x it probably still runs so an easy fix that pleases everyone would be checking the string for "win" and "windows", I'd be happy to change that.
The call to ssh::scp_upload() in upload_jap_scripts returns this error:
Error: SCP failure: scp status code 1d not valid
path arguments to scp_upload seem correct (the folders / files do exist) so that's not the issue.
Applying the great technique of googleing the error didn't return any precedent.
Any insight?
checking dependencies in R code ... WARNING
'::' or ':::' imports not declared from:
‘rstudioapi’ ‘usethis’
'library' or 'require' call not declared from: ‘rJava’
'library' or 'require' call to ‘rJava’ in package code.
Please use :: or requireNamespace() instead.
See section 'Suggested packages' in the 'Writing R Extensions' manual.
Unexported object imported by a ':::' call: ‘usethis:::scoped_path_r’
See the note in ?`:::` about the use of this operator.
I don't know what calls throw all these
@Giappo can you add the relevant :: ?
Also add the fields in Imports
The new set of default_X functions allows the user to input their choices, and store them in the .Rprofile.
As of now, default_X functions are asking for user input every time they are called, even if there is already a user choice saved in .Rprofile, in which case they just append a line, and the choice then has duplicate entries in .Rprofile.
It would be great to make the functions detect if there is already a corresponding entry in .Rprofile, and if yes, return it so that default choices can be automatically passed to other jap functions.
checking Rd \usage sections ... WARNING
Undocumented arguments in documentation object 'args_2_string'
‘args’
Undocumented arguments in documentation object 'build_description_file'
‘project_name’ ‘...’
Undocumented arguments in documentation object 'create_folder_structure'
‘projects_folder_name’ ‘home_dir’ ‘cluster_folder’ ‘project_name’
‘drive’
Undocumented arguments in documentation object 'delete_folder_structure'
‘projects_folder_name’ ‘disk’ ‘cluster_folder’ ‘drive’ ‘are_you_sure’
Undocumented arguments in documentation object 'download_project_folder'
‘projects_folder_name’ ‘disk’ ‘cluster_folder’ ‘project_name’
‘delete_on_cluster’ ‘drive’
Undocumented arguments in documentation object 'download_subfolder'
‘subfolder’ ‘projects_folder_name’ ‘disk’ ‘cluster_folder’
‘project_name’ ‘delete_on_cluster’ ‘drive’
Undocumented arguments in documentation object 'drive_dir.create'
‘dir’
Undocumented arguments in documentation object 'drive_list.files'
‘dir’
Undocumented arguments in documentation object 'find_github_folder'
‘folder_name’ ‘disk’
Undocumented arguments in documentation object 'get_function_list'
‘project_name’ ‘my_github’
Undocumented arguments in documentation object 'git_clone'
‘github_name’ ‘github_repo’ ‘...’
Undocumented arguments in documentation object 'git_pull'
‘github_name’ ‘github_repo’
Undocumented arguments in documentation object 'install_package'
‘package_name’ ‘github_name’
Undocumented arguments in documentation object 'list_githubs'
‘...’
Undocumented arguments in documentation object 'open_github_folder'
‘...’
Undocumented arguments in documentation object 'open_github_project'
‘github_repo’ ‘...’
Undocumented arguments in documentation object 'path_2_file.path'
‘path’
Undocumented arguments in documentation object 'plot_matrix'
‘mat’
Undocumented arguments in documentation object 'pocket_experiment'
‘projects_folder_name’ ‘github_name’ ‘project_name’ ‘function_name’
‘params’ ‘cluster_folder’ ‘drive’
Undocumented arguments in documentation object 'remote_dir.create'
‘dir’
Undocumented arguments in documentation object 'remote_dir.exists'
‘dir’
Undocumented arguments in documentation object 'remote_dir.remove'
‘dir’
Undocumented arguments in documentation object 'remote_file.exists'
‘file’
Undocumented arguments in documentation object 'remote_install.packages'
‘github_name’ ‘package_name’ ‘must_sleep’ ‘cluster_folder’
Undocumented arguments in documentation object 'remote_list.files'
‘dir’ ‘cluster_folder’
Undocumented arguments in documentation object 'remove_package'
‘package_name’
Undocumented arguments in documentation object 'run_function'
‘github_name’ ‘package_name’ ‘function_name’ ‘fun_arguments’
Undocumented arguments in documentation object 'run_function_from_file'
‘args_file’
Undocumented arguments in documentation object 'run_on_cluster'
‘github_name’ ‘package_name’ ‘function_name’ ‘fun_arguments’
‘cluster_folder’
Undocumented arguments in documentation object 'run_project_on_cluster'
‘project_name’ ‘function_name’ ‘fun_arguments’
Undocumented arguments in documentation object 'upload_bash_scripts'
‘project_name’
Undocumented arguments in documentation object 'upload_cluster_scripts'
‘project_name’
Undocumented arguments in documentation object 'upload_jap_scripts'
‘cluster_folder’
Functions with \usage entries need to have the appropriate \alias
entries, and all their arguments documented.
The \usage entries must correspond to syntactically valid R code.
See chapter ‘Writing R documentation files’ in the ‘Writing R
Extensions’ manual.
GitHub allows repositories to have wikis to document more complex use cases. It might be a nice idea to have tips and code examples stored there that can work cross-platform.
I volunteer to do so! 👍
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.