predictorsystem_public's Introduction

PredictorSystem

As organizations realize how data analysis helps them to harness their data and use it to identify new opportunities, the popularity of specialized programming languages like R rises. R is an open source programming language and an environment for statistical computing and graphics, whose increasing notoriety has attracted lots of new users, including everyday users that cannot take the most of the R’s capabilities due to a lack of computing resources. For these reasons, a Volunteer Computing (VC) platform for R software is currently being developed to allow public participants to, voluntarily, share their devices’ idle processing power in exchange for computing credits. These credits can then be used to request for computing power within the platform. In this work we propose a decision system for the mentioned platform that, through estimations, selects the most suitable execution site for a given R script. In order to generate such estimations we follow a history based approach, where we use previous function calls observations to create regression models. The results from this proposed system were validated using the R-Benchmark 25 script, which is globally used in the R community as an utility to measure the R’s performance under different machines.

Important Files

: .R :

rExecutor.R
realArgumentMiner.R
callTreeAnalysis.R
predictorFunct.R
mutualMinerFunctions.R
drawTree.R
historyReader.R

: .py :

saveModule.py
connectionModule.py
dummyVolunteers.py
dummyMarket.py
offPredictor.py

Francisco Banha's Files

client.R

Important Installations

Python 2.7
R 3.4.0
install Iperf3 on the machine
install: sudo apt-get install python-dev
install the necessary R packages
Use sudo apt-get install r-cran-car in case car package is not installing
install so that the banner can be printedsudo apt-get install xclip || echo "alias clipboard='xclip -sel clip'" >> ~/.bashrc

Important Notes

Código do Francisco Banha (sandbox) não está integrado. Precisa de uma versão modificada do próprio R.
A parte de gravar o ambiente de execução para depois fornecer ao incubator, não está a funcionar. Mas não é essencial visto que o que se quer é a integração com o interpretador do R (ver issue #6)
The price (a.k.a credits per time unit) is not being considered in the decision
The R files have to be run from the indicated folder (ver issue #1)

Main Commands

Run offPredictor.py, dummyVolunteers.py and dummyMarket.py

In order to run the example file:

(...)/PredictorSystem_public/CodeAnalysisRW/Demos$  Rscript testScript.R

Other Commands

In order to populate a certain package with CRAN example file:

(...)/PredictorSystem_public/CodeAnalysisRW$  Rscript populateFunction.R stats

In order to populate a certain function with CRAN example file:

(...)/PredictorSystem_public/CodeAnalysisRW$  Rscript populateFunction.R stats lm

In order to display the records a certain function has in the system:

(...)/PredictorSystem_public/CodeAnalysisRW$  Rscript historyReader.R stats lm

In order to display the records a certain package has in the system:

(...)/PredictorSystem_public/CodeAnalysisRW$  Rscript historyReader.R TOTAL stats

predictorsystem_public's People

Contributors

predictorsystem_public's Issues

Initial Loading Time

The initial loading of the required packages is currently an issue.
This is probably going to be resolved when converted into an R package, but it's an issue nonetheless.

The problem is that there are a few packages that need to be loaded so the code may work properly, which is currently taking up time every time a new analysis is ran.

Aggregate the Incubator to the R interpreter

Have the R interpreter collect data on regular executions and on doing so, populates the execution history of the several functions.

This would mean that a learning mode would have to be considered, where the performance may be somewhat worst than the best possible, but the learning and the generation of new data points would be immediate and "effortless"

Algorithm does not account for breaks

The algorithm does not account for “breaks”, so when inside a loop, it considers as if it executes the full code in every iteration.

Create Package with Code

The code needs to be converted into an R package, so that it may be installed in R like a normal package would.

File access

The whole file access needs to be changed. It currently depends on where the predictorSystemHandler() is called from.

→ Right now I'm always calling it from: ~/Documents/IST/PredictorSystem/CodeAnalysisRW/Demos
→ The function is in: ~/Documents/IST/PredictorSystem/RExecutor/rExecutor.R
→ So the import needs to be: source("../../RExecutor/rExecutor.R", chdir=TRUE)
→ Conclusion: The file access needs to rely on an installation file, which is read everytime the PredictorSystem is iniciated.