Giter VIP home page Giter VIP logo

intermine_boot's Introduction

Tests

intermine_boot

A little app to spin up local containers in which to build an InterMine

Description

An InterMine, commonly referred to as an InterMine instance or simply a mine, is one of many biological data warehouses based on the InterMine open source software. They provide a webapp and a webservice that multiple InterMine clients in different programming languages (eg. Python and JavaScript) can query to receive integrated biological data.

Building and running an InterMine is an arduous process which requires Linux system adminstration skills, and provisioned servers if you want your InterMine to be publicly available. InterMine Cloud attempts to solve this and lower the barrier to building and running an InterMine instance.

This tool is one part of InterMine Cloud, focused on providing its features in a local environment.

Installation

The package is available on PyPI and can be installed with PIP.

pip install intermine-boot

Make sure the Requirements are installed as well.

Features

Note: This tool is under development and the listed features are only the ones currently implemented (more are planned!).

  • Starting and stopping a complete biotestmine (intermine_boot start local and intermine_boot stop local)
  • Use a custom build of InterMine with flags --build-im, --im-repo and --im-branch

Requirements

Development

Install virtualenv if you haven't already.

$ git submodule update --init
$ virtualenv -p python3 venv
$ . venv/bin/activate
$ pip install --editable .
# Change the source code and call intermine_boot however you want.
$ intermine_boot
# Exit virtualenv when done.
$ deactivate

intermine_boot's People

Contributors

22poojagaur avatar heralden avatar leoank avatar niveditarufus avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

leoank heralden niveditarufus shreyagupta30 sugeeta32 wyrd00 brite107 mightynasty sabreen-parveen 22poojagaur jgarte

intermine_boot's Issues

Improve archive upload/download integration

Build upon the work done in #20

I think it would be a good idea to move the code out of docker.py and into its own file, as it's not very docker related (perhaps archive.py).

We should also use envvars instead of the dict defined in the source file. We can then uncomment the function calls in commands.py and put in a statement to exit early out of the docker.upload_archives and docker.download_archives functions if the AWS envvars aren't defined.

In commands.py, I think it makes sense to place docker.upload_archives under build and docker.download_archives under start. We would then also have to modify the start process to unpack the archive and use it, instead of building from scratch (this is a slightly bigger task than the above points).

Compare and document performance of different compression formats

Problem

We are creating archives of data created by containers for reusing them in future builds.
But, we are not sure which compression format to choose.

Outline of a Solution

Add a markdown that contains a table comparing parameters like speed and ratio of different compression algorithms.

Create archive of custom build of InterMine

Running intermine_boot build local will build and archive an InterMine instance, saving the archives to intermine_boot in XDG_DATA_HOME (use a search engine to find out which folder this is on your operating system).

Currently, only archives for postgres, solr and biotestmine are created. In the case where the user has specified the flag --build-im (optionally along with --im-repo or --im-branch), we should also create an archive for intermine. This is built in the intermine_builder container (https://github.com/intermine/docker-intermine-gradle/blob/a631235c95e5407d9ba928504217c276e319a1d7/intermine_builder/build.sh#L12).

Gradle will install these files to ~/.m2 inside the intermine_builder docker container, so I think archiving this folder will be enough. You will probably need to add a docker volume for this.

Note that this issue will require PRs to this repo, as well as https://github.com/intermine/docker-intermine-gradle.

You can start working on this issue immediately without waiting to be assigned (but please add a comment), even if someone has commented before you. We've decided to not assign issues and allow anyone interested to submit a PR. Obviously we won't be able to merge all the PRs, so don't worry about being first. All PRs whether merged or not, will be part of our evaluation of an intern.

It would also be great if you could explain your approach and rationale in the PR description. This only needs to be as long or as in depth as you feel it needs to be for the specific task.

Test on Windows without WSL (use docker desktop)

This tool has only been tested on Linux. It might require updates to code or documentation if it doesn't work, so please PR these!

You'll want to install this tool and run intermine_boot start local, open the logged web address to check it works, then intermine_boot stop local to stop and remove the docker containers.

You can start working on this issue immediately without waiting to be assigned (but please add a comment), even if someone has commented before you. Having more people test this is helpful!

Update intermine_builder to handle an already built mine

This would require a PR to https://github.com/intermine/docker-intermine-gradle, and probably this repository as well.

intermine_builder is a docker container you can find in the folder of the same name, in the repository linked above. Right now it will build an InterMine instance from scratch every time, but we want it to handle a previously built mine on which it will only run ./gradlew cargoDeployRemote to deploy to the tomcat docker container.

The already built mine will be present in $XDG_DATA_HOME/intermine_boot/docker/data/mine/biotestmine (on Linux this is ~/.local/share/intermine_boot). Currently, intermine_boot will delete the docker folder if it already exists, to start from scratch, so this behaviour will have to be changed to handle this new usecase.

In the future, we will also want to support building a mine saved as an archive, but that isn't a requirement for this issue.

You can start working on this issue immediately without waiting to be assigned (but please add a comment), even if someone has commented before you. We've decided to not assign issues and allow anyone interested to submit a PR. Obviously we won't be able to merge all the PRs, so don't worry about being first. All PRs whether merged or not, will be part of our evaluation of an intern.

It would also be great if you could explain your approach and rationale in the PR description. This only needs to be as long or as in depth as you feel it needs to be for the specific task.

following default instructions results in a build failure

To get here I ran:

pip install intermine_boot
intermine_boot start local

Maybe I did something wrong - let me know. Here's a huge snippet of the error

intermine_builder    | Fri May 15 11:27:19 UTC 2020
intermine_builder    |
intermine_builder    | starting command: /home/intermine/intermine/biotestmine/gradlew --stacktrace --no-daemon postprocess -Pprocess=summarise-objectstore
intermine_builder    |
intermine_builder    | Fri May 15 11:28:09 UTC 2020
intermine_builder    |
intermine_builder    | finished
intermine_builder    |
intermine_builder    |
intermine_builder    | Fri May 15 11:28:09 UTC 2020
intermine_builder    |
intermine_builder    | starting command: /home/intermine/intermine/biotestmine/gradlew --stacktrace --no-daemon postprocess -Pprocess=create-autocomplete-index
intermine_builder    |
intermine_builder    | Fri May 15 11:29:25 UTC 2020
intermine_builder    |
intermine_builder    | finished
intermine_builder    |
intermine_builder    |
intermine_builder    | Fri May 15 11:29:25 UTC 2020
intermine_builder    |
intermine_builder    | starting command: /home/intermine/intermine/biotestmine/gradlew --stacktrace --no-daemon postprocess -Pprocess=create-search-index
intermine_builder    |
intermine_builder    |
intermine_builder    | FAILURE: Build failed with an exception.
intermine_builder    |
intermine_builder    | * Where:
intermine_builder    | Build file '/home/intermine/intermine/biotestmine/webapp/build.gradle' line: 2
intermine_builder    |
intermine_builder    | * What went wrong:
intermine_builder    | A problem occurred evaluating project ':webapp'.
intermine_builder    | > Could not get resource 'https://raw.github.com/akhikhl/gretty/master/pluginScripts/gretty.plugin'.
intermine_builder    |    > Could not HEAD 'https://raw.github.com/akhikhl/gretty/master/pluginScripts/gretty.plugin'. Received status code 503 from server: Backend unavailable, connection timeout
intermine_builder    |
intermine_builder    | * Try:
intermine_builder    | Run with --info or --debug option to get more log output. Run with --scan to get full insights.
intermine_builder    |
intermine_builder    | * Exception is:
intermine_builder    | org.gradle.api.GradleScriptException: A problem occurred evaluating project ':webapp'.
intermine_builder    | 	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:92)
intermine_builder    | 	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl$2.run(DefaultScriptPluginFactory.java:204)
intermine_builder    | 	at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:77)
intermine_builder    | 	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:209)
intermine_builder    | 	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:61)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
intermine_builder    | 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
intermine_builder    | 	at org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:58)
intermine_builder    | 	at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:41)
intermine_builder    | 	at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
intermine_builder    | 	at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
intermine_builder    | 	at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.run(LifecycleProjectEvaluator.java:105)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
intermine_builder    | 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
intermine_builder    | 	at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:68)
intermine_builder    | 	at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:682)
intermine_builder    | 	at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:138)
intermine_builder    | 	at org.gradle.execution.TaskPathProjectEvaluator.configure(TaskPathProjectEvaluator.java:35)
intermine_builder    | 	at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:62)
intermine_builder    | 	at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:41)
intermine_builder    | 	at org.gradle.initialization.DefaultGradleLauncher$ConfigureBuild.run(DefaultGradleLauncher.java:266)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
intermine_builder    | 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
intermine_builder    | 	at org.gradle.initialization.DefaultGradleLauncher.configureBuild(DefaultGradleLauncher.java:179)
intermine_builder    | 	at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:138)
intermine_builder    | 	at org.gradle.initialization.DefaultGradleLauncher.executeTasks(DefaultGradleLauncher.java:121)
intermine_builder    | 	at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:77)
intermine_builder    | 	at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:74)
intermine_builder    | 	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:152)
intermine_builder    | 	at org.gradle.internal.work.StopShieldingWorkerLeaseService.withLocks(StopShieldingWorkerLeaseService.java:38)
intermine_builder    | 	at org.gradle.internal.invocation.GradleBuildController.doBuild(GradleBuildController.java:96)
intermine_builder    | 	at org.gradle.internal.invocation.GradleBuildController.run(GradleBuildController.java:74)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
intermine_builder    | 	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ValidatingBuildActionRunner.run(ValidatingBuildActionRunner.java:32)
intermine_builder    | 	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$3.run(RunAsBuildOperationBuildActionRunner.java:47)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
intermine_builder    | 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
intermine_builder    | 	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner.run(RunAsBuildOperationBuildActionRunner.java:43)
intermine_builder    | 	at org.gradle.tooling.internal.provider.SubscribableBuildActionRunner.run(SubscribableBuildActionRunner.java:51)
intermine_builder    | 	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:50)
intermine_builder    | 	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:46)
intermine_builder    | 	at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:74)
intermine_builder    | 	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:46)
intermine_builder    | 	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:32)
intermine_builder    | 	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:39)
intermine_builder    | 	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:25)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:80)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:53)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:62)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:34)
intermine_builder    | 	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:36)
intermine_builder    | 	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:25)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:43)
intermine_builder    | 	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:29)
intermine_builder    | 	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:59)
intermine_builder    | 	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:31)
intermine_builder    | 	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:59)
intermine_builder    | 	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:44)
intermine_builder    | 	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:46)
intermine_builder    | 	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:30)
intermine_builder    | 	at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:52)
intermine_builder    | 	at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:207)
intermine_builder    | 	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:402)
intermine_builder    | 	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:375)
intermine_builder    | 	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:37)
intermine_builder    | 	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:23)
intermine_builder    | 	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:368)
intermine_builder    | 	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:298)
intermine_builder    | 	at org.gradle.launcher.Main.doAction(Main.java:36)
intermine_builder    | 	at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
intermine_builder    | 	at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:60)
intermine_builder    | 	at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
intermine_builder    | 	at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
intermine_builder    | 	at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:31)
intermine_builder    | 	at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:108)
intermine_builder    | 	at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
intermine_builder    | Caused by: org.gradle.api.resources.ResourceException: Could not get resource 'https://raw.github.com/akhikhl/gretty/master/pluginScripts/gretty.plugin'.
intermine_builder    | 	at org.gradle.internal.resource.ResourceExceptions.failure(ResourceExceptions.java:74)
intermine_builder    | 	at org.gradle.internal.resource.ResourceExceptions.getFailed(ResourceExceptions.java:57)
intermine_builder    | 	at org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceAccessor.resolve(DefaultExternalResourceAccessor.java:67)
intermine_builder    | 	at org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceAccessor.resolveUri(DefaultExternalResourceAccessor.java:47)
intermine_builder    | 	at org.gradle.internal.resource.transfer.DefaultUriTextResourceLoader.loadUri(DefaultUriTextResourceLoader.java:43)
intermine_builder    | 	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyScript(DefaultObjectConfigurationAction.java:103)
intermine_builder    | 	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$000(DefaultObjectConfigurationAction.java:38)
intermine_builder    | 	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$1.run(DefaultObjectConfigurationAction.java:68)
intermine_builder    | 	at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:143)
intermine_builder    | 	at org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:46)
intermine_builder    | 	at org.gradle.api.internal.project.ProjectScript.apply(ProjectScript.java:34)
intermine_builder    | 	at org.gradle.api.Script$apply$0.callCurrent(Unknown Source)
intermine_builder    | 	at build_da72qw3j0wz3a0g7crxnokdmj.run(/home/intermine/intermine/biotestmine/webapp/build.gradle:2)
intermine_builder    | 	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:90)
intermine_builder    | 	... 88 more
intermine_builder    | Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException: Could not HEAD 'https://raw.github.com/akhikhl/gretty/master/pluginScripts/gretty.plugin'. Received status code 503 from server: Backend unavailable, connection timeout
intermine_builder    | 	at org.gradle.internal.resource.transport.http.HttpClientHelper.processResponse(HttpClientHelper.java:158)
intermine_builder    | 	at org.gradle.internal.resource.transport.http.HttpClientHelper.performHead(HttpClientHelper.java:76)
intermine_builder    | 	at org.gradle.internal.resource.transport.http.HttpResourceAccessor.getMetaData(HttpResourceAccessor.java:65)
intermine_builder    | 	at org.gradle.internal.resource.transfer.DefaultExternalResourceConnector.getMetaData(DefaultExternalResourceConnector.java:63)
intermine_builder    | 	at org.gradle.internal.resource.transfer.AccessorBackedExternalResource.getMetaData(AccessorBackedExternalResource.java:201)
intermine_builder    | 	at org.gradle.internal.resource.BuildOperationFiringExternalResourceDecorator$1.call(BuildOperationFiringExternalResourceDecorator.java:61)
intermine_builder    | 	at org.gradle.internal.resource.BuildOperationFiringExternalResourceDecorator$1.call(BuildOperationFiringExternalResourceDecorator.java:58)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:314)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:304)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
intermine_builder    | 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:100)
intermine_builder    | 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
intermine_builder    | 	at org.gradle.internal.resource.BuildOperationFiringExternalResourceDecorator.getMetaData(BuildOperationFiringExternalResourceDecorator.java:58)
intermine_builder    | 	at org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor$1.create(DefaultCacheAwareExternalResourceAccessor.java:101)
intermine_builder    | 	at org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor$1.create(DefaultCacheAwareExternalResourceAccessor.java:81)
intermine_builder    | 	at org.gradle.cache.internal.ProducerGuard$AdaptiveProducerGuard.guardByKey(ProducerGuard.java:97)
intermine_builder    | 	at org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor.getResource(DefaultCacheAwareExternalResourceAccessor.java:81)
intermine_builder    | 	at org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceAccessor.resolve(DefaultExternalResourceAccessor.java:60)
intermine_builder    | 	... 99 more
intermine_builder    |
intermine_builder    |
intermine_builder    | * Get more help at https://help.gradle.org
intermine_builder    |
intermine_builder    | BUILD FAILED in 5s
intermine_builder    | Fri May 15 11:29:31 UTC 2020
intermine_builder    |
intermine_builder    | finished
intermine_builder    |
intermine_builder    |
intermine_builder    | ERROR: 256
intermine_builder    | failed with exit code 0: /home/intermine/intermine/biotestmine/gradlew --stacktrace --no-daemon postprocess -Pprocess=create-search-index
intermine_builder    |
intermine_builder    | check log: pbuild.log
intermine_builder exited with code 1
Build completed. Visit http://localhost:9999/biotestmine to access your mine.

Support building from path to a mine instead of git repo

It seems important to be able to build a mine by its path instead of passing a git repo as an argument.

We would have to add a new argument for this, and handle it by creating a volume to mount that dir in the intermine_builder container.

Add CI end-to-end tests for common usecases

I was a bit reluctant to suggest this to begin with, since they take 15-30 minutes each to run, and if Travis is feeling overloaded it will happily fail them even though nothing is wrong. Optimally, they should also all run in parallel so it won't take 1 hour to run.

Bonus: Look into implementing this with GitHub Actions instead of Travis, which hopefully is more stable.

Here are the commands we should test:

  • intermine_boot start local
  • intermine_boot start local --build-im
  • intermine_boot start local --build-images

These all should be followed by running an HTTP request (probably with curl) to http://localhost:9999/biotestmine and checking for valid response.

Add tests

Problem

Intermine_boot have no tests

Outline of a Solution

Add Unit tests using pytest library

migrate from docker-compse.yml files to docker python library

Problem

Intermine_boot uses docker-compose.yml files from docker-intermine-gradle repo for launching containers. Intermine_boot clones docker-intermine-gradle repo for the files and then open up a shell to execute docker-compose commands.
This approach works, but we cannot guarantee predictable behaviour with this. What if files in the docker-intermine-gradle changes? Also, we have to clone entire docker-intermine-gradle for just two files.

Outline of a Solution

We want to use docker-py library.
Translate docker-compose.yml into python functions using docker-py library.
Then replace the old way of starting containers with these new functions.

Add Travis support

Problem

Intermine boot don't have any CI support yet

Outline of a Solution

  • Add a .travis.yml file.
  • Add scripts to run tests

Clean up containers on failure

It shouldn't be necessary to invoke intermine_boot stop local to clean up lingering containers after a failed build (it also seems that starting a new build in this state can cause faulty web services when the mine finishes building).

Detect when intermine_builder fails and show better exit message

Build completed. Visit http://localhost:9999/biotestmine to access your mine. is always shown when intermine_builder exits. We should check for errors and replace this message with a more appropriate one when things fail.

Checking the exit code might not be enough, as I once got this when gradle failed:

   b'2020-07-16T10:31:05.874461487Z ERROR: 256\n'
   b'2020-07-16T10:31:05.874469812Z failed with exit code 0: /home/intermine/intermine/biotestmine/gradlew --stacktrace --no-daemon builddb\n'
   b'2020-07-16T10:31:05.874478152Z \n'
   b'2020-07-16T10:31:05.874486280Z check log: pbuild.log\n'
   Build completed. Visit http://localhost:9999/biotestmine to access your mine.

failed with exit code 0 sounds like it exits with code 0, which means nothing is wrong! This means we can't rely on the exit code of intermine_builder, so we'll have to check the log strings (hopefully ERROR will be enough).

Support building other mines than biotestmine

The intention of the --im-repo flag is to build mines other than the default biotestmine. While the intermine_builder build.sh script is supposed to be universal, I doubt we'd be able to build a different mine right now without some fiddling.

We can start with testmine, which holds fake employee data that is used for testing. Something tricky about this mine is it's not its own repo like biotestmine, it's instead part of the intermine software repo. This means we can't use --im-repo and need to manually clone it before calling intermine_boot with a path (see #29).

--build-images cannot find docker directory when installed without --editable

This is after initializing the docker-intermine-gradle submodule, when you install intermine_boot into the venv using pip install . (notice that there's no --editable flag). I think this mimics how the package would be installed from pypi (ie. not from a local dir). The contents of intermine_boot dir gets added to venv/lib/python3.8/site-packages/intermine_boot, but not docker-intermine-gradle, hence the TypeError: You must specify a directory to build in path error.

I'm not exactly sure what tweaks need to be done to ensure that docker-intermine-gradle gets added as well. There's some information in #26 but it would need to be investigated further.

When this issue is fixed, I also suggest to install intermine_boot without --editable in the Github actions CI, to better replicate a production environment.

--build-images flag doesn't seem to work

intermine_boot start local --build-images returns the following traceback:

Traceback (most recent call last):
  File "/home/regen/prog/work/intermine_boot/venv/bin/intermine_boot", line 11, in <module>
    load_entry_point('intermine-boot', 'console_scripts', 'intermine_boot')()
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/regen/prog/work/intermine_boot/intermine_boot/__init__.py", line 38, in cli
    commands.invoke(options['mode'], options, env)
  File "/home/regen/prog/work/intermine_boot/intermine_boot/commands.py", line 65, in invoke
    return func(options, env)
  File "/home/regen/prog/work/intermine_boot/intermine_boot/commands.py", line 34, in start
    intermine_docker.up(options, env)
  File "/home/regen/prog/work/intermine_boot/intermine_boot/intermine_docker.py", line 94, in up
    tomcat_image = client.images.build(
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/docker/models/images.py", line 279, in buil
    resp = self.client.api.build(**kwargs)
  File "/home/regen/prog/work/intermine_boot/venv/lib/python3.8/site-packages/docker/api/build.py", line 148, in build
    raise TypeError("You must specify a directory to build in path")
TypeError: You must specify a directory to build in path

Include docker-intermine-gradle as git submodule

This way we can include the Dockerfiles when publishing intermine_boot on pip so that we won't need to clone docker-intermine-gradle anymore, in addition to being aware of the versions of each Dockerfiles that are being included with each pip release.

We'd need to include the other contents of each folder (intermine_builder, postgres, etc.) so we're able to build the dockerfiles.

You may have to experiment a bit on how to access the Dockerfiles when they're placed relative to the source code. This guide seems to be the right approach: https://python-packaging.readthedocs.io/en/latest/non-code-files.html
I would start with putting the submodule inside the intermine_boot directory and point to its contents using a path relative to __file__. You might have to create the MANIFEST.in file as well, and update setup.py.

For an introduction to git submodules, read the beginning of this article: https://git-scm.com/book/en/v2/Git-Tools-Submodules

Test on MacOS

This tool has only been tested on Linux. It might require updates to code or documentation if it doesn't work, so please PR these!

You'll want to install this tool and run intermine_boot start local, open the logged web address to check it works, then intermine_boot stop local to stop and remove the docker containers.

You can start working on this issue immediately without waiting to be assigned (but please add a comment), even if someone has commented before you. Having more people test this is helpful!

Modify remaining mentions of biotestmine to be dynamic

intermine_boot is nearly mine-agnostic; we just need to use _get_mine_name in two more places:

https://github.com/intermine/intermine_boot/blob/6b155b468381e99c93b374b9043ea1c3a5e409da/intermine_boot/archive.py

intermine_boot/intermine_boot/commands.py

Line 36 in a3d5fb7

click.echo('Build completed. Visit http://localhost:9999/biotestmine to access your mine.')

I recommend moving and renaming _get_mine_name to utils.py as get_mine_name, since it will no longer be private to that namespace.

--build-im doesn't seem to work

As a reminder, --build-im builds InterMine before building Biotestmine (so a custom build of InterMine can be used to build Biotestmine).

This might be simply because the dockerhub containers are outdated, so we should test this again combined with --build-images once #33 is fixed.

Add support to upload and fetch data archives from a cloud storage service (like amazon s3 buckets)

Problem

For testing InterMines on a CI platform like travis, we need to build InterMines from scratch. This takes a lot of time.

Outline of a solution

We want intermine_boot to upload data archives to a cloud storage service.
Start with aws s3 buckets. But, implement this in such a way that adding support for new cloud backends will be easier.
These uploaded archives will be reused on the CI platform to reduce build times

Note This issue can take serious amount of time and effort to solve.

Add command to pause docker containers

Something like intermine_boot pause local instead of stop. Instead of removing the docker containers, it would only stop them. resume (or maybe just add a check in start) would then start these existing docker containers (except intermine_builder) and the existing mine should start pretty quickly.

Although start already deploys without rebuilding, a previously built mine, this should be even quicker!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.