deardooley / agave-togo Goto Github PK

The current job submission form needs to be broken out into a service and directive and expanded to support proper parameter types, archiving behavior, runtime overrides, notification and permission settings, etc.

Add a Tagging directive and Service for free form tagging and tagging lookup to each resource and collection.

The existing system addition wizard is a little overwhelming. Users should be able to add a new system with sensible defaults by specifying just the required features. This probably calls for different, streamlined "Add Compute" and "Add Storage" wizards. Minimal set of fields includes:

• System Type: grid of protocol badges.
• Name: a username-qualified id for the system (ex. user picks S3, systemId is -S3. Check for system id availability right away.)
• Host/endpoint + port: basic connectivity info. This would be unnecessary for some services like DropBox and Box.com, but necessary for others like Enterprise Box, SFTP, S3, OwnCloud, etc.
• Credentials: prompt the user for the correct credentials for the given protocol. In the case of cloud and GSI auth, the auth flow should be walked as part of the onboarding process.
• Verify: check the credentials immediately by accessing the system. Prompting the user to fix them if auth failed or show a confirmation message and a link to "browse data", "run code", "invite collaborators", or "set up alerts".

There is currently no support for client application management in ToGo, though the SDK does have support. Minimal support should include:

• Creating new client applications
• Listing client applications
• Updating client description & callback urls
• Regenerating client keys
• Deleting client applications
• Fetching high level usage information about the client

Because the clients API requires password auth, allow the user to optionally encrypt and store their credentials in session storage. If the secrets service is available, optionally store it in there.

Add support for detecting refresh tokens in local storage vs the standard token object returned from an implicit flow. This would likely be a feature primarily used by developers to ensure long-term access, but it can also be helpful when:

• Walking a password flow in situations where the client is trusted or a native app
• Users have an existing local auth cache file such as the cli produces in $HOME/.agave/current and they want to use that rather than manually auth.
• Users want to import their credentials from an external url.

Ideally the deardooley/agave-angular-sdk would handle the token refresh for the user if a refresh token was found. Work would be needed to detect false negatives from the auth server and reset the UI counter upon auth failure.

Localization is primarily present in the file browser. This should be incorporated across the board into the UI. Use google translate api via the the gruntfile plugin to bootstrap the initial language set.

Cloning existing resources could be much easier. Improve this UX by adding Angular Services to handle the modals for custom values passed to the various clone operations.

Replace current dashboard with a live dashboard driven by the stats and notifications APIs.

• Realtime information
• Dashboard editor to add/remove predefined widgets and create user-defined views.
• Default widgets:
  • Data moved/rate
  • Systems to/from
  • Protocols
  • Jobs run total/by app/by date/by data in/by data out/by runtime
  • Apps published
  • Apps shared/run/cloned
  • Login attempts
  • Popular clients
  • System status/monitor events
  • etc...

Development is more productive when the feedback loop is short. ToGo can facilitate easier app on-boarding by enabling a REPL (Read, Evaluate, Print, Loop) GUI for the supported language images in the Exchange. MVP would provide the following features:

Creating a REPL

REPL environments are based on one of the favorite language images from the Exchange. These images are equipped with the appropriate Agave language SDK, dependency resolution, and preconfigured toolchains to run code.

Users can browse the favorite language images in the Exchange library view and create a new REPL by clicking the "Run as REPL" button in the image details view, or from the image right click context menu. Upon selection, a custom workspace is created in their cloud storage account and the REPL workspace view is loaded.

REPL workspace IDE

A simple web IDE is presented with markup and contextual support for each of the favorite language images from the Exchange. REPL workspaces have a file tree browser rooted at their REPL remote workspace folder as a left hand second level slideout menu. Copy, rename, delete, and mkdir are available commands. Contextual menu items and autocomplete is available for the contents of the workspace.

The user can add any additional assets to their workspace by importing from any public URL, agave URL, git repository, or uploaded from their local system. Additional install dependencies outside of the basic toolchain dependency management (pypy, conda, ant, maven, gradle, composer, npm, node, ppm, clib, conan) is not supported at this time.

Within the workspace, users can edit their files within the browser and save them back to the remote system. They can version, snapshot, or rollback at any time. Auto commit is available as a setting in each REPL session. Behind the scenes, each REPL is assigned a git repository. Interactive work is done against the remote files. Versioned work is pushed into the git repo. There is a hard cap on the file size persisted into the REPL repo.

Invoking a REPL

A user can run their REPL code by clicking the ">_ Run" button. This will result in the workspace being committed and tagged with a run-UUID. Users can go back to any previous REPL run as long as the REPL workspace exists and restore the state prior to the run.

This will open an output panel with the stdout and stderr of the REPL run. The output panel contents are cleared each time. Each REPL run is recorded and available in the REPL history. Total runtime, container id, host, start, stop, submit, and kill timestamps, workspace pre and post manifest, event log, stdout, stderr are captured with the history and available for search.

A copy of the workspace will be copied into the language container at runtime and copied out after the main process exists. This allows for non-overlapping concurrent runs of the workspace by multiple users.

Converting a REPL to an app

Any REPL run can be converted to an app automatically by clicking on the "Save as App" button. This will build a new image from the workspace and language image and save it to a private registry as both a docker and singularity image. The app runtime environment will be captured and the invocation JSON will be saved as a hidden parameter to the app. All this will make a fully reproducible version of the app.

Binding a REPL to an event

Any REPL can be run in response to an event in the Agave Platform. To convert the REPL to a Reactor, click on the "Save as Reactor" button. The user will be prompted for the event(s) that should trigger an execution of the current REPL. Once specified, the REPL workspace will be built into a new image and pushed into the reactor registry for reactive invocation.

To test the reactive behavior and to debug the REPL event handling code, you can use the environment wizard to add a test event to your REPL runtime environment and run the REPL. This will simulate the exact conditions under which your REPL workspace will be run when the event fires.

Publishing and sharing REPL & workspaces

Any REPL workspace data can be shared with another user or publisehd for public consumption. If another use recycles the workspace in a new REPL, the REPL metadata, git repository, history, etc will all be reset as a new REPL for the new user. To share the REPL itself, a user must click on the "Share" button and enter the username or group with whom they would like to share their REPL.

Similar to the Jobs API, both the REPL run and the output data can be shared and addressed directly through an authenticated URL. Standard publishing syntax used across the rest of the Science APIs allow for a REPL to be made public for viewing or execution by the general public via the REPL API.

Implement global search in the top search bar.

For bonus points, allow for omnibar support to perform basic actions.

The current searches can be improved vastly with some UI polish. One approach would be to support "smart" key/value terms similar to the github omnibar, 2017 aws search bar, and dnauck/angular-advanced-searchbox. The existing ui would simply be replaced with the search terms. This would allow us to restore proper pagination to every collection listing.

Currently all togo settings are stored in the local storage of the browser. The Metadata API provides one means of persistence, but does not isolate access from other applications or togo instances attempting to use the same namespace. The client, group, and profile settings APIs allow a client application to store data within a namespace unique to the client and user or group. Leveraging these services allows custom configurations to be stored securely by applications without fear of leakage or over exposure.

Push the ToGo config into the client settings service on startup and sync user settings with the profile settings service on auth success.

In theory, this would allow ToGo to enforce a whitelist/blacklist, autoconfig, etc without a backend. It wouldn't be the more secure thing in the world, but it would be enough for quite a few user groups.

Given the number of files and project size, we really should be using webpack. let's get on this.

Currently the file browser rewrites absolute paths as relative paths due to a bug in angular's url handling. This needs to be fixed so the root directory of a system can be viewed.

Add custom swagger-codegen templates to generate CRUD UI for boutique APIs to the base ToGo install. Recycling my existing ng-laravel-swagger project to serve as the generator might be the best way to go about it as it's very little work to incorporate and the ToGo dashboard UI is already supported.