We have the following two endpoints:

GET /user/{id}/aws/profiles/{profileId}/maxinstances
GET /user/{id}/aws/profiles/{profileId}/runninginstances

The max number instances that a user can create in a cluster should be the result of the first request minus the second.

Our deployment script creates a directory in the hydra collection and gives them access to it. @jourdain Why do we store the user data under the hydra collection? Doesn't it make more sense to store this user specific data under the user?

When simulation is complete these files are uploaded to Girder, we should provide a mechanism for them to be viewed in the client. They will be particularly important for debugging setup issues in a traditional cluster, for example this is the output for a job where the hydra executable was incorrectly defined:

--------------------------------------------------------------------------
mpiexec was unable to launch the specified application as it could not access
or execute an executable:

Executable: /hydra
Node: ulmus

while attempting to start process rank 0.
--------------------------------------------------------------------------

I keep clicking on it when trying to login. The arrow icon should probably say 'Sign in' or something like that and register button should be a smaller link.

I see the following in the console:

broadcasting SSE: task.status running

However, the simulation status is not updated

A simple overview of resources used, something list:

1 Running instance
2 Volumes

We can work out the endpoints to call.

@jourdain If we remove the ability to upload data to Girder then we will need to disable this feature, which I think we probably want todo anyway?

It should only copy over the simput files.

They don't accrue cost correct?

I'm not sure we want the user to have to type these in each time. We at least need to provide them with away to set defaults.

Hydra executable path
Parallel Environment
Number of slots
Job Output Directory

I have a test instance running at:

http://52.27.36.93/

When I click register nothing happens ... no errors in console.

Currently the simulation status is only updated when a SSE is received. If however the user refreshes their page or logs out when the event is received the simulation will not move into the correct state. So we need to query the status endpoint in these cases.

I am seeing the following error when entering the nuclear simulation project:

POST http://localhost:9000/api/v1/folder?parentId=hydra-th&parentType=collection&name=websimdev&description=websimdev%27s%20projects 400 (Bad Request)

The response is:

{"field": "id", "message": "Invalid ObjectId: hydra-th", "type": "validation"}

Looks like we are passing an invalid ObjectId.

When running visualization on a traditional cluster we will need to be able to configure the ParaView path.

Every time I try to start a simulation I have trouble finding the correct button ...

Enable or disable the upload of the simulation output. This should be used to determine if 'output.item.id' is passed when creating the task. If this value is not specified no upload will occur.

Still see the green rocket, even after termination.

We have cmb in the directory structure etc. Would be nice to clean this up.

I know there is a way fixing this but can't remember it ;-)

A toast appear reporting an error even if the test is successful.

It should be called:

"Output directory"
This should an optional full path to directory where the output of a job will be written to.

The value should be pass to /tasks/<task_id>/run as 'jobOuputDir'

AWS creates a default VPC per region. Starcluster creates its instances in this VPC by default so that the cluster instance are not accessible from the outside world. In order to do this Starcluster must be run from a machine that is in the VPC. Starcluster also has the option to assign public ips to all node in the cluster that then removes this restriction although there is potentially a greater security risk. So what does this mean client? We need to expose and check box indicating whether to use public ip on cluster nodes. This check box will have different behavior depending if the HPCCloud stack is deployed on EC2 or not.

EC2 deployment

• Check box is checked -
  We configure Starcluster to use public IP and the pull list of AWS regions is available to choose from.
• Check box is unchecked - We have to restrict the regions that can be used to the region the stack has been deployed in so Starcluster will have access to the default VPC. The region will be configured into a JSON file as part of the deployment process.

Non EC2 deployment

• In this case the only option is to use public IP for every node as Starcluster will be running outside EC2.
• The checkbox should be checked and disabled and all regions will be availble to the user to create clusters in.

(side note, I also filed this in the completely wrong repository).

Both these resources have a log endpoint. It would be nice to expose this in similar way you can view log file on Travis ...

Girder has a plugin that support this, it might be nice to add support in the UI for this. I have done this for another project so have a bit of a idea of what we need to do.

So once Kitware/cumulus#35 is merged the backend will produce server side events for status update, this should reduce the amount of polling that we will need to do in the UI. There might be some nice angular way of access these events, but here is some sample code in raw javascript.

        if (window.EventSource) {
            // The timeout is the number of second we should wait without an event before closing the connection ...
            url = girder.apiRoot + '/notification/stream?timeout=300';

            var eventSource = new window.EventSource(url);

            eventSource.onmessage = function (e) {
                var obj;
                try {
                    obj = window.JSON.parse(e.data);
                } catch (err) {
                    console.error('Invalid JSON from SSE stream: ' + e.data + ',' + err);
                    return;
                }
                /* Do something useful with the event. See description of event object below */
            };
        } else {
            console.error('EventSource is not supported on this platform.');
        }

The event object have the following form:

{
  "type": "<profile.status|job.status|cluster.status|task.status>",
  "data": {
    "_id": "<the id of the resource>",
    "status": "the new status"
  }
  // There are other property but these can be ignored for our purposes
}

Currently the only configurable parameter to expose to the user is the size.

The REST endpoint can be found here: http://localhost:8080/api/v1#!/volumes

These operations can take several seconds. We need some sort of progress indicator otherwise I will keep clicking buttons!

It should not be possible to delete a running simulation.

To clean up resources associated with a task the following endpoint should be called:

DELETE /tasks/<task_id>

It would be nice to be able to run another simulation as well has hydra, @jourdain and @patrickoleary can probably advise here.

An AWS profile has the following properties:

• Access Key ID
• Secret Access Key
• Availability Zone
• Name
• Region

The REST endpoint can be found here: http://localhost:8080/api/v1#!/user/create_profile

We could probably overall the style while we are at it

@TristanWright Did we every resolve this issue?

[12:18:50] 'watch' errored after 196 ms
[12:18:50] TypeError: Cannot read property 'prototype' of undefined
at Object.exports.inherits (util.js:556:43)
at Object. (/home/cjh/work/source/HPCCloud/node_modules/browser-sync/node_modules/http-proxy/lib/http-proxy/index.js:111:17)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/cjh/work/source/HPCCloud/node_modules/browser-sync/node_modules/http-proxy/lib/http-proxy.js:4:17)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/cjh/work/source/HPCCloud/node_modules/browser-sync/node_modules/http-proxy/index.js:13:18)

kw.girder.js is getting unreasonably large for one single service much less file. We should split it up into specific endpoints e.g. /user, /tasks, etc; and injecting them as they're needed. Further splitting them into specific girder and plugin endpoints would help a lot too.

The task resource has a log property that we should expose to the user. The property is of the following form:

  "log": [
    {
      "$ref": "http://ulex:9001/api/v1/jobs/56030771f657105ab7913b18/log", ...
    }

This is basically an aggregate log of all the resource create by the task for example the cluster and jobs that are submitted to it. So the pseudo code for tail the task log would be as follows:

while task is running:
fetch task:
for each log in task['log']:
fetch log provide offset of lines already fetched:
add lines to log in ui

The log entries from the resources are of the following form:

{
      "args": [],
      "created": 1442957579.78141,
      "exc_info": null,
      "exc_text": null,
      "filename": "job.py",
      "funcName": "_tail_output",
      "levelname": "INFO",
      "levelno": 20,
      "lineno": 288,
      "module": "job",
      "msecs": 781.4099788665771,
      "msg": "Skipping tail of 5601c8e4f657105a8f7ce936/output/stat.txt as file doesn't currently exist",
      "name": "starcluster",
      "pathname": "/home/cjh/work/source/cumulus/cumulus/starcluster/tasks/job.py",
      "process": 22983,
      "processName": "Worker-2",
      "relativeCreated": 1952257.1358680725,
      "thread": 139862483904320,
      "threadName": "MainThread"
}

For now I think we what to present the following information:

With created converted to a human readable format. I might be nice to allow a user to double click on an entry which would reveal all the log information.

Until a valid profile is configured with should disable the functionality

I think we need to support routes, currently trying to move back breaks things ...

If I switch regions for example from 'us-east-1' to 'us-west-2' the POST request still sends the availability zone for the previous region in this case 'us-east-1a'.