Sandbox test implementation code about gauge-js HOT 12 CLOSED

So what is our concern here? Is it about test implementation code overwriting globals like gauge(), step-registry, etc.,? Even if the test code is sandboxed, globals like gauge(), beforeSuite() should be made available in the sandbox for the implementation to work.

from gauge-js.

Isolation of test code and runner code.

We will have to expose globals. Consolidating them in one namespace is a different topic, #27.

from gauge-js.

NodeJS VM doesn't allow access to functions like require if we run it in a sandbox. You have to explicitly specify them.
This is how actually module loading in node works. Take a look at here and here.
This stackoverflow question talks about it. I found this library called VM2 which gives a workaround for this.

from gauge-js.

We have to run it in a sandbox. vm2 looks promising. Need to dig into it.

Here's a use case why we need a sandbox:

Currently, if a test code calls process.exit(), it kills the gauge-js process and gauge remains waiting for messages. If we sandbox this, gauge-js does not get killed. It can report back to gauge that the test code has exited and carry on with the rest of the execution.

I see 3 options at the moment:

1. Use node's vm API, and create a sandbox of the required things we want to provide, or,
2. Use the vm2 module, provided it is stable enough, or,
3. Execute test code in a separate child process. This is safest, but I need to research on how to actually implement it.

from gauge-js.

Another thing to note here: Ideally, we should not expose things like stepRegistry, hookRegistry or customMessageRegistry. Test code should have access to only the gauge global object and access its properties and methods.

from gauge-js.

We can discuss the pros and cons of each method and adopt one of them. Moving stepRegistry, hookRegistry and customMessageRegistry out of the global will be done as part of #27 .

from gauge-js.

Yes. There is an overlap with #27.

I looked at vm2. It looks like a nifty wrapper around vm.

The question that has been bothering me is: will calling methods like gauge.step() in the sandbox/child-process actually add to stepRegistry in the parent gauge-js environment/process? If not, shall we need to parse the test code and collect the functions and then let the code execute in a separate environment?

I may be overthinking this problem, but that may be because this is Sunday. Your inputs are appreciated.

from gauge-js.

Well, since the context is sand-boxed, a better approach would be to maintain the stepRegistry inside the sandbox context and when all done, return the stepRegistry and handle it outside.

PS: vm2 seems to solve pretty much of all the drawbacks of vm as quoted in https://github.com/getgauge-contrib/gauge-js/issues/23#issuecomment-183861674

from gauge-js.

The question that has been bothering me is: will calling methods like gauge.step() in the sandbox/child-process actually add to stepRegistry in the parent gauge-js environment/process?

Well, I am not sure either. 😕 Let us try these methods and find out.

from gauge-js.

I tried this vm2.NodeVM() and still got ReferenceError:

var assert = require("assert");
             ^
ReferenceError: require is not define

Here's the code I tested with in impl-loader.js:

var fileUtil = require("./file-util"),
    fs = require("fs"),
    path = require("path"),
    NodeVM = require("vm2").NodeVM;

function loadImpl(projectRoot) {

  var sandbox = {
    gauge: global.gauge,
    beforeScenario: global.beforeScenario,
    dataStore: global.dataStore
  };

  var vm = new NodeVM({ require: true, requireExternal: true, sandbox: sandbox });

  fileUtil.getListOfFilesFromPath(path.join(projectRoot, "tests")).forEach(function(filePath) {
    process.env.GAUGE_STEPFILEPATH = filePath;
    var content = fs.readFileSync(filePath).toString("utf-8");
    vm.run(content);
  });
}

Hope this helps.

from gauge-js.

The main reason for sandboxing (at least in other gauge runners) is to avoid dependency conflicts at runtime. Say gauge-js depends on version 'x' of 'foo' and the test code depends on version 'y', then sandboxing should allow both dependencies to exist in the right context.

You guys are better judge of how to do this, I don't have the JavaScript skills.

Regarding implementation details, the C# runner loads the test code in the sandbox, and builds the Step and Hook registry from within the sandbox. The runner invokes sandbox methods via reflection (perhaps not applicable in JavaScript).

HTH..

from gauge-js.

In this code, we can send any of the response and can access the global variable with global function and can access the whole data returning from VM Sandbox.

 function compiler(){
     let add = x*y
     Return (add);
 }
 compiler();

const vm = new NodeVM({
   sandbox: {
     Return(data) {
       console.log('Data:', data);
     },
     x : 10,=
     y : 20
   },
   require: {
     external: true,
     builtin: ["fs", "path"],
     root: "./",
     mock: {
       fs: {
         readFileSync() {
           return "Nice try!";
         }
       }
     }
   }
 });
 try {
   vm.run(req.query.code);
 } catch (error) {
   console.log("error: ", error);
 }

from gauge-js.