this code in file bin/datafire
#!/usr/bin/env node
if (process.version > 'v6') {
module.exports = require('../src/cmd');
} else {
module.exports = require('../distribution/cmd');
}

if process.version = v10
then v10 > v6 gives false . We need to fix it.

✗ High severity vuln found in [email protected], introduced via @datafire/[email protected]
Description: Prototype Override Protection Bypass
Info: https://snyk.io/vuln/npm:qs:20170213
From: @datafire/[email protected] > [email protected] > [email protected] > [email protected] > [email protected]

Just an idea:

# DataFire.yml
paths:
  /hello:
    get:
      action: ./function/hello
      cache: 5

Many times i find myself caching endpoint responses to prevent expense async calls / speed up responses.
I usually use this middleware module cache.js:

var mcache = require('memory-cache');

// usage: 
//   var cache = require('cache')
//   app.get("/foo/bar*", cache(5) ) // cache 5 seconds
 
module.exports = (duration) => {
  return (req, res, next) => {
    let key = JSON.stringify({method:req.method, url:req.url, query:req.query, body:req.body, headers:req.headers })
    let cachedBody = mcache.get(key)
    if (cachedBody) {
      res.header(cachedBody.headers)
      res.send(cachedBody.body)
      return
    } else {
      res.sendResponse = res.send
      res.send = (body) => {
        var obj = {body:body, headers:res.header()._headers}
        mcache.put(key, obj, duration * 1000);
        res.sendResponse(body)
      }
      next()
    }
  }
}

Is this something which might be interesting to include in the datafire module?
It might also give the public datafire.io-server some headroom.

cheers

Just need to resolve $refs from external integrations

I'm getting an issue with a schema that is valid on the latest version of ajv but fails to compile on the version bundled to datafire. It looks like datafire is using ~4 and the latest is on ~6. Is there a reason this wasn't already updated?

e.g.

{
  "page": 3,
  "total": 5034,
  "items": ["..."]
}

We should have a way to maintain state each time an action is run. For instance, a task might crawl a different page of a result set each time it runs, keeping track of state.pageNumber.

It would be easy to keep a small amount of state in-memory, but if the process is restarted, we'll often want to pick up where we left off.

I think loadState and saveState should be optional, asynchronous functions that can be specified by the user. They can write to a local file, a remote database, an in-memory object, or whatever else. They should be able to be specified once for the project as a whole, and overridden for individual actions (maybe inside context?)

Hi,

Is there a codesnippet to embed datafire as middleware for express?
Or the otherway around?..i really would like to add extra middleware before the datafire-middleware kicks in.

This is how i launch it now on my server:

let YAML = require('yamljs');
let path = require('path');
let fs = require('fs');
let datafire = require('datafire/src/lib');

args = {}
args.port = process.env.PORT || 3000;
args.directory = process.env.DATAFIRE_DIR || process.cwd();
let project = datafire.Project.fromDirectory(args.directory);
project.serve(args);

I tried hacking my middleware into express inside a datafire function, but without succes:

var app = require('express')()

app.use( function(req,res,next){
  console.log("unfortunately im not called")
  next()
})

// hack: cut the line in the middleware-chain
app._router.stack.unshift( app._router.stack.pop() )

module.exports = {
  handler: (input) => { .... }
}

However, I would love to add some express middleware like so:

let project = datafire.Project.fromDirectory(args.directory);
project.middleware([
  (req,res,next) => { console.log("i am custom middleware, ran before datafire endpoints") } 
])
project.serve(args);

That way we can re-use existing middleware components from npm.
I know its a niche feature, and not really the point of FaaS, but it would be great to have some analytics on the side like express-status-monitor etc.

Any thoughts on this?

Based on the documentation of the request library, you're incorrectly always defaulting to formData when form should be used for the content type application/x-www-form-urlencoded.

https://www.npmjs.com/package/request#forms

https://github.com/DataFire/DataFire/blob/master/src/lib/openapi-action.js#L62

Given this schema:

        "/users/login": {
            "post": {
                "responses": {
                    "200": {
                        "description": "Successful authentication", 
                        "schema": {
                            "items": {
                                "properties": {
                                    "token": {
                                        "type": "string"
                                    }, 
                                    "id": {
                                        "type": "string"
                                    }, 
                                    "tokenExpires": {
                                        "type": "string"
                                    }
                                }
                            }
                        }
                    }, 
                    "400": {
                        "description": "Error in authentication\n", 
                        "schema": {
                            "items": {
                                "properties": {
                                    "reason": {
                                        "type": "string"
                                    }, 
                                    "error": {
                                        "type": "number"
                                    }
                                }
                            }
                        }
                    }, 
                    "default": {
                        "description": "Error in authentication\n"
                    }
                }, 
                "parameters": [
                    {
                        "required": true, 
                        "type": "string", 
                        "description": "Your username\n", 
                        "in": "formData", 
                        "name": "username"
                    }, 
                    {
                        "in": "formData", 
                        "description": "Your password\n", 
                        "format": "password", 
                        "required": true, 
                        "type": "string", 
                        "name": "password"
                    }
                ], 
                "tags": [
                    "Login"
                ], 
                "operationId": "login", 
                "consumes": [
                    "application/x-www-form-urlencoded", 
                    "application/json"
                ], 
                "summary": "Login with REST API"
            }
        }
    }

You can see it clearly states that it only allows 2 data forms. It cannot accept multipart/form-data, which is what formData is used for in request.

The docs for Swagger 2.0 also show an example that this is the way the swagger spec expects it to be: https://swagger.io/docs/specification/2-0/describing-parameters/#form-parameters

This means that the code should be checking for what the API endpoint consumes and then changing the usage of form or formData accordingly.

Hi,

I tried to follow the hello world example, but I got some problems with it.

module.exports = new datafire.Action({
  handler: (input, context) => {
    if (input.name === 'Voldemort') {
      return new datafire.Response({
        statusCode: 401,
        headers: {'Content-Type': 'text/html'},
        body: "<h1>Nope.</h1>",
      });
    } else {
      return {context, input};
    }
  }
})

In this example the input is always empty. If I check the output, I got the following result.

{
  "context": {
    "results": {},
    "type": "http",
    "accounts": {},
    "startTime": "2017-06-08T08:20:04.600Z",
    "request": {
      "query": {
        "name": "world"
      },
      "headers": {
        "host": "localhost:3000",
        "user-agent": "curl/7.49.1",
        "accept": "*/*"
      },
      "body": {},
      "path": "/hello",
      "method": "GET"
    }
  },
  "input": {}
}

node: v6.10.3
npm: 5.0.3
datafire: v2.2.0

I currently have an issue where I need to ensure secure connections are made with no lower than TLS v1.2.

This can be set using the agentOptions parameter when making a request:

request.get({
    url: 'https://api.some-server.com/',
    agentOptions: {
        secureProtocol: 'TLSv1_2_method'
    }
});

The ability to set request options also has the added benefit of being able to add a certificate and key to make a connection. Not sure what the best way would be to implement this as certificates will vary across services.

Being able to set it for each integration would be best so something like this would be ideal:

var hackerNews = require('@datafire/hacker_news').create();
hackerNews.requestOptions({
    agentOptions: {
        secureProtocol: 'TLSv1_2_method'
    }
});

OR

var hackerNews = require('@datafire/hacker_news').create();
hackerNews.requestOptions({
    agentOptions: {
        cert: fs.readFileSync(certFile),
        key: fs.readFileSync(keyFile),
        passphrase: 'password',
        securityOptions: 'SSL_OP_NO_SSLv3'
    }
});

Thoughts?

• how does the integrate flag work?
• how does one get generated api docs (i noticed swagger-middleware as dependency in package.json), and what's the url?

[sqz@localhost/tmp]  $ node -v
v6.9.1
[sqz@localhost/tmp]  $ datafire integrate --openapi http://petstore.swagger.io/v2/swagger.json
Created integration petstore_swagger in ./integrations/petstore_swagger
path.js:7
    throw new TypeError('Path must be a string. Received ' + inspect(path));
    ^

TypeError: Path must be a string. Received undefined
    at assertPath (path.js:7:11)
    at Object.join (path.js:1211:7)
    at callback (/tmp/node_modules/datafire/src/commands/integrate.js:55:33)
    at fs.writeFile.e (/tmp/node_modules/datafire/src/commands/integrate.js:90:11)
    at /tmp/node_modules/graceful-fs/graceful-fs.js:43:10
    at FSReqWrap.oncomplete (fs.js:123:15)

Not sure if I can explain this right, much less figure it out after a day of banging away but I have an express boilerplate I've been hacking on for a few weeks and don't want to give it up but really, really like your platform and integrations so hoping to get a positive answer...

The stack is here: https://github.com/kevoj/nodetomic-api-swagger and it uses the /api' path. I've tried setting up Datafire on the '/api' path and also '/v1' and also '/df' without success. I can get it to register without any errors (I think is is registering...) but just cannot get any paths or actions to fire. Can I use Datafire in conjunction with an express server already set up for apis on another path than /api so I can add some integrations?

I know this question will sound stupid, but I am developing a DataFire.Action using VSCode. How do I properly debug the handler? Thanks in advance.

Is there an API for DataFire itself so CRUD operations can be performed against integrations, actions, triggers, etc? I know there's DataFire.io but say I wanted to manage these points with another interface, is there an established way of doing that?

Hopefully I didn't miss this somewhere obvious. Thanks!

Hi, is this project no longer in development? there's no update since mid last year. Can anyone update the project status?

thx

Project Server
DataFire listening on port 3333
✓ should respect top-level cache option (130ms)
✓ should respect cache override for path (230ms)
✓ should respet cache disable for path

Version...
Integrate...
npm ERR! Test failed. See above for more details.

Hi,

The link #CustomIntegrationModal seems to be hidden on the Integrations page. it appears for a half a second (see image) but is hidden with CSS. I changed the CSS to:

[hidden] {
    display: inline-block;
}

and the modal popped up - I assume we are not supposed to be using it at this time? (as it is clearly intentionally hidden)

Update: NB: I want to have a try at adding WooCommerce API.

e.g. zoomconnect expects both "user" and "token" headers

I love how the YAML can reference actions from integrations, this promotes re-use of actions, instead of hardcoding methods into the integration.

paths:
    "/db/:id":
        get:
            action: mongodb/findOne

But, how can i reference an action of my own custom integration?

paths:
    "/db/:id":
        get:
            action: mongodb/findOne
        post:
            action: ./myfile.js/mypostAction

This does not work :/ Probably because it applies require() on it.
How about an optional spaceseparated action? something like this:

paths:
    "/db/:id":
        get:
            action: mongodb/findOne
        post:
            action: ./myfile.js mypostAction

or this:

paths:
    "/db/:id":
        get:
            action: mongodb/findOne
        post:
            file: ./myfile.js
            action: mypostAction

Currently hard-coded as 3333

Should be able to detect new/changed items automatically by providing a local file
or a database where changes can be tracked.

datafire.monitor('issues', {
  do: github.get('/user/issues'),
  check: item, cb => {
    cb(null, require('issues.json').indexOf(item.id) === -1) 
  },
  stash: item, cb => {
    let issues = require('issues.json')
    issues.push(item.id)
    fs.writeFile('issues.json', JSON.stringify(issues), cb)
  }
})

for the purpose of taking that JSON dump and rendering it with a static site generator?