ql-io / ql.io Goto Github PK

A node.js based declarative, data-retrieval and aggregation gateway for quickly consuming HTTP APIs

License: Other

Shell 0.14% JavaScript 99.86%

ql.io's Introduction

ql.io is a declarative, data-retrieval and aggregation gateway for quickly consuming HTTP APIs. See ql.io for docs, demos and examples.

How to Build ql.io

To build ql.io on your own, you need node (version 0.8.0 or later) and npm. In Ubuntu, you need libexpat-dev additionally. Once you have these set up, do the following:

git clone git://github.com/ql-io/ql.io.git
cd ql.io
make install

Note If you get "ERR! Error: EACCES, Permission denied" errors, please take a look at npm issue #194 and what-no-sudo.

These steps will link ql.io modules locally so can you refer to those modules from your apps using npm link.

To run tests

make test

ql.io source is organized into several modules that you can test independently.

cd modules/engine
make test

Using ql.io as a Stand-Alone Server

If you are interested in using ql.io as a stand-alone server, setup a new ql.io app and start the server.

mkdir myapp
cd myapp
curl https://raw.github.com/ql-io/ql.io/master/modules/template/init.sh | bash
bin/start.sh

Using latest versions of Firefox or Chrome, go to http://localhost:3000 to see ql.io's Web Console. See the Quickstart Guide for more details.

Using ql.io in a Node App

If you are interested in using ql.io in your node app, use

npm install ql.io-engine

After that you can simply execute the core engine.

var Engine = require('ql.io-engine');
var engine = new Engine({
    connection: 'close'
});

var script = "create table geocoder " +
             "  on select get from 'http://maps.googleapis.com/maps/api/geocode/json?address={address}&sensor=true' " +
             "     resultset 'results.geometry.location'" +
             "select lat as lattitude, lng as longitude from geocoder where address='Mt. Everest'";

engine.execute(script, function(emitter) {
    emitter.on('end', function(err, res) {
        console.log(res.body[0]);
    });
});

Making Contributions

Fixes and features via pull requests are welcome as long as the contributor agrees to the Contributor License Agreement. Print, sign, and email a scanned copy to subbu/AT/ebaysf/DOT/com before submitting the first pull request.

To help move pull requests quickly, consider socializing your idea in the email group.

Discussions

Subscribe to the google group.

ql.io's People

Contributors

Stargazers

Watchers

Forkers

arekatla jcleblanc protez casualjim bizworld earlyster littlemerman ghosthamlet trotter lcameroon xingyunshisui dilipkrish uknownothingsnow tcng idralyuk neopunisher sjcarriere debashissaha narayanaag codesburner kmanley mtipnis ithinkihaveacat nkolekar jocelyn-monitor mulder alexkai souravzzz s3u joonyli fayaza bcriscuolo-ebay triptych jasxie nvdnkpr hochang jmcgdz ayuwxf devcrust ajosepheb revskill10 shimonchayim shotishu sikancil yayat nxbdi skopp cheokman elaherawati patrick-steele-idem inexplicable seanzhou1023 hfeeki saurabhdhupar mobilogic vikramravindhran hashir svoflee fizz jbrittain soulhakr ainewsbot davgit globocom-admin globocom squallssck palnes sourcec0de hunslater-magento n0rig is00hcw imclab dphipps4 web5design ericvicenti ivanstoyanov deepak28290 diorahman eugeneliang xbloom utkarshx halcyonandon animeshinvinci marcusgc bawerd utsengar agilee hot5173 rakesh-mohanta hyramduke deathtrix defconcepts kindy vsteam doooyo yonglehou objectiveous christoffer-ekblom microserviceprovider publicgeneseeq

ql.io's Issues

Schedule statements on rhs of return or inside in clauses sooner

Set fixedGutter option

See http://codemirror.net/manual.html#option_fixedGutter. This requires CodeMirror upgrade.

Be able to consume CSV

There is a lot of CSV data out there!

Show number of requests made in the console

Don't run index-based selectors on strings in JSONPath

It is bad to run foo[2] when foo resolves to string.

Support CORS

Add SHOW Routes

Update CodeMirror for line-wrapping support

count?

How do I know the number of results returned?

Show request response traces in the console

Only the last one survives when a statement makes multiple requests.

Support UPDATE

in should allow numbers

While

select * from foo where id = 1234

is possible,

select * from foo where in in (1234, 3456)

is not. But

select * from foo where in in ('1234', '3456')

is possible.

Upgrade to node >=0.6

Potential things to watch out

Incompatibilities in modules?
Cluster?
Mon module may need significant tweaks

Write log events emitted by engine to std out/err streams

Our logging is very poor - this needs to be improved.

Add required param check for the body

We do this for URI templates but no where else!

Support CSV format

Add an upper limit on no of requests/connections made per script

Also consider req/resp sizes

Assoc array style hashing

This is an idea from Juan.

items1 = select item from foos;
items2 = select item from bars;

Here items1 and items2 are objects with some common fields. Assume that these items have a unique key. The idea is provide a join between these two and hash the response using that key so that the joined response might look like

{
  "key1": {...},
  "key2": {...},
  "key3": {...}.
  ...
}

Provide a way to join arrays

Need to think of a shorter path to supporting such features.

Add timestamp next to inflight requests

Socket timeouts for outgoing requests

Disable auto-run in the console for writes

Such as INSERT and DELETE and UPDATE - run them only when the user says so in the console.

Limit clause

In some cases, we might be interested in getting only the first X results returned by a query. For instance, if we have an API that might return many items that match a given search criteria, we could use a limit clause to get the first 5 of them, like this:

items = select Item from itemAPI where {some condition} limit 5

Additionally, this could also be used to implement pagination for those APIs that support such feature.

items = select Item from itemAPI where {some condition} limit 5, 5

would return the items 5 to 10 from the original response

Something very similar to MySQL's LIMIT syntax (http://dev.mysql.com/doc/refman/5.0/es/select.html):

[LIMIT {[offset,] row_count | row_count OFFSET offset}]

Support XML for output

Really? OK.

Document that table names in from clause must be in the order of dependencies in case of join statements

For instance,

select e.Title as title, e.ItemID as id, g.geometry.location as latlng, e.Location as loc from ebay.shopping.item as e, google.geocode as g where 
  e.itemId in (select itemId from ebay.finding.items where keywords = 'mini') 
 and e.Location = g.address

is valid while

select e.Title as title, e.ItemID as id, g.geometry.location as latlng, e.Location as loc 
     from google.geocode as g, ebay.shopping.item as e where 
      e.itemId in (select itemId from ebay.finding.items where keywords = 'mini') 
     and e.Location = g.address

is not.

select e.Title as title, e.ItemID as id, g.geometry.location as latlng, e.Location as loc from ebay.shopping.item as e,     
    google.geocode as g where 
    e.itemId in (select itemId from ebay.finding.items where keywords = 'mini') 
   and e.Location = g.address

This does rubbish results as the joiner part of the statement sends an undefined location to google.geocode. In other cases, the result set will be empty as the joiner would find no data.

On the otherhand, the same statement without aliases works fine

select e.Title, e.ItemID, g.geometry.location, e.Location from ebay.shopping.item as e,     
    google.geocode as g where 
    e.itemId in (select itemId from ebay.finding.items where keywords = 'mini') 
   and e.Location = g.address