tarantool / nginx_upstream_module Goto Github PK

Post, Get, Put, Patch, Delete

Hi, can you test README.md file for english-english

Now we can send any request without any check. Need to have strict mode for each request type:
for example we can enable only PUT request for tnt_pass and all others requests must return 405 'Method is not allowed'.

Subj.

We always wish to get more details about connection like user agent, cookie and etc. The best way I think pass all data to procedures like this:

function nice_method(data, ....)
end

data is json data.
... some data like headers handle defined params at nginx.conf

Need to allow GET, PUT and DELETE queries

GET
for url /nginx_proxy/func_name/
must call stored procedure named func_name without arguments

after #25 it must call stored procedure with request object

PUT
Tarantool api is equal to POST

DELETE
Tarantool api is equal to POST

Хочется переиспользовать код взаимодействия с тарантулом в своем nginx-модуле.

По сути, нам для этого требуется из сишного кода

• инициализировать новый upstream со своими параметрами (это, вроде, можно и так делать)
• делать вызовы тарантул-функции с параметрами
• получать статус выполнения функции (что вернул тарантул/ошибку тарантула/ошибку связи)

Что-то в духе

void callback(id, result) {
 ...
}

void some_nginx_handler(..) {
  tnt_call(func_name, func_args, id, &callback);
}

Please add RPM/DEB packages to tarantool.org repositories using https://github.com/tarantool/build

1. Refactoring existing log levels for query/json_body handlers
2. Add log level debug/info messages -> see dd()
3. Module must print own version to log

we have URL like this:
http://www.example.com/users/323/orders?param1=10
or
http://www.example.com/users/323/messages?param1=10

in nginx config we should write something like this:

server {
      # HTTP [GET | POST | PUT | DELETE] 
      location /users/:id/orders {
        # REST mode on
        tnt_http_rest_methods get
        tnt_method userOrders
        # Pass http headers and uri
        tnt_pass_http_request on;
        # Module on
        tnt_pass backend;
      }

Params from URL and QueryString will be merged into req.query lua table.

t['id'] = 323
t['param1'] = 10

Add a Docker image with preconfigured nginx + nginx_upstream_module.

Need to have 5 options for url http://site.com/x/y/z/?a=1&b=2

{
  uri='http://site.com/x/y/z/?a=1&b=2'
  args= {a='1', b='2'},

  --new options
  method='GET',
  path= '/x/y/z/',
  host= 'site.com',
  proto='http'
}

Subj

As usual some projects can have any protocols over HTTP or serialization isn't equal JSON like google pb, apache avro and etc. Apache avro has own content-type:

Content-Type: "avro/binary"

Nginx module should pass all data to lua procedure into tarantool without deserialization data with headers like that:

function someaction(client, headers, params, data)
    -- do something here
    return headers, result
end

• client should contain ip address and other information
• headers are lua table
• params pased params to http request and also lua table
• data is body of request

Where someaction() is nginx location where will passed data over HTTP.

For each internal execution error nginx returns invalid json.
Example:

function reprod(request)
    local c = {}
    return c.a.b
end

result: GET http://localhost:8080/reprod

    {
        "id": 0,
        "error": {
            "message":"/usr/local/share/tarantool/app.lua:149: attempt to index field 'a' (a nil value)",
            "code":-32800
        }

Missing }, please fix it

Subj on BSD, CentOS etc.
@zloidemon and @Sulverus all is ok?

Hello,

It would be nice to have a possibility to add a header before passing request to tnt.
I don't think it should work any different from proxy_set_header or fastcgi_param.

For example:

tnt_pass_http_request on;
tnt_set_header X-GEO-COUNTRY $geoip_country_code;

This should add X-GEO-COUNTRY value to req.headers table.

Nginx can't see tnt_* options, solved with disabling new style build in config:

#if [ ! -n "$ngx_module_link" ]; then
# __old_style_build=no
#fi

Probably, new style build is incompatible with el/6

2016/08/23 15:00:08 [crit] 1652#1652: *1886 [BUG] failed to transcode output. errcode: '-32603', errmsg: 'json_encode_string: invalid arguments' while reading response header from upstream, client: 127.0.0.1, server: tnt_test, request: "GET /ucp/config/0001 HTTP/1.1", upstream: "tnt://127.0.0.1:4301", host: "sh2.tarantool.org", referrer: "http://sh2.tarantool.org/ucp/config/0001"

request:

GET http://sh2.tarantool.org/ucp/config/0001
Headers:
Bauth-token: <token>

response: HTTP 595

You can reproduce it by returning a large msgpack from tarantool (~1MB+)

Default nginx answer is:

{
  "id": 0,
  "result": [
    [
      {
        "debug": {
          "front": "app-1",
          "auth": false,
          "source": false
        },
        "status": {
          "code": 200,
          "text": "OK"
        }
      }
    ]
  ]
}

We need to switch it to raw tarantool response:

      {
        "debug": {
          "front": "app-1",
          "auth": false,
          "source": false
        },
        "status": {
          "code": 200,
          "text": "OK"
        }
      }

it can be configurable with tnt_raw on; option

./configure: error: SSL modules require the OpenSSL library.
You can either do not enable the modules, or install the OpenSSL library
into the system, or build the OpenSSL library statically from the source
with nginx by using --with-openssl=<path> option.
error: Bad exit status from /var/tmp/rpm-tmp.HQNNhG (%build)

Need to update rpm spec to build nginx in fedora(19,20,21) for i386 and x86_64 arch

We need to update nginx module API:
Old way

function my_stored_proc(arg1, arg2, arg3, ...)
-- some code here
end

New way

function my_stored_proc(request, arg1, arg2, arg3, ...)
-- some code here
end

Request object is a lua table with uri and headers:

request = {
    uri="http://someurl.com/a/b/c/d/e/f",
    headers={
        header1="abc",
        header2="321",
        header3="456",
    }
}

Tested on:

1. 1.7.X, 1.6.X.

JSON RPC usually uses with one location where you pass method and that method called. For that case would be nice fix that:

Call location /api with data:

 {"jsonrpc": "2.0", "method": "sum", "params": [1,2,4], "id": "1"}

That means module has to call remote tarantool server method sum and pass params.

apt-get install libpcre-dev zlib1-dev

E: Unable to locate package libpcre-dev
E: Unable to locate package zlib1-dev

can't build without this libraries
how to solve it?

https://www.nginx.com/blog/dynamic-modules-nginx-1-9-11/?utm_content=buffer7a7f1&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer

Motivation
Tarantool needs build-in http server, thx to @rtsisyk for idea.

v 0.1

• Move transcoding of headers/query from ngx_*_handler to tp_transcode lib.
• Add normal way to pass option to tp_transcode (currently it is abnormal and pure hack!).

• compiling: mention dependencies
• installing - mention how to install when compiling from source, and have systemd/service scripts working (yes, sorry, nobody likes local-dir hacks)
• example: more commens, it should be sufficient to copy-paste the example to sites-available and have it actually work

No need in abbrevation here, it's one time option.

proposal config (all contexts):

location / {
  tnt_user STR;
  tnt_password STR;
}

Subj

nginx config:

upstream backend {
    server 127.0.0.1:3301;
    keepalive 32;
}
server {
    keepalive_requests 100;
    listen       8080;
    server_name  tnt;
    location /subscribe {
        tnt_http_rest_methods post;
        tnt_method "subscribe";
        tnt_pass backend;
    }
}

But it possible to send GET request and receive HTTP 200

tnt_next_upstream option supports only few default strategies to pass a request to the next server. In some cases one may need to do it on Lua errors.

Possible solutions are:

1. Return HTTP 500 on Lua errors and use default mask http_500.
2. Add a special mask for this case (e.g. lua_error).

• In http path method name detection name = fisrt path element. For example, query http://host.com/one/two/three/ method name is equeal three, must be one
• If there is a body in request(POST, PUT), and body contains method field, need to disable http path method name detection
• (optional) need to update README.md with upstream configuration options

config:

 location /ucp {
      # answers check infinity timeout
      tnt_read_timeout 60m;
      # enable lua request object
      if ( $request_method = GET ) {
         tnt_method "read";
      }
      tnt_http_rest_methods get;
      tnt_multireturn_skip_count 2;
      tnt_pure_result on;
      tnt_pass_http_request on parse_args;
      tnt_pass tnt;
    }

result:

{
  "id": 0,
  "result": {
    "status": {
      "code": 200,
      "text": "OK"
    },
    "meta": {
      "debug": {
        "front": "app-2",
        "auth": true,
        "source": false
      },
      "exec_time": 0.005933,
      "related": [
        "/ucp/0001/accounts/account/79031234567/services"
      ]
    },
    "data": {
      "p2": 79031234568,
      "p1": 79031234567,
      "account": 79031234567
    },
    "aux": {
      "hello": "hello 79031234567"
    }
  }
}

Need find the way to catching/avoiding the garbage.
After experiments clearly that is a rare case probably depends from Tarantool version.

Some cases:

1. Parallel in 6 processes where each process calls around 1000000 of pcalls. -- I can't reproduce it since 27/5/2015.

Subj.

GET -> select
DELETE -> remove
PUT -> insert or update or replace

'+'
reduce parsing / Tarantool and Nginx CPU usage;
access_log for debugging.

'-'
URL limitation.

Need to implement test coverage for character escaping. Example:

#put
{
    "method": "insert",
    "params":[{
        "uid":79031234567,
        "date":201607251753,
        "text":"\"201607251753\""
     }]
}

After insert:

#get
{
  "date": 201607251753,
  "uid": 79031234567,
  "text": ""201607251753""
 }

Please add this test case and check yajl

Need to parse URI query params
/restapi?param1=value1&param2=value2
to object like that:
{ "param1": "value1", "param2": "value2" }

Yajl use malloc/realloc/free it adds memory fragmentation.
This behavior could be avoided by using Nginx alocator functions inside Yajl.

For reduce traffic/parse time we may add follow feature:
module in some cases (configurable) will not await method name within incoming JSON, module will read method name from nginx.conf, for instance:

location /tarantool_call
{
    tnt_set_method_name [ $location | on | off ]; <- _always_ call 'tarantool_call' | set name at this location
    tnt_pass some_backend;
}