Giter VIP home page Giter VIP logo

nginx_upstream_module's Introduction

Tarantool NginX upstream module


Key features:

  • Both nginx and Tarantool features accessible over HTTP(S).
  • Tarantool methods callable via JSON-RPC or REST.
  • Load balancing with elastic configuration.
  • Backup and fault tolerance.
  • Low overhead.

See more about:

Limitations


  1. WebSockets are not supported until Tarantool supports out-of-band replies.
  2. This module does not support Tarantool 1.6.x starting with 2.4.0. Since then it uses Tarantool 1.7 protocol features.

Docker images


Tarantool NginX upstream module: https://hub.docker.com/r/tarantool/tarantool-nginx

Tarantool: https://hub.docker.com/r/tarantool/tarantool

Status


  • v0.1.4 - Production ready.
  • v0.2.0 - Stable.
  • v0.2.1 - Production ready.
  • v0.2.2 - Stable.
  • v2.3.1 - Production ready.
  • v2.3.2 - production ready.
  • v2.3.7 - Production ready.
  • v2.4.0-beta - Beta.
  • v2.4.6-rc1 - Stable.
  • v2.5-rc{1,2} - Stable.
  • v2.5-stable - Stable.
  • v2.6-rc3 - Stable.

Contents


How to install


Build from source

git clone https://github.com/tarantool/nginx_upstream_module.git nginx_upstream_module
cd nginx_upstream_module
git submodule update --init --recursive
git clone https://github.com/nginx/nginx.git nginx

# Ubuntu
apt-get install libpcre++0 gcc unzip libpcre3-dev zlib1g-dev libssl-dev libxslt-dev

make build-all

Build via nginx 'configure'

Requirements (for details, see REPO_ROOT/Makefile)

libyajl >= 2.0(https://lloyd.github.io/yajl/)
libmsgpuck >= 2.0 (https://github.com/rtsisyk/msgpuck)

$ ./configure --add-module=REPO_ROOT && make

Install on Mac OS X

brew tap denji/nginx
brew install nginx-full --with-tarantool-module

Configure

    ## Typical configuration, for more see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream
    upstream backend {
        server 127.0.0.1:9999 max_fails=1 fail_timeout=30s;
        server 127.0.0.1:10000;

        # ...
        server 127.0.0.1:10001 backup;

        # ...
    }

    server {
      location = /tnt {
        tnt_pass backend;
      }
    }

Back to contents

Test run and notes for contributors

This paragraph is actual only if you choose build from source installation. So that, nginx git repository should be placed into nginx directory within the module repository. But now, you should use make configure-for-testing. It points the path to the necessary for testing configs, so it is needed to run tests.

To run all tests with different nginx versions use ./test/auto.sh. It will checkout several nginx versions, rebuild nginx with the module and run tests on each of those nginx versions.

For selective running, start:

  1. nginx in other terminal with ./nginx/objs/nginx -c conf/nginx.conf,
  2. Tarantool in another terminal with tarantool test/test.lua,
  3. run separate test file, e.g. ./test/basic_features.py.

If you want to add new test, see examples: *_features.py files. Don't forget to include it to test/run_all.sh.

You can set VERBOSE environment variable to True to enable debug info.

Back to contents

REST


Note: since v0.2.0

With this module, you can call Tarantool stored procedures via HTTP REST methods (GET, POST, PUT, PATCH, DELETE).

Example:

    upstream backend {
      # Tarantool hosts
      server 127.0.0.1:9999;
    }

    server {
      # HTTP [GET | POST | PUT | PATCH | DELETE] /tnt_rest?q=1&q=2&q=3
      location /tnt_rest {
        # REST mode on
        tnt_http_rest_methods get post put patch delete; # or all

        # Pass http headers and uri
        tnt_pass_http_request on;

        # Module on
        tnt_pass backend;
      }
    }
-- Tarantool procedure
function tnt_rest(req)
 req.headers -- http headers
 req.uri -- uri
 return { 'ok' }
end
 $> wget NGX_HOST/tnt_rest?arg1=1&argN=N

Back to contents

JSON


Note: since v0.1.4

The module expects JSON posted with HTTP POST, PUT (since v0.2.0), or PATCH (since v2.3.8) and carried in request body.

Server HTTP statuses:

  • OK - response body contains a result or an error; the error may appear only if something wrong happened within Tarantool, for instance: 'method not found'.

  • INTERNAL SERVER ERROR - may appear in many cases, most of them being 'out of memory' error.

  • NOT ALLOWED - in response to anything but a POST request.

  • BAD REQUEST - JSON parse error, empty request body, etc.

  • BAD GATEWAY - lost connection to Tarantool server(s). Since both (i.e. json -> tp and tp -> json) parsers work asynchronously, this error may appear if 'params' or 'method' does not exists in the structure of the incoming JSON, please see the protocol description for more details.

    Note: this behavior will change in the future.

Input JSON form

[ { "method": STR, "params":[arg0 ... argN], "id": UINT }, ...N ]
  • method - a string containing the name of the Tarantool method to be invoked (i.e. Tarantool "call")
  • params - a structured array. Each element is an argument of the Tarantool "call".
  • id - an identifier established by the Client. MUST contain an unsigned number not greater than unsigned int. May be 0.

These all are required fields.

Output JSON form

[ { "result": JSON_RESULT_OBJECT, "id":UINT, "error": { "message": STR, "code": INT } }, ...N ]
  • result - Tarantool execution result (a json object/array, etc). Version 2.4.0+ outputs a raw result, i.e. JSON_RESULT_OBJECT. May be null or undefined.

  • id - DEPRECATED in 2.4.0+ - request id. May be null or undefined.

  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Example

For instance, Tarantool has a stored procedure echo:

function echo(a, b, c, d)
  return a, b, c, d
end

Syntax:

--> data sent to Server
<-- data sent to Client

rpc call 1:

--> { "method": "echo", "params": [42, 23], "id": 1 }
<-- { "id": 1, "result": [42, 23]

rpc call 2:

--> { "method": "echo", "params": [ [ {"hello": "world"} ], "!" ], "id": 2 }
<-- { "id": 2, "result": [ {"hello": "world"} ], "!" ]}

rpc call of a non-existent method:

--> { "method": "echo_2", "id": 1 }
<-- { "error": {"code": -32601, "message": "Method not found"}, "id": 1 }

rpc call with invalid JSON:

--> { "method": "echo", "params": [1, 2, 3, __wrong__ ] }
<-- { "error": { "code": -32700, "message": "Parse error" } }

rpc call Batch:

--> [
      { "method": "echo", "params": [42, 23], "id": 1 },
      { "method": "echo", "params": [ [ {"hello": "world"} ], "!" ], "id": 2 }
]
<-- [
      { "id": 1, "result": [42, 23]},
      { "id": 2, "result" : [{"hello": "world"} ], "!" ]},
]

rpc call Batch of a non-existent method:

--> [
      { "method": "echo_2", "params": [42, 23], "id": 1 },
      { "method": "echo", "params": [ [ {"hello": "world"} ], "!" ], "id": 2 }
]
<-- [
      { "error": {"code": -32601, "message": "Method not found"}, "id": 1 },
      {"id": 2, "result": [ {"hello": "world"} ], "!" ]}
]

rpc call Batch with invalid JSON:

--> [
      { "method": "echo", "params": [42, 23, __wrong__], "id": 1 },
      { "method": "echo", "params": [ [ {"hello": "world"} ], "!" ], "id": 2 }
]
<-- { "error": { "code": -32700, "message": "Parse error" } }

Back to contents

HTTP headers and status


Sometimes you have to set status or headers which came from Tarantool. For this purpose, you have to use something like ngx_lua or ngx_perl, etc.

With the methods, you can also transform the result from Tarantool into something else.

Here is an example with ngx_lua:

  -- Tarantool, stored procedure
  function foo(req, ...)
    local status = 200
    local headers = {
      ["X-Tarantool"] = "FROM_TNT",
    }
    local body = 'It works!'
    return status, headers, body
  end
  # Nginx, configuration

  # If you're experience an problem with lua-resty-core like
  # https://github.com/openresty/lua-nginx-module/issues/1509
  # it can be disabled by the following directive.
  #
  # lua_load_resty_core off;

  # If you're not using lua-resty-core you may need to manually specify a path
  # to cjson module. See the documentation:
  # https://github.com/openresty/lua-nginx-module#lua_package_cpath
  #
  # lua_package_cpath "/path/in/lua/cpath/format/?.so";

  upstream tnt_upstream {
     server 127.0.0.1:9999;
     keepalive 10000;
  }

  location /tnt_proxy {
    internal;
    tnt_method foo;
    tnt_buffer_size 100k;
    tnt_pass_http_request on parse_args;
    tnt_pass tnt_upstream;
  }

  location /api {
    default_type application/json;
    rewrite_by_lua '

       local cjson = require("cjson")

       local map = {
         GET = ngx.HTTP_GET,
         POST = ngx.HTTP_POST,
         PUT = ngx.HTTP_PUT,
         -- ...
       }
       -- hide `{"params": [...]}` from a user
       ngx.req.read_body()
       local body = ngx.req.get_body_data()
       if body then
            body = "{\\"params\\": [" .. body .. "]}"
       end
       local res = ngx.location.capture("/tnt_proxy", {
         args = ngx.var.args,
         method = map[ngx.var.request_method],
         body = body
       })

       if res.status == ngx.HTTP_OK then
         local answ = cjson.decode(res.body)

         -- Read reply
         local result = answ["result"]

         if result ~= nil then
           ngx.status = result[1]
           for k, v in pairs(result[2]) do
             ngx.header[k] = v
           end

           local body = result[3]
           if type(body) == "string" then
             ngx.header["content_type"] = "text/plain"
             ngx.print(body)
           elseif type(body) == "table" then
             local body = cjson.encode(body)
             ngx.say(body)
           else
             ngx.status = 502
             ngx.say("Unexpected response from Tarantool")
           end
         else
           ngx.status = 502
           ngx.say("Tarantool does not work")
         end

         -- Finalize execution
         ngx.exit(ngx.OK)
       else
         ngx.status = res.status
         ngx.say(res.body)
       end
       ';
    }

Back to contents

Directives


tnt_pass

syntax: tnt_pass UPSTREAM

default: no

context: location

Specify the Tarantool server backend.

  upstream tnt_upstream {
     127.0.0.1:9999
  };

  location = /tnt {
    tnt_pass 127.0.0.1:9999;
  }

  location = /tnt_next_location {
     tnt_pass tnt_upstream;
  }

Back to contents

tnt_http_methods

syntax: tnt_http_methods post, put, patch, delete, all

default: post, delete

context: location

Allow to accept one or many http methods. If a method is allowed, the module expects JSON carried in the request body. If tnt_method is not set, then the name of the Tarantool stored procedure is the protocol JSON.

Example:

  location tnt {
    tnt_http_methods delete;
    tnt_pass 127.0.0.1:9999;
  }
  # Call tarantool_stored_procedure_name()
  $> wget --method=delete --body-data='{"method":"lua_function", "params": [], "id": 0}' NGINX_HOST/tnt

Back to contents

tnt_http_rest_methods

syntax: tnt_http_rest_methods get, post, put, patch, delete, all

default: no

context: location

NOTICE: This does not restrict anything. The option just says to NGINX: use this methods for allowing REST requests.

If you have a wish to set some methods as not allowed methods, then please use "if" inside locations.

For example:

if ($request_method !~ ^(GET|POST|HEAD)$) {
    return 405 "Please use HEAD, PATCH and so on";
}

Allow to accept one or more REST methods. If tnt_method is not set, then the name of the Tarantool stored procedure is the first part of the URL path.

Example:

  location tnt {
    tnt_http_rest_methods get;
    tnt_pass 127.0.0.1:9999;
  }
  # Call tarantool_stored_procedure_name()
  $> wget NGINX_HOST/tarantool_stored_procedure_name/some/mega/path?q=1

Back to contents

tnt_pass_http_request

syntax: tnt_pass_http_request [on|off|parse_args|unescape|pass_body|pass_headers_out|parse_urlencoded|pass_subrequest_uri]

default: off

context: location, location if

Allow to pass HTTP headers and queries to Tarantool stored procedures.

Examples #1:

  location tnt {
    # Also, tnt_pass_http_request can be used together with JSON communication
    tnt_http_rest_methods get;

    # [on|of]
    tnt_pass_http_request on;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_stored_procedure_name(req, ...)
    req.headers -- lua table
    req.query -- string
    return { 'OK' }
  end

  -- With parse_args
  function tarantool_stored_procedure_name_1(req, ...)
    req.headers -- lua table
    req.query -- string
    req.args -- query args as lua table
    return { 'OK' }
  end

  -- With pass_body
  function tarantool_stored_procedure_name_2(req, ...)
    req.body -- request body, type string
  end

Examples #2 (pass_headers_out):

  location @tnt {
    tnt_http_rest_methods get;
    tnt_pass_http_request on pass_headers_out;
    tnt_method tarantool_stored_procedure_name;
    tnt_pass 127.0.0.1:9999;
  }

  location / {
    add_header "X-Req-time" "$request_time";
    proxy_pass http://backend;
    post_action @tnt;
  }
  function tarantool_stored_procedure_name(req, ...)
    req.headers -- lua table
    req.headers['X-Req-time'] -- set by add_header
    req.query -- string
    return true
  end

Examples #3 (parse_urlencoded):

  location /tnt {
    tnt_http_rest_methods post;
    tnt_pass_http_request on parse_urlencoded;
    tnt_method tarantool_stored_procedure_name;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_stored_procedure_name(req, ...)
    req.headers -- a lua table
    req.query -- a string
    req.args.q -- 1
    req.args_urlencoded.p -- 2
    return true
  end
  # Call tarantool_stored_procedure_name()
  $> wget NGINX_HOST/tarantool_stored_procedure_name/some/mega/path?q=1 --post-data='p=2'

Examples #4 (pass_subrequest_uri):

  • Origin (unparsed) uri
  location /web {
    # Backend processing /web/foo and replying with X-Accel-Redirect to
    # internal /tnt/bar
    proxy_pass http://x-accel-redirect-backend;
  }
  location /tnt {
    internal;
    tnt_pass_http_request on;
    tnt_method tarantool_xar_handler;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_xar_handler(req, ...)
    print(req.uri) -- /web/foo
    return true
  end
  • Subrequest uri
  location /web {
    # Backend processing /web/foo and replying with X-Accel-Redirect to
    # internal /tnt/bar
    proxy_pass http://x-accel-redirect-backend;
  }
  location /tnt {
    internal;
    tnt_pass_http_request on pass_subrequest_uri;
    tnt_method tarantool_xar_handler;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_xar_handler(req, ...)
    print(req.uri) -- /tnt/bar
    return true
  end

Back to contents

tnt_pass_http_request_buffer_size

syntax: tnt_pass_http_request_buffer_size SIZE

default: 4k, 8k

context: location

Specify the size of the buffer used for tnt_pass_http_request.

Back to contents

tnt_method

syntax: tnt_method STR

default: no

context: location, location if

Specify the Tarantool call method. It can take a nginx's variable.

Examples:

  location tnt {
    # Also tnt_pass_http_request can mix with JSON communication [[
    tnt_http_rest_methods get;
    tnt_method tarantool_stored_procedure_name;
    #]]

    # [on|of]
    tnt_pass_http_request on;
    tnt_pass 127.0.0.1:9999;
  }

  location ~ /api/([-_a-zA-Z0-9/]+)/ {
    # Also tnt_pass_http_request can mix with JSON communication [[
    tnt_http_rest_methods get;
    tnt_method $1;
    #]]

    # [on|of]
    tnt_pass_http_request on;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_stored_procedure_name(req, ...)
    req.headers -- lua table
    req.query -- string
    return { 'OK' }
  end

  function call(req, ...)
    req.headers -- lua table
    req.query -- string
    return req, ...
  end
  # OK Call tarantool_stored_procedure_name()
  $> wget NGINX_HOST/tarantool_stored_procedure_name/some/mega/path?q=1

  # Error Call tarantool_stored_procedure_XXX()
  $> wget NGINX_HOST/tarantool_stored_procedure_XXX/some/mega/path?q=1

  # OK Call api_function
  $> wget NGINX_HOST/api/call/path?q=1

Back to contents

tnt_set_header

syntax: tnt_set_header STR STR

default: no

context: location, location if

Allows redefining or appending fields to the request header passed to the Tarantool proxied server. The value can contain text, variables, and their combinations.

Examples:

  location tnt {
    # Also tnt_pass_http_request can mix with JSON communication [[
    tnt_http_rest_methods get;
    tnt_method tarantool_stored_procedure_name;
    #]]

    tnt_set_header X-Host $host;
    tnt_set_header X-GEO-COUNTRY $geoip_country_code;

    # [on|of]
    tnt_pass_http_request on;
    tnt_pass 127.0.0.1:9999;
  }
  function tarantool_stored_procedure_name(req, ...)
    req.headers['X-Host'] -- a hostname
    req.headers['X-GEO-COUNTRY'] -- a geo country
    return { 'OK' }
  end
  # OK Call tarantool_stored_procedure_name()
  $> wget NGINX_HOST/tarantool_stored_procedure_name/some/mega/path?q=1

Back to contents

tnt_send_timeout

syntax: tnt_send_timeout TIME

default: 60s

context: http, server, location

The timeout for sending TCP requests to the Tarantool server, in seconds by default.

It's wise to always explicitly specify the time unit to avoid confusion. Time units supported are: s(seconds), ms(milliseconds), y(years), M(months), w(weeks), d(days), h(hours), and m(minutes).

Back to contents

tnt_read_timeout

syntax: tnt_read_timeout TIME

default: 60s

context: http, server, location

The timeout for reading TCP responses from the Tarantool server, in seconds by default.

It's wise to always explicitly specify the time unit to avoid confusion. Time units supported are: s(seconds), ms(milliseconds), y(years), M(months), w(weeks), d(days), h(hours), and m(minutes).

Back to contents

tnt_connect_timeout

syntax: tnt_connect_timeout TIME

default: 60s

context: http, server, location

The timeout for connecting to the Tarantool server, in seconds by default.

It's wise to always explicitly specify the time unit to avoid confusion. Time units supported are: s(seconds), ms(milliseconds), y(years), M(months), w(weeks), d(days), h(hours), and m(minutes). This time must be strictly less than 597 hours.

Back to contents

tnt_buffer_size

syntax: tnt_buffer_size SIZE

default: 4k, 8k

context: http, server, location

This buffer size is used for reading Tarantool replies, but it's not required to be as big as the largest possible Tarantool reply.

Back to contents

tnt_next_upstream

syntax: tnt_next_upstream [ error | timeout | invalid_response | off ]

default: error timeout

context: http, server, location

Specify which failure conditions should cause the request to be forwarded to another upstream server. Applies only when the value in tnt_pass is an upstream with two or more servers.

Back to contents

tnt_next_upstream_tries

syntax: tnt_next_upstream_tries SIZE

default: 0

context: http, server, location

Limit the number of possible tries for passing a request to the next server. The 0 value turns off this limitation.

Back to contents

tnt_next_upstream_timeout

syntax: tnt_next_upstream_timeout TIME

default: 0

context: http, server, location

Limit the time during which a request can be passed to the next server. The 0 value turns off this limitation.

Back to contents

tnt_pure_result

syntax: tnt_pure_result [on|off]

default: off

context: http, server, location

Whether to wrap Tarantool response or not.

When this option is off:

{"id":0, "result": [ 1 ]}

When this option is on:

[[1]]

Back to contents

tnt_multireturn_skip_count

DEPRECATED in 2.4.0+, RETURNED IN 2.5.0-rc2+

syntax: tnt_multireturn_skip_count [0|1|2]

default: 0

context: http, server, location

Note: Use this option wisely, it does not validate the outgoing JSON! For details you can check this issue: #102

The module will skip one or more multireturn parts when this option is > 0.

When it is set to 0:

{"id":0, "result": [[1]]}

When it is set to 1:

{"id":0, "result": [1]}

When it is set to 2:

{"id": 0, "result": 1}

Back to contents

Format

syntax: tnt_{OPT} [ARGS] [FMT]

Tarantool stores data in tuples. A tuple is a list of elements. Each element is a value or an object, and each element should have a strong type. The tuple format is called MsgPack, it's like JSON in a binary format.

The main goal of Format (see [FMT] above) is to enable conversion between a query string and MsgPack without losing type information or value.

The syntax is: {QUERY_ARG_NAME}=%{FMT_TYPE}

Please look carefully for yours url encoding!

A good example is (also see examples tnt_update and tnt_upsert):

HTTP GET ... /url?space_id=512&value=some+string
it could be matched by using the following format 'space_id=%%space_id,value=%s'

Also this works with HTTP forms, i.e. HTTP POST, HTTP PUT and so on.

Here is a full list of {FMT_TYPE} types:

TUPLES

%n - int64
%f - float
%d - double
%s - string
%b - boolean

Special types

%%space_id - space_id
%%idx_id - index_id
%%off - [select](#tnt_select) offset
%%lim - [select](#tnt_select) limit
%%it - [select](#tnt_select) iterator type, allowed values are:
                             eq,req,all,lt,le,ge,gt,all_set,any_set,
                             all_non_set,overlaps,neighbor

KEYS (for [tnt_update](#tnt_update))

%kn - int64
%kf - float
%kd - double
%ks - string
%kb - boolean

Operations (for [tnt_upsert](#tnt_upsert))

%on - int64
%of - float
%od - double
%os - string
%ob - boolean

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Back to contents

tnt_insert

syntax: tnt_insert [SIZE or off] [FMT]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing an insert query with Tarantool.

  • The first argument is a space id.
  • The second argument is a format string.

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Examples can be found at:

  • simple_rest_client.py
  • simple_rest_client.sh

Back to contents

tnt_replace

syntax: tnt_replace [SIZE or off] [FMT]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing a replace query with Tarantool.

  • The first argument is a space id.
  • The second argument is a format string.

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Back to contents

tnt_delete

syntax: tnt_delete [SIZE or off] [SIZE or off] [FMT]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing a delete query with Tarantool.

  • The first argument is a space id.
  • The second argument is an index id.
  • The third argument is a format string.

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Back to contents

tnt_select

syntax: tnt_select [SIZE or off] [SIZE or off] [SIZE] [SIZE] [ENUM] [FMT]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing a select query with Tarantool.

  • The first argument is a space id.
  • The second argument is an index id.
  • The third argument is an offset.
  • The fourth argument is an limit.
  • The fifth argument is an iterator type, allowed values are: eq, req, all, lt ,le,ge, gt, all_set, any_set, all_non_set, overlaps, neighbor.
  • The six argument is a format string.

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Back to contents

tnt_select_limit_max

syntax: tnt_select_limit_max [SIZE]

default: 100

context: server, location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This is a constraint to avoid large selects. This is the maximum number of returned tuples per select operation. If the client reaches this limit, then the client gets an error on its side.

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Back to contents

tnt_allowed_spaces

syntax: tnt_allowed_spaces [STR]

default: **

context: server, location, location if

This is a constraint to prohibit access to some spaces. The directive takes an array of Tarantool space id-s (numbers), and each space in the list is allowed to access from the client side.

Example:

location {
  ...
  tnt_allowed_spaces 512,523;
  tnt_insert off "s=%%space_id,i=%%idx_id";
}

Back to contents

tnt_allowed_indexes

syntax: tnt_allowed_indexes [STR]

default: **

context: server, location, location if

This directive works like [tnt_allowed_spaces], but for indexes.

Back to contents

tnt_update

syntax: tnt_update [SIZE or off] [KEYS] [FMT]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing an update query with Tarantool.

  • The first argument is a space id.
  • The second argument is a KEYS (for UPDATE) string. it has special request form: [OPERATION_TYPE],[FIELDNO],[VALUE]
Possible OPERATION_TYPE (char) are:
 + for addition (values must be numeric)
 - for subtraction (values must be numeric)
 & for bitwise AND (values must be unsigned numeric)
 | for bitwise OR (values must be unsigned numeric)
 ^ for bitwise XOR (values must be unsigned numeric)
 : for string splice
 ! for insertion
 # for deletion
 = for assignment

FIELDNO (number) -  what field the operation will apply to. The field number can
be negative, meaning the position from the end of tuple. (#tuple + negative field number + 1)

VALUE (int64, float, double, string, boolean) – what value will be applied

More details could be found here [tarantool.org] (https://tarantool.org/en/doc/1.7/book/box/box_space.html?highlight=update#lua-function.space_object.update)

Before go further and start use this feature please read an example for this section [1].

  • The third argument is a format string.

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Back to contents

tnt_upsert

syntax: tnt_upsert [SIZE or off] [FMT] [OPERATIONS]

default: None

context: location, location if

HTTP methods GET, POST, PUT, PATCH, DELETE

Content-Typer default, application/x-www-form-urlencoded

This directive allows executing an upsert query with Tarantool.

  • The first argument is a space id.
  • The second argument is a format string.
  • The third argument is a OPERATIONS (for UPSERT) string. it has special request form: [OPERATION_TYPE],[FIELDNO],[VALUE]
Possible OPERATION_TYPE (char) are:
 + for addition (values must be numeric)
 - for subtraction (values must be numeric)
 & for bitwise AND (values must be unsigned numeric)
 | for bitwise OR (values must be unsigned numeric)
 ^ for bitwise XOR (values must be unsigned numeric)
 : for string splice
 ! for insertion
 # for deletion
 = for assignment

FIELDNO (number) -  what field the operation will apply to. The field number can
be negative, meaning the position from the end of tuple. (#tuple + negative field number + 1)

VALUE (int64, float, double, string, boolean) – what value will be applied

More details could be found here [tarantool.org] (https://tarantool.org/en/doc/1.7/book/box/box_space.html?highlight=update#box-space-upsert)

Before go further and start use this feature please read an example for this section [1].

[1] Example

Examples can be found at:

  • examples/simple_rest_client.py
  • examples/simple_rest_client.sh

Returns HTTP code 4XX if client's request doesn't well formatted. It means, that this error raised if some of values missed or has wrong type.

Returns HTTP code 5XX if upstream is dead (no ping).

Also it can return an HTTP code 200 with an error formatted in JSON. It happens when Tarantool can't issue a query.

Here is a description:

 {"error": { "message": STR, "code": INT } }
  • error - a structured object which contains an internal error message. This field exists only if an internal error occurred, for instance: "too large request", "input json parse error", etc.

    If this field exists, the input message was probably not passed to the Tarantool backend.

    See "message"/"code" fields for details.

Back to contents

Examples


Python test: test/basic_features.py, test/v20_feautres.py, nginx.dev.conf.

Client-side javascript example: example/echo.html, example/echo.lua.

Back to contents

Performance tuning


Back to contents

Copyright & license


LICENSE

Back to contents

See also


Back to contents

Contacts

Please report bugs at https://github.com/tarantool/nginx_upstream_module/issues.

We also warmly welcome your feedback in the discussion mailing list, [email protected]

Back to contents

nginx_upstream_module's People

Contributors

bigbes avatar dedok avatar icamys avatar igormunkin avatar jobs-git avatar kostja avatar lenkis avatar makorne avatar mngmntt avatar opomuc avatar romanhabibov avatar rtsisyk avatar snoopcatt avatar toogle avatar totktonada avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nginx_upstream_module's Issues

Transparent proxy to the tarantool procedure via nginx

As usual some projects can have any protocols over HTTP or serialization isn't equal JSON like google pb, apache avro and etc. Apache avro has own content-type:

Content-Type: "avro/binary"

Nginx module should pass all data to lua procedure into tarantool without deserialization data with headers like that:

function someaction(client, headers, params, data)
    -- do something here
    return headers, result
end
  • client should contain ip address and other information
  • headers are lua table
  • params pased params to http request and also lua table
  • data is body of request

Where someaction() is nginx location where will passed data over HTTP.

Add Docker images

Add a Docker image with preconfigured nginx + nginx_upstream_module.

transcode bug

2016/08/23 15:00:08 [crit] 1652#1652: *1886 [BUG] failed to transcode output. errcode: '-32603', errmsg: 'json_encode_string: invalid arguments' while reading response header from upstream, client: 127.0.0.1, server: tnt_test, request: "GET /ucp/config/0001 HTTP/1.1", upstream: "tnt://127.0.0.1:4301", host: "sh2.tarantool.org", referrer: "http://sh2.tarantool.org/ucp/config/0001"

request:

GET http://sh2.tarantool.org/ucp/config/0001
Headers:
Bauth-token: <token>

response: HTTP 595

You can reproduce it by returning a large msgpack from tarantool (~1MB+)

Pass headers and etc to remote procedures

We always wish to get more details about connection like user agent, cookie and etc. The best way I think pass all data to procedures like this:

function nice_method(data, ....)
end

data is json data.
... some data like headers handle defined params at nginx.conf

Fix inverted method selection in http path

  • In http path method name detection name = fisrt path element. For example, query http://host.com/one/two/three/ method name is equeal three, must be one
  • If there is a body in request(POST, PUT), and body contains method field, need to disable http path method name detection
  • (optional) need to update README.md with upstream configuration options

nginx does not respect the HTTP method restriction

nginx config:

upstream backend {
    server 127.0.0.1:3301;
    keepalive 32;
}
server {
    keepalive_requests 100;
    listen       8080;
    server_name  tnt;
    location /subscribe {
        tnt_http_rest_methods post;
        tnt_method "subscribe";
        tnt_pass backend;
    }
}

But it possible to send GET request and receive HTTP 200

Method(i.e. call) as url

For reduce traffic/parse time we may add follow feature:
module in some cases (configurable) will not await method name within incoming JSON, module will read method name from nginx.conf, for instance:

location /tarantool_call
{
    tnt_set_method_name [ $location | on | off ]; <- _always_ call 'tarantool_call' | set name at this location
    tnt_pass some_backend;
}

tnt_pure_request option does't work

config:

 location /ucp {
      # answers check infinity timeout
      tnt_read_timeout 60m;
      # enable lua request object
      if ( $request_method = GET ) {
         tnt_method "read";
      }
      tnt_http_rest_methods get;
      tnt_multireturn_skip_count 2;
      tnt_pure_result on;
      tnt_pass_http_request on parse_args;
      tnt_pass tnt;
    }

result:

{
  "id": 0,
  "result": {
    "status": {
      "code": 200,
      "text": "OK"
    },
    "meta": {
      "debug": {
        "front": "app-2",
        "auth": true,
        "source": false
      },
      "exec_time": 0.005933,
      "related": [
        "/ucp/0001/accounts/account/79031234567/services"
      ]
    },
    "data": {
      "p2": 79031234568,
      "p1": 79031234567,
      "account": 79031234567
    },
    "aux": {
      "hello": "hello 79031234567"
    }
  }
}

Reduce memory fragmentation

Yajl use malloc/realloc/free it adds memory fragmentation.
This behavior could be avoided by using Nginx alocator functions inside Yajl.

Strict REST compatibility

Need to allow GET, PUT and DELETE queries

GET
for url /nginx_proxy/func_name/
must call stored procedure named func_name without arguments

after #25 it must call stored procedure with request object

PUT
Tarantool api is equal to POST

DELETE
Tarantool api is equal to POST

Build problem el/6 (centos/rhel)

Nginx can't see tnt_* options, solved with disabling new style build in config:

#if [ ! -n "$ngx_module_link" ]; then
# __old_style_build=no
#fi

Probably, new style build is incompatible with el/6

Build failed on fedora

./configure: error: SSL modules require the OpenSSL library.
You can either do not enable the modules, or install the OpenSSL library
into the system, or build the OpenSSL library statically from the source
with nginx by using --with-openssl=<path> option.
error: Bad exit status from /var/tmp/rpm-tmp.HQNNhG (%build)

Need to update rpm spec to build nginx in fedora(19,20,21) for i386 and x86_64 arch

Garbage after pcall

Need find the way to catching/avoiding the garbage.
After experiments clearly that is a rare case probably depends from Tarantool version.

Some cases:

  1. Parallel in 6 processes where each process calls around 1000000 of pcalls. -- I can't reproduce it since 27/5/2015.

Make module more verbose

  1. Refactoring existing log levels for query/json_body handlers
  2. Add log level debug/info messages -> see dd()
  3. Module must print own version to log

Put http path and headers in stored procedure args

We need to update nginx module API:
Old way

function my_stored_proc(arg1, arg2, arg3, ...)
-- some code here
end

New way

function my_stored_proc(request, arg1, arg2, arg3, ...)
-- some code here
end

Request object is a lua table with uri and headers:

request = {
    uri="http://someurl.com/a/b/c/d/e/f",
    headers={
        header1="abc",
        header2="321",
        header3="456",
    }
}

Disable response wrapper

Default nginx answer is:

{
  "id": 0,
  "result": [
    [
      {
        "debug": {
          "front": "app-1",
          "auth": false,
          "source": false
        },
        "status": {
          "code": 200,
          "text": "OK"
        }
      }
    ]
  ]
}

We need to switch it to raw tarantool response:

      {
        "debug": {
          "front": "app-1",
          "auth": false,
          "source": false
        },
        "status": {
          "code": 200,
          "text": "OK"
        }
      }

it can be configurable with tnt_raw on; option

tp_transcode as standalone library

Motivation
Tarantool needs build-in http server, thx to @rtsisyk for idea.

v 0.1

  • Move transcoding of headers/query from ngx_*_handler to tp_transcode lib.
  • Add normal way to pass option to tp_transcode (currently it is abnormal and pure hack!).

Incorrect output with tarantool runtime error

For each internal execution error nginx returns invalid json.
Example:

function reprod(request)
    local c = {}
    return c.a.b
end

result: GET http://localhost:8080/reprod

    {
        "id": 0,
        "error": {
            "message":"/usr/local/share/tarantool/app.lua:149: attempt to index field 'a' (a nil value)",
            "code":-32800
        }

Missing }, please fix it

Unable to locate package

apt-get install libpcre-dev zlib1-dev

E: Unable to locate package libpcre-dev
E: Unable to locate package zlib1-dev

can't build without this libraries
how to solve it?

Test coverage for escaped characters

Need to implement test coverage for character escaping. Example:

#put
{
    "method": "insert",
    "params":[{
        "uid":79031234567,
        "date":201607251753,
        "text":"\"201607251753\""
     }]
}

After insert:

#get
{
  "date": 201607251753,
  "uid": 79031234567,
  "text": ""201607251753""
 }

Please add this test case and check yajl

Feature request: Command for adding arbitrary headers

Hello,

It would be nice to have a possibility to add a header before passing request to tnt.
I don't think it should work any different from proxy_set_header or fastcgi_param.

For example:

tnt_pass_http_request on;
tnt_set_header X-GEO-COUNTRY $geoip_country_code;

This should add X-GEO-COUNTRY value to req.headers table.

Disable not allowed request types

Now we can send any request without any check. Need to have strict mode for each request type:
for example we can enable only PUT request for tnt_pass and all others requests must return 405 'Method is not allowed'.

Add an option to pass a request to the next server on Lua error

tnt_next_upstream option supports only few default strategies to pass a request to the next server. In some cases one may need to do it on Lua errors.

Possible solutions are:

  1. Return HTTP 500 on Lua errors and use default mask http_500.
  2. Add a special mask for this case (e.g. lua_error).

export tarantool api for other nginx modules

Хочется переиспользовать код взаимодействия с тарантулом в своем nginx-модуле.

По сути, нам для этого требуется из сишного кода

  • инициализировать новый upstream со своими параметрами (это, вроде, можно и так делать)
  • делать вызовы тарантул-функции с параметрами
  • получать статус выполнения функции (что вернул тарантул/ошибку тарантула/ошибку связи)

Что-то в духе

void callback(id, result) {
 ...
}

void some_nginx_handler(..) {
  tnt_call(func_name, func_args, id, &callback);
}

JSON RPC one location issue

JSON RPC usually uses with one location where you pass method and that method called. For that case would be nice fix that:

Call location /api with data:

 {"jsonrpc": "2.0", "method": "sum", "params": [1,2,4], "id": "1"}

That means module has to call remote tarantool server method sum and pass params.

Restful API

GET -> select
DELETE -> remove
PUT -> insert or update or replace

'+'
reduce parsing / Tarantool and Nginx CPU usage;
access_log for debugging.

'-'
URL limitation.

parsing params from URL like in RESTful

we have URL like this:
http://www.example.com/users/323/orders?param1=10
or
http://www.example.com/users/323/messages?param1=10

in nginx config we should write something like this:

server {
      # HTTP [GET | POST | PUT | DELETE] 
      location /users/:id/orders {
        # REST mode on
        tnt_http_rest_methods get
        tnt_method userOrders
        # Pass http headers and uri
        tnt_pass_http_request on;
        # Module on
        tnt_pass backend;
      }

Params from URL and QueryString will be merged into req.query lua table.

t['id'] = 323
t['param1'] = 10

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.