Support tuples as result type about crud HOT 9 CLOSED

I like the idea of using tomap in the option name. (use_tomap, tuples_tomap etc.)

Moreover, (in the future) I would like to have two functions for insert - insert_raw (which accepts tuples) and insert (which accepts objects). Rather than other operations, insert accepts tuples or objects, not only returns.

I like the idea to use tuples by default (as @knazarov suggested).

I don't see any reason to discuss on #10 PR until we have a fully discussed and accepted by @mtrempoltsev and @knazarov RFC here.

My suggestion (within #10 ) is:

1. CRUD operations return objects by default (until it's not possible to return tuple with :tomap() method).
2. CRUD operations return tuples (firstly, a simple table objects) if use_tomap/tuples_tomap option is specified.

Later, all operations should return box.tuple objects by default.
In this case, maybe we need to get rid of *_tomap option and simply introduce pair operations: insert -insert_map/insert_obj.

from crud.

NB: A tuple has :tomap() method. Maybe it would be good to borrow the option name from it.

from crud.

We discussed this with @mtrempoltsev and came to the conclusion that it would be impossible to have tuple format appear on the client auto-magically. There are ways to modify Tarantool core to attach format to arbitrary tuples, but it will (a) be a long discussion and (b) will not happen in Tarantool 1.10 (LTS).

For those reasons, we decided to mimic the return style of Tarantool SQL. Namely:

---
- metadata:
  - name: id
    type: unsigned
  - name: first_name
    type: string
  - name: last_name
    type: string
  rows:
  - [1, 'Ivan', 'Ivanov']
  - [2, 'John', 'Doe']
...

This will allow to:

• Use crud from current connectors, without any special support. The return format is quite obvious.
• Save some space by only returning the format once for the result set.

There are also possible optimizations for connectors that would like to add native support of crud. We can add an option to all operations that will return metadata ID instead of the complete description of fields, and create a separate function called crud.get_metadata(id) that will fetch the complete description by the ID. Clients will then be able to cache metadata. This will allow skipping the extra expensive payload, and thus help better utilize the bandwidth.

from crud.

crud.get_metadata(id)

It requires to keep some state into lua land that mean we should manage them by hand and clear after usage.
I suppose It's easy way to get OOM in luajit.

As alternative we could implement dry_run option to return only metadata.

tarantool> crud.select(..., {dry_run = true})
---
- metadata:
  - name: id
    type: unsigned
  - name: first_name
    type: string
  - name: last_name
    type: string

Also we could add skip_metadata option to returns only rows (even enabled by default):

tarantool> crud.select(..., {skip_metadata = true})
---
- rows:
  - [1, 'Ivan', 'Ivanov']
  - [2, 'John', 'Doe']

I think connectors use static set of queries with different arguments.
So it's also a good way to run "dry_run" at initialization step cache schema and then continue work with skip_metadata flag.

from crud.

@olegrok to be honest, I didn't understand your comment. What state are you talking about? You already have this state: when you connect to storage nodes from the router, the router downloads schema from storages.

from crud.

Well, what did you mean when said "client"? I suppose it's some external service that works with crud via connector.

client <--> router <--> storage.

Direct connection to storage always has storage schema: netbox_connecton.space.space_name:format() in case of Tarantool but other connectors fetch schema as well. It means router always knows information about storage schema. And all we need to have a function allows transfer this schema to "client".
Initially it could be really simple:

-- something like
local function get_metadata(space_name)
    local replicasets = vshard.router.routeall()
    local replicaset = select(2, next(replicasets))
    local space = replicaset.master.conn.space[space_name]
    return space:format()
end

In future it could be easily extended if we want to get not all fields but only some subset of fields.

And if I correctly understood you, my suggestion was to allow router proxies schema from storage to external client using special flags to select/insert/update/...

from crud.

@olegrok yes, we agree on the meaning of "client". And the fact that routers have an up-to-date schema.

With a basic implementation, when you return schema with the result set (or metadata, if we name it after the field name), you have the exact format of the tuple when you receive a reply from storage to the router. You can then just pass it back to the client. No additional long-term storage is necessary. You can get rid of the pointer to format as soon as you return the reply.

For the optimized version, you have to keep the format for some time. It is possible that between the select and the request of the format by the client, space format on the storage will change. This indeed will require the router to cache the older format. This can be mitigated by putting a TTL on such records, or just storing old formats in a temporary space, which will save the Lua heap from growing.

And you need to remember that the optimized version is just that -- and optimization. We don't have to implement it right away.

from crud.

As for dry_run and skip_metadata: you can't do that, because the format on the server-side can change. You need a sequence number to differentiate between versions of the format (as is currently implemented in net.box).

from crud.

Ok. Thanks for explanation.

from crud.