Pagination API about sea-orm HOT 27 CLOSED

Can we avoid the turbo fish?

fruit::Entity::find().paginate::<fruit::Model>

from sea-orm.

So, both Select and SelectTwo can be converted into Paginator?

let's say...

• Select => SelectPaginator
• SelectTwo => SelectTwoPaginator

from sea-orm.

Yes, any Select* can be converted into Paginator.
But no, there should only be one Paginator, which is generic over any Select.
Similar idea to the built in traits Iterator and IntoIterator.

from sea-orm.

Getting Stream from db connection

pub trait Connection {
    // new
    async fn fetch(&self, stmt: Statement) -> Pin<Box<dyn Stream<Item = Result<QueryResult, QueryErr>>>>;
}

Will be impl by SelectModel and SelectTwoModel to generalize the conversion from QueryResult to Item

i.e. Item returned by one() and all()

• FromQueryResult
• (FromQueryResult, FromQueryResult)
• serde_json::Value
• etc...

pub trait QueryResultConverter {
    type Item;

    fn convert_query_result(res: &QueryResult) -> Result<Self::Item, QueryErr>;
}

Paginator that accept any QueryResultConverter and stream corresponding Item

pub struct Paginator<C>
where
    C: QueryResultConverter,
{
    pub(crate) query: SelectStatement,
    pub(crate) convertor: PhantomData<C>,
}

impl<C> Paginator<C>
where
    C: QueryResultConverter,
{
    pub async fn fetch(self) -> Pin<Box<dyn Stream<Item = Result<C::Item, QueryErr>>>> {
        todo!();
        let item: C::Item = C::convert_query_result(&row);
    }
}

Still thinking who to keep the pagination state

• page_num
• page_size

from sea-orm.

The Paginator...

use crate::{Database, QueryErr, SelectorTrait};
use futures::Stream;
use sea_query::SelectStatement;
use std::{
    marker::PhantomData,
    pin::Pin,
    task::{Context, Poll},
};

#[derive(Clone, Debug)]
pub struct Paginator<'d, S>
where
    S: SelectorTrait,
{
    pub(crate) query: SelectStatement,
    pub(crate) page: usize,
    pub(crate) page_size: usize,
    pub(crate) buffer: Vec<S::Item>,
    pub(crate) db: &'d Database,
}

impl<'d, S> Paginator<'d, S>
where
    S: SelectorTrait,
{
    pub fn new(query: SelectStatement, page_size: usize, db: &'d Database) -> Self {
        Self {
            query,
            page: 0,
            page_size,
            buffer: Vec::with_capacity(page_size),
            db,
        }
    }
}

impl<S> Stream for Paginator<'_, S>
where
    S: SelectorTrait,
{
    type Item = S::Item;

    fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
        todo!();
    }
}

from sea-orm.

Go see see the POC on pagination branch

from sea-orm.

Need to print some timestamps to assess whether the time sequence is right

from sea-orm.

What do u mean? loll

from sea-orm.

Just found a related article
https://gendignoux.com/blog/2021/04/01/rust-async-streams-futures-part1.html

from sea-orm.

@tyt2y3 check pagination branch again

from sea-orm.

Can we avoid the turbo fish?

fruit::Entity::find().paginate::<fruit::Model>

Why this is called turbo fish? loll

from sea-orm.

https://techblog.tonsser.com/posts/what-is-rusts-turbofish

from sea-orm.

See latest commit

from sea-orm.

Just figure out how to Pin the Stream, so user don't have to Pin it manually

But the method signature be like...

// paginator.rs

pub type PinBoxStream<'db, Item> = Pin<Box<dyn Stream<Item = Item> + 'db>>;

pub fn into_stream(mut self) -> PinBoxStream<'db, Result<Vec<S::Item>, QueryErr>> { ... }
pub fn into_stream_future(mut self) -> PinBoxStream<'db, impl Future<Output = Result<Vec<S::Item>, QueryErr>>> { ... }

from sea-orm.

Trying to get rid of stream! macro

Ref: https://gist.github.com/TimLuq/83a35453405f4c6e0f63fb2a0caa9f6e

from sea-orm.

Please open a PR when it is ready.

from sea-orm.

Can you add some test cases to Paginator?

from sea-orm.

Some how it stuck in infinite loop when running the example

from sea-orm.

Some how it stuck in infinite loop when running the example

I got the same issue. Fixed on latest commit, error caused by code refactoring.

from sea-orm.

Can you add some test cases to Paginator?

How should we test the Paginator?

Perform assert_eq between each elements from...

• Entity::find().all()
• Stream.try_next()

Is this a reasonable test case?

from sea-orm.

Oh it was my bad. But this is the exact problem we'd like to catch in test cases.
Since paging works across multiple database calls, it could only be fully tested with a mock connection.

from sea-orm.

So unit test for paginator will be implemented once mock connection is available?

from sea-orm.

Seems so.

from sea-orm.

Will mark it down on todo list

from sea-orm.

There is now MockDatabase for testing

from sea-orm.

See the draft PR for the first paginator unit test. See if that make sense.

from sea-orm.

1. Should we add some helper to avoid this?
   

   sea-orm/src/executor/paginator.rs

   Lines 113 to 123 in 2b85526

   	// TODO: auto impl
   	impl From<fruit::Model> for MockRow {
   	fn from(model: fruit::Model) -> Self {
   	let map = maplit::btreemap! {
   	"id" => Into::<Value>::into(model.id),
   	"name" => Into::<Value>::into(model.name),
   	"cake_id" => Into::<Value>::into(model.cake_id),
   	};
   	map.into()
   	}
   	}

2. And should we introduce trait IntoMockRow to avoid the manual into.
   

   sea-orm/src/executor/paginator.rs

   Lines 144 to 158 in 2b85526

   	let db = MockDatabase::new()
   	.append_query_results(vec![
   	// TODO: take any IntoMockRow
   	page1
   	.clone()
   	.into_iter()
   	.map(|model| Into::<MockRow>::into(model))
   	.collect(),
   	page2
   	.clone()
   	.into_iter()
   	.map(|model| Into::<MockRow>::into(model))
   	.collect(),
   	])
   	.into_database();

3. Also, I cannot get the transaction log from MockDatabase once it turns into Database.

from sea-orm.