verdict-project / verdict Goto Github PK

Super interesting work. I'm curious to see how it could improve query times in the Druid timeseries database.

One possible reference might be the datasketches extension - though I'm not at all sure whether that's how it would be implemented, knowing little about the technical details of either Druid or Datasketches. 🙂 I'd be interested to see how Verdict would compare to Datasketches in that context.

User should be able to choose any combination of:

• Confidence intervals
• Estimated errors
• Estimated variances

The correct confidence interval is [a,b], where:
a = 2t'-t''_i, b = 2t'-t''_j, t' is the estimated value and t''s are the bootstrap values.

example:
create sample tss from t with size 10%columnsstratified by name;

Don't use UDF and calculate in the middleware.

example:
when result of count on the sample is a and the confidence interval is [s,t] where s < a, more accurate answer c.i. is [a,t]. Because the true answer cannot be smaller than the result on the sample.

good for testing and experiments

maybe cache results?

takes a little bit more space but we can avoid the join and maybe extra groupby

Maybe separate their schemas?

Currently, one can see the DBMS password by:
GET hive.password;

• when creating samples or querying, if user does not specify, use the current schema.
• when creating samples store the full name with schema.

Don't show "impala is not supported by Verdict"

Add seed argument to UDAs + add scale factor argument to conf-int

change output type of AVG + add seed argument for all UDAs + add scale factor argument to conf-int

Consider extending the SQL syntax itself to support options such as sample size, etc.

reason: poisson() is not recalculated for each row

Maybe select the biggest one

Add user manual and tutorial.

Impala and SparkSQL support caching

Exmple:
2*sum(x)
sum(x)+count(y)