Giter VIP home page Giter VIP logo

tresql's Introduction

tresql Latest versionBuild Status

TreSQL (Tree SQL) is a query language built on top of SQL that can select data into hierarchical JSON objects. This allows to select data for complex input form using one simple query, without any additional server-side programming. TreSQL has very compact notation and provides many powerful shortcuts. It aims to provide complete functionality of SQL for query and data manipulation purposes. In a web application or a client-server application, it can fullfill all the database querying needs for the client. It's Scala API is much more concise than JDBC and can be used for all database requests.

With TreSQL, we provide:

  1. Query and data manipulation language with SQL functionality
  2. Database API from Scala or Java
  3. Web service and simple web form to run queries and receive data in JSON format.
Reasons to use TreSQL: 1. Get hierarchical instead of flat results. 2. API that is much shorter than JDBC. 3. Syntax shortcuts. TreSQL infers information about primary keys, foreign keys, column types [TODO], array binding etc. 4. No superfluous programming or object declaration needed.

Background

TreSQL is written in Scala on top of JDBC. It provides a query language that can be used as a web service, i.e. from Javascript. It also provides an API that can be used from Scala or Java. Since it uses JDBC, it should be compatible with most JDBC supported databases.

One shortcoming of SQL is that it can select data only in terms of tables and records. In a client/web application, you often have a GUI screen where data is based on more than one table, a simple example if which is a parent/child table relationship (master-detail input form). A common solution here requires a server-side layer that executes several SQL queries to extract parent table record and related child records, converts them to corresponding objects and passes to client. One example of such solution is Hibernate to convert table records to their corresponding objects. However, Hibernate requires to redefine data structure in terms of objects that is already nicely defined in terms of SQL tables. It also lacks flexibility filtering result data, requires extra coding, etc.

TreSQL allows client to gather data from database without any server-side programming (or any client-side programming) except a single query to gather data. It does not require declaration of any other custom data objects because all the results are passed in type-unspecific JSON format. It does not sacrifice performance trying to combine a lot of data into one huge SQL query, because one TreSQL query can get split into several SQL statements on the server.

See documentation below how TreSQL can make your life easier (some familiarity with SQL is desirable).

Try it online

You can test TreSQL queries, including all examples from this guide, directly using our web service:
http://tresql.org/querytest/get

This web service is set up with test data described in Appendix Data. (We do no allow data manipulation at this URL).
Note: With this web service you can not only receive result data, but also see SQL statement(s) that are generated from your TreSQL query. This way you can understand the meaning of any TreSQL query even without explanations below.

Quickstart

Consider the following simplified tables for employees and departments with parent-child relationship:

dept(deptno (primary key), dname)
emp(empno (primary key), ename, deptno (references dept))

Consider the following data in those tables:

dept[10, "ACCOUNTING"]
emp[7839, "BLAKE", 10]
emp[7839, "SCOTT", 10]

Example 1. To select employees with their department name, use the following TreSQL query:
emp/dept {empno, ename, dname}

The query returns a JSON object:

[
  {"empno": 7698, "ename": "BLAKE", "dname": "ACCOUNTING"}, 
  {"empno": 7788, "ename": "SCOTT", "dname": "ACCOUNTING"}
]

Example 2. Now let's select parent record and the child records:

dept {
  deptno, 
  dname, 
  |emp [deptno = :1(1)] {empno, ename} emp
}

Result:

[
    {
	"deptno": 10, 
	"dname": "ACCOUNTING", 
	"emp": [
            {"empno": 7698, "ename": "BLAKE"}, 
            {"empno": 7788, "ename": "SCOTT"}
        ]
    }
]

This query returns information from both tables in a JSON object. It is ready to use by the client application.
You can also choose to receive data more concisely in an array format:

[
  [
    10, "ACCOUNTING", [
      [7698, "BLAKE"], 
      [7788, "SCOTT"]
    ]
  ]
]

Syntax explanation

In example 1, the following TreSQL syntax has been used:
emp/dept {empno, ename, dname}

Figure brackets {} denote columns to be selected.
Slash "/" is an Xpath-like notation to follow a foreign key to parent table.
In this example, TreSQL has inferred the foreign key relationship between the two tables. It was not necessary to specify a join condition.

In the example 2:

dept {
  deptno, 
  dname, 
  |emp [deptno = :1(1)] {empno, ename} emp
}

Figure brackets {} again denote objects to be selected. The 3d object is a subselect from emp table to return employee records.
Square brackets [] denote a WHERE condition.
Notation :1(1) is a little tricky. Colon ":" denotes a binding variable. :1(1) instructs to bind value to parent SELECT statement (1 level up) , 1st value. The TreSQL query in this example produces 2 SELECT statements:

1. select deptno, dname from dept  
2. select empno, ename from emp where deptno = :1(1)  

Therefore, :1(1) substitutes deptno value from first select as deptno value in second select statement.

Further reading

Compiling and installing
TreSQL language guide

tresql's People

Contributors

alvitunk avatar guntiso avatar jhult avatar mrumkovskis avatar muntis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

tresql's Issues

Env to immutable

scala> Env.idExpr = _ => "good"
Env.idExpr: String => String = $$Lambda$7389/1661072554@58cc4e77

scala> val goodEnv = Env.withQueryTimeout(0)
goodEnv: org.tresql.Resources = Resources_(conn = null, metadata = org.tresql.metadata.JDBCMetadata$$anon$1@2f57aa8d, dialect = null, idExpr = $$Lambda$7389/1661072554@58cc4e77, queryTimeout = 0, fetchSize = 0, maxResultSize = 0, params = Map())

scala> Env.idExpr
idExpr   idExpr_=

scala> Env.idExpr = _ => "bad"
Env.idExpr: String => String = $$Lambda$7492/1482649145@74ab14c6

scala> goodEnv.idExpr("")
res0: String = bad

Query don't support batching

Then there are case to insert 100000 or more rows executing update is too slow (hours instead of minutes). JDBC suppose to use for this batches, that is not used in TreSQL:

Connection con = null;
PreparedStatement pstmt = null;
try {
pstmt = con.prepareStatement(sql);
for(r:data) {
pstmt.set...(...);
...
pstmt.addBatch();
}
pstmt.executeBatch();
} catch (Exception e) {
....
} finally { //By the way looking on TreSQL code it doesn't close resources in finally block and it also create resource leak on java and db side in case of exception
try{pstmt.close();}catch(Exception e){}
try{con.close();}catch(Exception e){}
}

As I need modify existing project, do you have any function to get sql or prepared statement and bind data to it? Looks like required methods currently is private.

tresql cols minus not except

joins: >-
  rēķins         (# doctype, numurs,  summa, piesaistītā_summa) {
    norekini_rekins doc
    [klients_id = :klients_id & datums_izraksts < :datums? &
     numurs !%~% 'K']
    {'rēķins', numurs, aprekinats,
     (norekini_sasaiste [rēķina_id             = doc.id] {sum(sasaistes_summa)}) -
     (norekini_sasaiste [kredītrēķina_id       = doc.id] {sum(sasaistes_summa)})
    }
  },

produces "except", bet "-" is expected. Hints:

  • in column clause
  • in parentheses
  • single column
  • aggregate function result

optional query parameter with like condition breaks query

The following line in where clause produces error:
lower(nosaukums) ~ like_template(:nosaukums) ?

java.lang.RuntimeException: [1.144] failure: ]' expected but?' found
scala.sys.package$.error(package.scala:27)
org.tresql.QueryBuilder.org$tresql$QueryBuilder$$build(QueryBuilder.scala:514)
org.tresql.QueryBuilder$.apply(QueryBuilder.scala:530)
org.tresql.Query$.apply(Query.scala:19)

Full query:
vklients kli
[lig.kli_id = kli.id] ligums ? lig
[
klienta_nr = :klienta_nr ?
& reg_nr = :reg_nr ?
& lower(nosaukums) ~ like_template(:nosaukums) ?
& fiz_adrese = :fiz_adrese ?
& jur_adrese = :jur_adrese ?
& lig.rekina_adrese = :rekina_adrese ?
& lower(tel) ~ like_template(:tel)
& epasts = :epasts ?
]
{
kli.id
, kli.ir_jurid
, kli.klienta_nr
, kli.ir_aktivs
, kli.fiz_adrese
, kli.jur_veids
, kli.reg_nr
, kli.nosaukums
, kli.jur_adrese
, kli.epasts
, kli.url
, kli.tel
, kli.fakss
, kli.piezimes
}

re. column conversion to lowercase

My suggestion is not to convert all selected columns to lowercase, but only to make the default case lowercase when all columns from table are selected.

I.e. query with just a table name should produce columns in lowercase.
But otherwise columns, esp. with case-sensitive names should not be converted.

I.e. in the following query I have case-sensitive column name nodeId, because is it mapped directly to Javascript object, which uses case-sensitive naming of course:

ipasums [id = :nodeId]
{
adrese_teksts text,
|pum [pum.ipa_id = :nodeId] {
id nodeId,
("PUM " || pum_nr) text,
|poz [pum_id = :1(1)] {
id nodeId,
("POZ " || poz_nr) text,
|psnorma [poz_id = :1(1)] {
id nodeId,
("PS/Norma " || ps_numurs) text,
"true" leaf
} children
} children
} children
}

insert multiple values in one statement does not work in PostgreSQL

Query:

DEPT {DEPTNO, DNAME, LOC} + [10, "ACCOUNTING", "NEW YORK"], [20, "RESEARCH", "DALLAS"]

compiles to:

insert into DEPT (DEPTNO, DNAME, LOC) values ((10, 'ACCOUNTING', 'NEW YORK'),(20, 'RESEARCH', 'DALLAS'))

but double brackets does not work in PostgreSQL, should be:

insert into DEPT (DEPTNO, DNAME, LOC) values (10, 'ACCOUNTING', 'NEW YORK'),(20, 'RESEARCH', 'DALLAS')

compact syntax for insert

+some_table {...} some_other_table { col1 c1_alias, col2, 2*2 col_x }

where {...} (or similar) could be a shortcut for "all column names as returned by the query to the right", i.e.

insert into some_table(c1_alias, col2, col_x)
  select col1 c1_alias, col2, 2*2 col_x
   from some_other_table

Or support some other syntax where insertable column list is implied or paired with value expression like in sql update.

escape to sql

'2000.01.01'::`timestamp without time zone`

21 * `interval '1 day'`

`double precision '3.5' * interval '1 hour'`

sql_with_params for seq broken

val ids = List(1, 2, 3)
tresql"per_karte k1 [id in $ids] {k1.kartes_nr}".list[String]
select k1.kartes_nr from per_karte k1 where id in (?,?,List(1, 2, 3))

Expected:

select k1.kartes_nr from per_karte k1 where id in (1, 2, 3)

Perhaps start/end of seq placeholders can be marked with some special comment?
Like id in (*_0[*/?,?,?/*]_0*/) instead of id in (?,?,?/*_0*/)

Test

This is test issue.

AJAX loader

cross join for postgres broken

Cross join fails on postgres if not in tail position.

per_karte k1 / per_persona [] per_karte k2;
per_persona / per_karte k3 {k1.kartes_nr}@(10)
select k1.kartes_nr from per_karte k1
join per_persona on k1.per_persona_id = per_persona.id ,
per_karte k2
left join per_karte k3 on per_persona.latest_karte_id = k3.id limit 10
ERROR: invalid reference to FROM-clause entry for table "per_persona"

This produces valid sql:

per_karte k1 / per_persona [true] per_karte k2;
per_persona / per_karte k3 {k1.kartes_nr}@(10)
select k1.kartes_nr from per_karte k1
join per_persona on k1.per_persona_id = per_persona.id
join per_karte k2 on true
left join per_karte k3 on per_persona.latest_karte_id = k3.id limit 10

tresql sql function cast to query

Enable sql function result interpretation as query. Maybe cast to special table type with columns or somehow use CTE syntax. Example of unsupported sql:

select sum(n::int) from unnest(string_to_array('1, 2, 3', ', ')) n

java example

If anyone has developed a simple test example using java instead of scala that would be helpful?

In constraint violations exceptions there is no way of telling where exception occurred.

When inserts and updates are done to multiple tables and in different locations in data structure with one tresql ORT call and SqlException is thrown there is no way of telling what operation caused it.

Exception that is thrown must not only contain constraint violation message from database but contain path in savable structure and table where exception is thrown, optionally sql statement with params.

Example:
Structure: parent -> Map(child-> Map(grand_child -> Map(age -> 2, name - "Bob" )))
Exception:

message: surname must not be null
path: [parent, child, grand_child]
table: grand_child
sql: insert into grand_child(name, age) values (? , ?)
params: Map(age -> 2, name - "Bob" )

In given example currently user gets message only and there is no way of telling if surname is required for parent, child or grand_child.

Tresql result sorting order differs from Oracle with Latvian letters

I have set Oracle NLS_LANG=LATVIAN. Oracle sql results are sorted correctly using Latvian collation sequence, such as:

SQL> select nosaukums from klasifikators where veids = 'DISPECERA_RAJONS' order by nosaukums asc;

NOSAUKUMS

Āboliņa Baiba
Gunta Muke
Kalpisa Inita
Kazoka Inese
K?imovica Dzintra
Lindersone Mirdza
Lisjonoka Elita
Vidzemes priekspils?ta
Ziemele Velta
Zvaigzne Gita

10 rows selected.

However, same select executed by Tresql puts Āboliņa at the end of the list:

klasifikators [ veids = 'DISPECERA_RAJONS'] { nosaukums }#(nosaukums)

C:\proj\rugis_git\klints>scala -Djdbc.drivers=oracle.jdbc.OracleDriver -Duniso.query.db=jdbc:oracle:thin:rugis/[email protected]:1521:orcl -Duniso.query.user=rugis -Duniso.query.password=gis123 -classpath lib/ojdbc14.jar;lib/tresql_2.9.1-1.2.1.jar test.scala
----------start----------
oracle.jdbc.driver.T4CConnection@466e06d7
select nosaukums from klasifikators where veids = 'DISPECERA_RAJONS' order by nosaukums asc
Bind vars:
nosaukums:Gunta Muke
nosaukums:Kalpiša Inita
nosaukums:Kažoka Inese
nosaukums:Kļimoviča Dzintra
nosaukums:Lindersone Mirdza
nosaukums:Lisjonoka Elita
nosaukums:Vidzemes priekšpilsēta
nosaukums:Ziemele Velta
nosaukums:Zvaigzne Gita
nosaukums:Āboliņa Baiba
----------end----------

Test case:

Class.forName("oracle.jdbc.OracleDriver")
import org.tresql._
import org.tresql.result._
import scala.collection.mutable.HashMap

val conn = java.sql.DriverManager.getConnection(System.getProperty("uniso.query.db"),System.getProperty("uniso.query.user"), System.getProperty("uniso.query.password"))
val md = metadata.JDBCMetaData("orcl", "rugis")

Env update conn
Env update md

val LOG_LEVEL = 1;

if (LOG_LEVEL > 0)
Env update ((msg, level) => println (msg))

println("----------start----------")
log(Env.conn)

val rowSelect = "klasifikators [ veids = 'DISPECERA_RAJONS'] { nosaukums }#(nosaukums)"

Query.foreach(rowSelect)(row => {
log("nosaukums:" + row(0))
})

println("----------end----------")

def log(s:Any) = {
if (LOG_LEVEL > 0) println(s)
}

def inform(s:Any) = println(s)
def error(s:Any) = println("ERROR: " + s)

Problems with arithmetic operation

  • Query.build("data(#){ {1 a } },atad(#){ {2 b }} {a * b / 6 c}")
    14.04.2021 11:32:15.195 DEBUG org.wabase.tresql.tresql - data(#){ {1 a } },atad(#){ {2 b }} {a * b / 6 c}
    java.util.NoSuchElementException: key not found: b

  • Query.build("data(#){ {1 a } },atad(#){ {2 b }} { (1 / 6) * a * b c}")
    14.04.2021 11:37:15.372 DEBUG org.wabase.tresql.tresql - data(#){ {1 a } },atad(#){ {2 b }} { (1 / 6) * a * b c}
    res3: org.tresql.Expr = with data as (select 1 as a), atad as (select 2 as b) select (1 / 6) * a * b as c

Inconsistent query syntax

a)
Query.build("data(# ){{1 a}}, atad(# ){{2 b}} data d [] atad ? a { d.a, b.a }").sql
res3: String = with data as (select 1 as a), atad as (select 2 as b) select d.a, b.a from data d join atad a on true

b)
Query.build("data(# ){{1 a}}, atad(# ){{2 b}} data d [ 1 = 1 ] atad ? a { d.a, b.a }").sql
res4: String = with data as (select 1 as a), atad as (select 2 as b) select d.a, b.a from data d left join atad a on 1 = 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.