Giter VIP home page Giter VIP logo

elasticsql's Introduction

 _____ _         _     ____ _____ ___  ____  ____   ___   _
| ____| |       / \   / ___|_   _|_ _|/ ___|/ ___| / _ \ | |
|  _| | |      / _ \  \___ \ | |  | || |    \___ \| | | || |
| |___| |___  / ___ \  ___) || |  | || |___  ___) | |_| || |___
|_____|_____|/_/   \_\|____/ |_| |___|\____||____/ \__\_\|_____|

Overview

Build Status Go Documentation Coverage Status Go Report Card

This tool converts sql to elasticsearch dsl

Currently support:

  • sql and expression
  • sql or expression
  • equal(=) support
  • not equal(!=) support
  • gt(>) support
  • gte(>=) support
  • lt(<) support
  • lte(<=) support
  • sql in (eg. id in (1,2,3) ) expression
  • sql not in (eg. id not in (1,2,3) ) expression
  • paren bool support (eg. where (a=1 or b=1) and (c=1 or d=1))
  • sql like expression (currently use match phrase, perhaps will change to wildcard in the future)
  • sql order by support
  • sql limit support
  • sql not like expression
  • field missing check
  • support aggregation like count(*), count(field), min(field), max(field), avg(field)
  • support aggregation like stats(field), extended_stats(field), percentiles(field) which are not standard sql function
  • null check expression(is null/is not null)
  • join expression
  • having support

Usage

go get -u github.com/cch123/elasticsql

Demo :

package main

import (
    "fmt"

    "github.com/cch123/elasticsql"
)

var sql = `
select * from aaa
where a=1 and x = '三个男人'
and create_time between '2015-01-01T00:00:00+0800' and '2016-01-01T00:00:00+0800'
and process_id > 1 order by id desc limit 100,10
`

func main() {
    dsl, esType, _ := elasticsql.Convert(sql)
    fmt.Println(dsl)
    fmt.Println(esType)
}

will produce :

{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "a": {
                            "query": "1",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "match": {
                        "x": {
                            "query": "三个男人",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "range": {
                        "create_time": {
                            "from": "2015-01-01T00:00:00+0800",
                            "to": "2016-01-01T00:00:00+0800"
                        }
                    }
                },
                {
                    "range": {
                        "process_id": {
                            "gt": "1"
                        }
                    }
                }
            ]
        }
    },
    "from": 100,
    "size": 10,
    "sort": [
        {
            "id": "desc"
        }
    ]
}

aaa

If your sql contains some keywords, eg. order, timestamp, don't forget to escape these fields as follows:

select * from `order` where `timestamp` = 1 and `desc`.id > 0

Warning

To use this tool, you need to understand the term query and match phrase query of elasticsearch.

Setting a field to analyzed or not analyzed will get different results.

Details

For more details of convertion, please refer to the wiki

Other info

When writing this tool, I tried to avoid the deprecated dsl filters and aggregations, so it is compatible with most versions of the elasticsearch

If you have any advices or ideas, welcome to submit an issue or Pull Request!

License

MIT

elasticsql's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

elasticsql's Issues

能否支持escape?

原始sql

SELECT
    `familyCode`
FROM
    `myTable`
WHERE
    (`admin` LIKE '%v!_xxx%' ESCAPE '!')
ORDER BY
    `updateTime` DESC
LIMIT
    10

其中要对admin字段双侧LIKE字符串v_xxx
输出的结果:

{
    "query": {
        "bool": {
            "must": [
                {
                    "match_phrase": {
                        "admin": {
                            "query": "v!_xxx"
                        }
                    }
                }
            ]
        }
    },
    "from": 0,
    "size": 10,
    "sort": [
        {
            "updateTime": "desc"
        }
    ]
}

其中query.bool.must[0].match_phrase.admin.query字段的值中,! 没有去掉。

支持这种子查询吗?

select count(DISTINCT deviceId) from uv where deviceId in (select deviceId from uv where userId = 'unknown') and uv.userId != 'unknown';

Multiple Group by errors

sql like : select count(id) from A where A.id > 5 group by A.a, A.b;
because of
innerMap["terms"] = msi{ "field": colName.Name.String(), "size": 0, }
will gen the dsl like: "terms":{"field":"field2","size":0}
and then make a error: "reason":"[size] must be greater than 0
using es7.X

sql translate 咨询

请问一下 opendistro的_sql/_translate 您了解吗?我实验sql转换为dsl失败了

Large Query

Hello i need to know when we make it large query have alot of join inside SQL possible to make by your Go solution to Elastic Query match ?

Thanks,
George

简单查询语句报错

select * from aaa where remote_addr="111.206.36.1" limit 10;
转换为:
{"query" : {"bool" : {"must" : [{"match" : {"remote_addr" : {"query" : "111.206.36.1", "type" : "phrase"}}}]}},"from" : 0,"size" : 10}

用该dsl语句查询的时候:
{ "error": { "root_cause": [ { "type": "parsing_exception", "reason": "[match] query does not support [type]", "line": 1, "col": 84 } ], "type": "parsing_exception", "reason": "[match] query does not support [type]", "line": 1, "col": 84 }, "status": 400 }

必须要删除 "type" : "phrase" 才行

date_histogram转换DSL有问题?

使用例程转换下列语句为DSL的时候,来源
select * from table_name group by date_histogram(field='create_time', interval='1h')
报错
syntax error at position 81

interval要改成value么?

Is the example correct?

First of all thanks for the great work!

The functionality check-list mentions that like operator is implemented for phrase search.

However, in the example, although the strict equality = is used, the generated elasticsearch query is a phrase match. In that case, as far as I understood, shouldn't it generate a term search instead?

If the example is correct, what is the way to generate a term search query?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.