Slings from a data source to a data target.
See docs.slingdata.io for more details.
Sling is a passion project turned into a free CLI Product which offers an easy solution to create and maintain high volume data pipelines using the Extract & Load (EL) approach. It focuses on data movement between:
- Database to Database
- File System to Database
- Database to File System
sling.demo.2023.10.4k.mp4
Some key features:
- Single Binary deployment (built with Go). See installation page.
- Use Custom SQL as a stream:
--src-stream='SELECT * from my_table where col1 > 10' - Manage / View / Test / Discover your connections with the
sling connssub-command - Use Environment Variable as connections if you prefer (
export MY_PG='postgres//...)' - Provide YAML or JSON configurations (perfect for git version control).
- Powerful Replication logic, to replication many tables with a wildcard (
my_schema.*). - Reads your existing DBT connections
- Use your environment variable in your YAML / JSON config (
SELECT * from my_table where date = '{date}') - Convenient Transformations, such as the
flattenoption, which auto-creates columns from your nested fields. - Run Pre & Post SQL commands.
- many more!
Example Replication:
Available Connectors:
- Databases:
bigquerybigtableclickhouseduckdbmariadbmotherduckmysqloraclepostgresredshiftsnowflakesqlitesqlserverstarrocksprometheus - File Systems:
azureb2dospacesgslocalminior2s3sftpwasabi - File Formats:
csv,parquet,xlsx,json,avro,xml,sas7bday
Here are some additional links:
Ever wanted to quickly pipe in a CSV or JSON file into your database? Use sling to do so:
cat my_file.csv | sling run --tgt-conn MYDB --tgt-object my_schema.my_tableOr want to copy data between two databases? Do it with sling:
sling run --src-conn PG_DB --src-stream public.transactions \
--tgt-conn MYSQL_DB --tgt-object mysql.bank_transactions \
--mode full-refreshSling can also easily manage our local connections with the sling conns command:
$ sling conns set MY_PG url='postgresql://postgres:myPassword@pghost:5432/postgres'
$ sling conns list
+--------------------------+-----------------+-------------------+
| CONN NAME | CONN TYPE | SOURCE |
+--------------------------+-----------------+-------------------+
| AWS_S3 | FileSys - S3 | sling env yaml |
| FINANCE_BQ | DB - BigQuery | sling env yaml |
| DO_SPACES | FileSys - S3 | sling env yaml |
| LOCALHOST_DEV | DB - PostgreSQL | dbt profiles yaml |
| MSSQL | DB - SQLServer | sling env yaml |
| MYSQL | DB - MySQL | sling env yaml |
| ORACLE_DB | DB - Oracle | env variable |
| MY_PG | DB - PostgreSQL | sling env yaml |
+--------------------------+-----------------+-------------------+
$ sling conns discover LOCALHOST_DEV
9:05AM INF Found 344 streams:
- "public"."accounts"
- "public"."bills"
- "public"."connections"
...brew install slingdata-io/sling/sling
# You're good to go!
sling -hscoop bucket add sling https://github.com/slingdata-io/scoop-sling.git
scoop install sling
# You're good to go!
sling -hcurl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
&& tar xf sling_linux_amd64.tar.gz \
&& rm -f sling_linux_amd64.tar.gz \
&& chmod +x sling
# You're good to go!
sling -hgit clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli
bash scripts/build.sh
./sling --helpgit clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli
.\scripts\build.ps1
.\sling --helppip install sling
Then you should be able to run sling --help from command line.
sling run --src-conn POSTGRES_URL --src-stream myschema.mytable \
--tgt-conn SNOWFLAKE_URL --tgt-object yourschema.yourtable \
--mode full-refreshOr passing a yaml/json string or file
sling run -c '
source:
conn: $POSTGRES_URL
stream: myschema.mytable
target:
conn: $SNOWFLAKE_URL
object: yourschema.yourtable
mode: full-refresh
'
# OR
sling run -c /path/to/config.jsonpackage main
import (
"log"
"github.com/slingdata-io/sling-cli/core/sling"
)
func main() {
// cfgStr can be JSON or YAML
cfgStr := `
source:
conn: $POSTGRES_URL
stream: myschema.mytable
target:
conn: $SNOWFLAKE_URL
object: yourschema.yourtable
mode: full-refresh
`
cfg, err := sling.NewConfig(cfgStr)
if err != nil {
log.Fatal(err)
}
err = sling.Sling(cfg)
if err != nil {
log.Fatal(err)
}
}An example. Put this in https://jsonschema.net/
--src-conn/source.conn and --tgt-conn/target.conn can be a name or URL of a folder:
MY_PG(connection ref in db, profile or env)$MY_PG(connection ref in env)postgresql://user:[email protected]:5432/databases3://my_bucket/my_folder/file.csvgs://my_google_bucket/my_folder/file.jsonfile:///tmp/my_folder/file.csv(local storage)
--src-stream/source.stream can be an object name to stream from:
TABLE1SCHEMA1.TABLE2OBJECT_NAMEselect * from SCHEMA1.TABLE3/path/to/file.sql(if source conn is DB)
--tgt-object/target.object can be an object name to write to:
TABLE1SCHEMA1.TABLE2
{
"source": {
"conn": "MY_PG_URL",
"stream": "select * from my_table",
"options": {}
},
"target": {
"conn": "s3://my_bucket/my_folder/new_file.csv",
"options": {
"header": false
}
}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent