I love exploring data ๐๏ธ.
k0rsakov / connectors_to_databases Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Need replace primary key with a list of keys
sql = f'''
INSERT INTO {target_table_schema_name}.{target_table_name}
(
{cls.list_columns_in_str_with_double_quotes(list_columns=list_columns)}
)
SELECT
{cls.list_columns_in_str_with_double_quotes(list_columns=list_columns)}
FROM
{source_table_schema_name}.{source_table_name}
ON CONFLICT ({pk}) {replace}
'''
Add abstractmethod in BaseOperator for method _authorization_database
Add the ability to include variables from the environment.
Add the ability to specify environment variables for class initialization. Instead of prescribing them manually or importing them from *.py
The method should allow inserting data "as is" from the dataframe via INSERT VALUES, but it should also be possible to add types for columns, for example, uuid, timestamptz, etc
Exception: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above
Fix: psycopg/psycopg2#1360 (comment)
Fix: pip install psycopg2==2.9.3
You need to remove ch_client
and networks
from docker-compose.yml
add method with SQL:
SELECT
all_columns.table_schema,
schema_info.schema_description,
all_columns.table_name,
table_info.table_description AS table_description,
all_columns.column_name,
all_columns.data_type,
columns_info.description AS column_description
FROM
information_schema.columns AS all_columns
LEFT JOIN (
SELECT
*
FROM
pg_catalog.pg_statio_all_tables AS st
LEFT JOIN pg_catalog.pg_description pgd
ON pgd.objoid = st.relid
LEFT JOIN information_schema.columns AS c
ON pgd.objsubid = c.ordinal_position
AND c.table_schema = st.schemaname
AND c.table_name = st.relname
) AS columns_info
ON all_columns.table_schema = columns_info.schemaname
AND all_columns.table_name = columns_info.relname
AND columns_info.column_name = all_columns.column_name
LEFT JOIN (
SELECT
*,
pg_catalog.obj_description(pgc.oid, 'pg_class') AS table_description
FROM
information_schema.tables AS t
INNER JOIN pg_catalog.pg_class AS pgc
ON t.table_name = pgc.relname
WHERE
t.table_type = 'BASE TABLE'
AND pg_catalog.obj_description(pgc.oid, 'pg_class') IS NOT NULL
) AS table_info
ON
table_info.table_name = all_columns.table_name
AND table_info.table_schema = all_columns.table_schema
INNER JOIN (
SELECT
nspname,
obj_description(oid) AS schema_description
FROM
pg_catalog.pg_namespace
) AS schema_info
ON schema_info.nspname = all_columns.table_schema
WHERE
all_columns.table_schema != 'pg_catalog';
It is necessary to add a new method that will allow you to create a generator for an SQL query to the database.
The generator will allow you to process each line in the literature and additionally modify it.
Most likely, you need to implement it through the use of the cursor, but I'm not exactly sure about the implementation plan.
This is debatable.
add method with SQL:
SELECT
tables.database AS table_schema,
tables.name AS table_name,
columns.name AS column_name,
TYPE AS data_type,
columns.comment AS column_description,
tables.comment AS table_description
FROM
system.tables AS TABLES
LEFT JOIN system.columns
ON system.columns.table = system.tables.name
Add the ability to choose what to install: PostgreSQL, ClickHouse, S3, etc
New tests are needed to better test classes against different edge cases.
We need checks for the insertion of specific types: lists, uuids, json, jsonb, etc.
Also time insertion check
Add processing for escaping all prohibited characters in a string for an SQL-query
Example:
pg.execute_to_df(
'''
SELECT
*
FROM
table
WHERE
name LIKE '%foo%'
'''
)
need bug fix
from psycopg2._psycopg import ( # noqa
ImportError: dlopen
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.