Giter VIP home page Giter VIP logo

connectors_to_databases's Introduction

About me

Korsakov Ivan

Data engineer

I love exploring data ๐Ÿ—ƒ๏ธ.

connectors_to_databases's People

Contributors

k0rsakov avatar rajivsunar07 avatar

Watchers

 avatar

connectors_to_databases's Issues

Implement a new insert method with types

The method should allow inserting data "as is" from the dataframe via INSERT VALUES, but it should also be possible to add types for columns, for example, uuid, timestamptz, etc

Refuse packages clickhouse-sqlalchemy, psycopg2

  • Reject the psycopg2 package, as it is no longer used in modules, but pulled as a dependency
  • Reject the clickhouse-sqlalchemy package, since it works on the sqlalchemy logic and needs to be rewritten to use clickhouse_driver

add method with describe database (PostgreSQL)

add method with SQL:

SELECT
	all_columns.table_schema,
	schema_info.schema_description,
	all_columns.table_name,
	table_info.table_description AS table_description,
	all_columns.column_name,
	all_columns.data_type,
	columns_info.description AS column_description
FROM
	information_schema.columns AS all_columns
LEFT JOIN (
	SELECT
		*
	FROM
		pg_catalog.pg_statio_all_tables AS st
	LEFT JOIN pg_catalog.pg_description pgd 
		ON pgd.objoid = st.relid
	LEFT JOIN information_schema.columns AS c
		ON pgd.objsubid = c.ordinal_position
		AND c.table_schema = st.schemaname
		AND c.table_name = st.relname 
	) AS columns_info
		ON all_columns.table_schema = columns_info.schemaname
		AND all_columns.table_name = columns_info.relname
		AND columns_info.column_name = all_columns.column_name
LEFT JOIN (
	SELECT
		*,
		pg_catalog.obj_description(pgc.oid, 'pg_class') AS table_description
	FROM
		information_schema.tables AS t
	INNER JOIN pg_catalog.pg_class AS pgc 
		ON t.table_name = pgc.relname
	WHERE
		t.table_type = 'BASE TABLE'
		AND pg_catalog.obj_description(pgc.oid, 'pg_class') IS NOT NULL
	) AS table_info
		ON	
			table_info.table_name = all_columns.table_name
			AND table_info.table_schema = all_columns.table_schema
INNER JOIN (
	SELECT
		nspname,
		obj_description(oid) AS schema_description
	FROM
		pg_catalog.pg_namespace 
	) AS schema_info 
		ON schema_info.nspname = all_columns.table_schema
WHERE
	all_columns.table_schema != 'pg_catalog';

Add a new method with a generator for PostgreSQL

It is necessary to add a new method that will allow you to create a generator for an SQL query to the database.

The generator will allow you to process each line in the literature and additionally modify it.

Most likely, you need to implement it through the use of the cursor, but I'm not exactly sure about the implementation plan.

This is debatable.

add method with describe database (ClickHouse)

add method with SQL:

SELECT
	tables.database AS table_schema,
	tables.name AS table_name,
	columns.name AS column_name,
	TYPE AS data_type,
	columns.comment AS column_description,
	tables.comment AS table_description
FROM
	system.tables AS TABLES
LEFT JOIN system.columns 
	ON system.columns.table = system.tables.name

Add new test cases

New tests are needed to better test classes against different edge cases.

We need checks for the insertion of specific types: lists, uuids, json, jsonb, etc.

Also time insertion check

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.