Comments (4)
Thanks for the report, the error is unrelated to read_csv
however.
read_csv
store_rejects
handles errors that happen during reading of the csv file
This error originates from writing the data to the table, which is not related to csv read.
The constraint information could perhaps be pushed down into the csv reader so it's aware and can act accordingly, but that's definitely a feature request
from duckdb.
As mentioned this is not an issue with the CSV file or the rejects table - as such this is out-of-scope for the CSV reader. You could perhaps use e.g. INSERT OR IGNORE
or other SQL constructs to get the desired behavior instead.
from duckdb.
@Mytherin @Tishj Thank you for your answers. I understand the rationale.
How could I add such cases to reject_errors table? My goal is to use duckdb + sqlalchemy as a data quality assessment framework.
from duckdb.
The reject_errors
table is specifically meant for finding errors within CSV files. Duplicate keys is not an error within a CSV file but a manually inserted constraint. As such you would need to do this outside of the rejects table.
from duckdb.
Related Issues (20)
- DuckDB v1.0.0 released
- [C-API] Problem adding empty strings to vector with duckdb_vector_assign_string_element[_len]. HOT 3
- Segmentation fault with v1.0.0 on OSX (x86_64) HOT 2
- Performance difference with `read_csv` between a file and stdin HOT 1
- Query on Arrow views fail with specific order of filters HOT 2
- Review patches from duckdb-r HOT 1
- READ_CSV doesn't release memory in between files it imports HOT 3
- Double inserts HOT 1
- make parameter handling orthogonal, please HOT 2
- TPC-H extension generates identical values on WASM client
- read parquet file from s3 1.0.0 slower than 0.9.2 HOT 5
- terminate called after throwing an instance of 'duckdb::InvalidInputException' when read parquet from minio HOT 4
- Incorrect file type when writing files to google cloud bucket via httpfs
- BETWEEN incorrectly pushed into ASOF JOIN HOT 4
- cmake target exports use fixed paths (not relocatable)
- .lint fkey-indexes terminates cli HOT 1
- Regression in 1.0 for representation of SQLNULL in Python API HOT 1
- INTERNAL Error reading from CSV file HOT 3
- C++ API crashes when running in debug mode with release mode dll.
- Invalid Input Error: CSV Error on Line: 729 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from duckdb.