Comments (5)
Yes, removing the sort allows the connector to start syncing.
For reference, here is a comparison of the costs for misordering the keys:
EXPLAIN SELECT * FROM document_snippets ORDER BY id, source_document_id LIMIT 1000 OFFSET 2000;
Limit (cost=2440052.09..2440168.76 rows=1000 width=1405)
-> Gather Merge (cost=2439818.74..2679752.32 rows=2056430 width=1405)
Workers Planned: 2
-> Sort (cost=2438818.72..2441389.25 rows=1028215 width=1405)
Sort Key: id, source_document_id
-> Parallel Seq Scan on document_snippets (cost=0.00..448904.15 rows=1028215 width=1405)
EXPLAIN SELECT * FROM document_snippets ORDER BY source_document_id,id LIMIT 1000 OFFSET 2000;
Limit (cost=5294.09..7940.85 rows=1000 width=1405)
-> Index Scan using document_snippets_pkey on document_snippets (cost=0.56..6531468.94 rows=2467717 width=1405)
from connectors.
I believe this is actually a symptom of incorrectly ordering the primary keys.
This table has a compound primary key:
CREATE TABLE document_snippets(
source_document_id text,
id text,
PRIMARY KEY (source_document_id, id)
)
But in the error, the components are reversed ORDER BY id, source_document_id
which will lead to a very expensive query.
from connectors.
The primary keys are sorted here, making the order inefficient.
from connectors.
Hi @Dig-Doug, have you tested if removing sorting makes it work for you?
from connectors.
If you want you can make a contribution to us with the change you've made. If not, we'll address it but probably a bit later.
Would you like to contribute?
from connectors.
Related Issues (20)
- Make connectors work with python 3.10, 3.11 and 3.12
- Incremental Sync should consider updates that doesn't change timestamp
- Connectors services don't shut down immediately if new task added to pool
- In the github connector, for Access Control Sync, allow_access_control field is missing for the document indexed. HOT 4
- Retirement of Office 365 connectors within Microsoft Teams HOT 1
- Postgres advanced sync rules missing in Native Connectors HOT 2
- [MSSQL] Impossible to sync tables that have columns of MSSQL-specific types or even standard datetime
- '_asyncio.Task' object has no attribute 'name' HOT 2
- [MySQL] Connector dies if connection to MySQL times out due to slow ingestion
- Throttle based on cluster capacity HOT 2
- [Dropbox] `file_path` property does not take directory hierarchy into account
- [Confluence] body content for some records includes HTML tags
- Add more INFO logs and metrics when running connectors with data extraction service
- Connector scheduling doesn't handle restarts after scheduled sync time
- MS SQL Connector: Client does not have encryption enabled but it is required by server HOT 6
- Dependency Dashboard
- Elasticsearch OneDrive native connector is missing to index the field mime-type. HOT 2
- Network Drive connector DLS mapping CSV gives error - IndexError: list index out of range HOT 4
- [ServiceNow] Connector to support custom tables HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connectors.