Comments (7)
failed, checking that imported data was completely removed: "pq: job 952221520862117889: node liveness error: restarting in background"
error executing 'SELECT * FROM t ORDER BY i': pq: relation "t" is offline: importing
Sounds like the logic here is not robust enough for retry-able failures:
cockroach/pkg/sql/importer/import_stmt_test.go
Line 5381 in 387e464
from cockroach.
It looks like a node died during this test, but I see no clues as to why.
from cockroach.
The test itself stops one node in the cluster. In this case it seems that the import was paused due to the node failure, so the table remained offline - this seems like expected behavior in some cases.
from cockroach.
The test itself stops one node in the cluster.
That's unrelated to the error, though, isn't it? Otherwise we'd hit this case every time.
from cockroach.
I think it is related. IIUC what happens is the following:
- we have a distributed import plan. When the import starts, the table is taken offline
- after some time we shut down one of the nodes
- import has an internal retry loop, so it can usually survive that single node failure and actually finish. After finishing, the table is taken online
- however, depending on at which point in time the node death is recognized, that retry loop might not retry the error and return it to the client. In this case we currently check that all imported data was rolled back (as of 14ef3c0).
I think what's missing is that in the last bullet point the table might be still offline, and this is what this test failure tells us. I'm not sure whether it is always offline or only sometimes offline in case the import fails.
from cockroach.
IIUC the table is still offline because the import is being retried: restarting in background
. So this is an internal retry error that we don't properly trap?
from cockroach.
Hm, this error message is used in two places where the job coordinator detects a context cancellation error. The error message suggests that the node that is coordinating the import job is overloaded. The import's retry loop exits if it detects the context cancellation, so it's expected that we won't succeed here.
Note that this is a race config on the engflow environment. We enabled race config somewhat recently in #117906 - perhaps we'll need to skip this test under race now that we've transitioned to running race CI on the engflow.
from cockroach.
Related Issues (20)
- pkg/sql/logictest/tests/local/local_test: TestLogic_tenant failed HOT 1
- plpgsql: add support for CALL statements
- ccl/backupccl: TestDataDriven_user_defined_functions failed HOT 2
- pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test: TestLogic_select_for_update failed HOT 3
- pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed HOT 3
- roachtest: admission-control/elastic-io failed HOT 1
- pkg/sql/logictest/tests/local/local_test: TestLogic_upsert_non_metamorphic failed HOT 1
- plpgsql: add support for complex variable references
- roachtest: allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed HOT 1
- Sentry: error.go:89: unexpected error from the vectorized engine: interface conversion: tree.Datum is *tree.DTimestamp, not *tree.DTimestampTZ (1) assertion failure Wraps: (2) attached stack trace -... HOT 1
- roachprod: MIG quota and resource errors HOT 2
- pkg/sql/logictest/tests/local-mixed-23.2/local-mixed-23_2_test: TestLogic_upsert_non_metamorphic failed HOT 1
- pkg/sql/logictest/tests/local-mixed-23.1/local-mixed-23_1_test: TestLogic_upsert_non_metamorphic failed HOT 1
- Sentry: pebble.go:1224: log.Fatal: local corruption detected: start timestamp mismatch (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricE...
- pcr: prevent resuming replication after flashback HOT 2
- roachtest: failover/chaos/read-only failed HOT 6
- sql: track UPDATE SET column dependencies in UDFs and SPs
- cdc: TestChangefeedContinuousTelemetryOnTermination is flaky
- roachtest: cluster_creation failed HOT 1
- kvclient: don't lose tracing on proxy requests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.