Comments (2)
lbradstreet
commented on June 14, 2024
I think this is a good idea, and I’ve considered it in the past. Ideally we could have a way to review a diff for the log files as part of the release script to prevent breaking log changes being included in minor releases.
… On 27 Sep 2018, at 11:33, Ryan R Sundberg ***@***.***> wrote:
Upgrading Onyx versions between patches shouldn't require the user to use a new tenancy id. If the cluster-log format changes in an incompatible way, that should require a minor or major version number change. Otherwise, it can be very troublesome migrate job checkpoints, state, etc. to a new tenancy-id.
clojure.lang.ExceptionInfo: Incompatible versions of the Onyx cluster coordination log.
A new, distinct, :onyx/tenancy-id should be supplied when upgrading or downgrading Onyx.
{:cluster-version "0.13.3-alpha4", :peer-version "0.13.5"}
I propose that we go ahead and modify the check to ignore differences in major.minor.PATCH versions.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
from onyx.
sundbry
commented on June 14, 2024
What do you think about decoupling the internal data version from the library API version?
In other words, if we ever want to break the tenancy log data structure in an incompatible way, we should increment the internal data structure version. However, this should probably be decoupled from the actual project version, since for example, an upgrade to 0.14.0 or even 1.0.0 might be perfectly compatible with the existing data structures even if there are incompatible API changes or new features.
We might be able to accomplish this in the present form, without too much code, by freezing the version in onyx.peer.log-version and only changing that when the internal data structures change moving forward.
As you mentioned about checking for breaking changes, a good point to start may be to identify which of the schema definitions are used in the persisted data structures. We might be able to programmatically detect then when those schema definitions change. Something like
(def peer-log-schemas {....})
(diff peer-log-schemas previous-peer-log-schemas)
Ultimately though, it will still just be a matter of good design discipline and code review to make sure the internal data versions stay compatible.
from onyx.
Related Issues (20)
- output task with a window does not record the window checkpoint size correctly
- Is Dire still used in Onyx platform? HOT 7
- Upgrade Apache Curator framework to 4.0.x - SSL Support HOT 6
- Flow conditions validation shows bad error message when tasks are not connected. HOT 3
- Peer group manager where communicator fails to start is recoverable HOT 2
- Validation error for min-max-n-peers for flux policy is not printed correctly.
- Input plugin's poll! continues to be invoked after completed? HOT 2
- Move task-lifecycle backoff-until-task-start! into state machine.
- http://www.onyxplatform.org/ links to https://github.com/onyx-platform/onyx/releases/tag/0.13.x which does not exist
- Resume point AssertionError: Assert failed: (= slot-migration :direct) for {:mode :initialize}
- Output from job-snapshot-coordinates does not match input schema for build-resume-point
- Clojure 1.10.0-beta4 isn't happy with Onyx HOT 1
- Co-located task scheduler does not respect capacity contrains
- Output plugin :after-task-stop lifecycle fn doesn't receive :onyx.core/scheduler-event HOT 1
- Onyx hangs when provide ":onyx.messaging.aeron/media-driver-dir" setting in peer-config HOT 1
- Project maintanence going forward HOT 28
- Aeron Reliability HOT 4
- Feature request: Do not try to recover output checkpoints for plugins that don't use it HOT 3
- IndexOutOfBoundsException from aeron HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onyx.