Comments (7)
What if we cache the schedules once every 30s and then eval each second? It's a compromise that I think most would accept.
from pg-boss.
The scheduling section in the docs explains that minute-level precision is the minimum cron precision. Does this answer your question?
from pg-boss.
I asked this question after reading the scheduling section of the docs.
Even though second-level precision is discouraged due to the amount of times the database needs to be hit, does it make sense to allow for an override of the checking precision to allow for second-level precision?
Would you be open to a pull request that makes it possible to override how often the schedule is checked and thus allow for second-level precision use cases?
from pg-boss.
As a preface, we've been looking at pg-boss
for queuing / scheduling and this was one of our concerns as well. When I saw this issue, the approach you mentioned is exactly what I was thinking. Query periodically and hold any items which would occur between now and the next query window in memory and execute them at their appropriate time.
I just wasn't intimately familiar with the codebase and if this was reasonable. In other words, I wasn't sure if the system needed to maintain a 'lock' on each job/entry and if there were limits on how long those locks could be held before being ack'ed. From a quick look at the docs, I suppose this is the job moving from created
→ active
and how long it's allowed to stay in that state before it transitions to expired
and it looks like the default is 15 minutes.
from pg-boss.
The current implementation uses an internal queue with denouncing hard-coded to 60s. For per-second cron, this would need to be changed to 1s
from pg-boss.
Isn't the internal queue responsible for calling the onCron()
function which is grabbing the scheduled items, filtering to schedules that should have run in the last 60 seconds, and sending them to their actual destination queue for work?
If I understand correctly, onCron()
currently filters the list of jobs using shouldSendIt()
to determine if the schedule is within the time range to send (last 60 seconds) and then 'sends' those tasks.
I was under the impression that you wanted to limit the heavier queries to the database to retrieve the full list of scheduled items, so you wanted to keep the at least 60 second debounce in place... as such, I was thinking something along the lines of changing the way the jobs were enqueued.
In other words, instead of onCron()
filtering for schedules which should have already passed and enqueuing those with send()
, it could filter to upcoming schedules and then it could use sendAfter()
so the enqueued jobs get scheduled with second level precision.
from pg-boss.
The debounce I'm mentioning is to ensure 2 pgboss instances running cron monitoring can't create more than 1 job within the cron expression.
from pg-boss.
Related Issues (20)
- Multiple start call HOT 1
- Maintenance Interval Issues / config options ignored HOT 1
- feature request: worker's job filter HOT 7
- bug: swallowed error when connection is closed HOT 1
- The library does not support PostgreSQL 11, just 13+ HOT 1
- Bug: debounce mechanism schedule multiple jobs HOT 1
- High Database CPU when processing backlog HOT 9
- feature-request: It should be possible to predict a job key or id. HOT 4
- Use pg-boss with postgres.js HOT 5
- Stopping pg-boss gracefully does not wait until stop timeouts HOT 9
- 'stopping' state missing in TypeScript Types HOT 1
- Easier observability for expired jobs HOT 1
- Is pg-boss actively maintained? HOT 2
- Limiting Concurrent Active Jobs Across Multiple Queues HOT 2
- Scheduling many individual jobs for the same queue HOT 3
- Document onComplete option HOT 1
- Job heartbeat support HOT 4
- Silently rejecting jobs when the db is not connected HOT 1
- New UUIDv7 increased performance and indexes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pg-boss.