felixmosh / bull-board Goto Github PK
View Code? Open in Web Editor NEWπ― Queue background jobs inspector
License: MIT License
π― Queue background jobs inspector
License: MIT License
Testing out 1.0.0-alpha.5...
Current package version
Steps Taken
1.npm i [email protected]
2. tsc -p project_src
However tsc fails and I get the error below
node_modules/bull-board/dist/index.d.ts:2:34 - error TS2307: Cannot find module 'bullmq'. 2 import { Queue as QueueMq } from 'bullmq';
node_modules/bull-board/dist/index.d.ts:3:30 - error TS2307: Cannot find module 'express-serve-static-core'.
3 declare const router: import("express-serve-static-core").Express;
Note I tested 1.0.0-alpha.4 to confirm and got the same error
Also manually installed bullmq package and error did go away however I received this new error
node_modules/bullmq/dist/interfaces/sandboxed-job.d.ts:3:57 - error TS2304: Cannot find name 'Omit'.
3 export interface SandboxedJob<T = any, R = any> extends Omit<JobJson, 'data' | 'opts' | 'progress' | 'log' | 'returnValue'> {
Due to CSP browser restrictions UI is not working any more in Chrome/FF. Please use window.location.pathname instead of passing basePath via window
object (this would allow us to skip inlined <script> tag).
I'm using bull-board in a cluster setup and whenever it pulls for new data, the page will become completely empty probably due to different data returned from the workers.
A migration from JS to TS would mean type exposure, which is nice.
does all the orders appears in one page, no pagaination ?
i could implement one if no any
Hi, I'm trying to test bull-board but using the example but I'm getting a screen with:
loading...
and not bull-board management screen:
Is the same as the example
const { createQueues } = require('bull-board')
const redisConfig = {
redis: {
port: 6379,
host: "127.0.0.1"
},
}
const queues = createQueues(redisConfig)
const helloQueue = queues.add('helloQueue') // adds a queue
const helloQueueWithOpts = queues.add('helloQueue', {
prefix: 'hello',
}) // adds a queue with QueueOptions https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queue
// defines how the queue works
helloQueue.process(async job => {
console.log(`Hello ${job.data.hello}`)
})
helloQueue.add({ hello: 'world' }) // adds a job to the queue
And later
const { UI } = require('bull-board')
app.use('/admin/bullDash', UI)
Someone knows what is happening?
When creating a job with the following properties
await queue.add('job-name', { foo: 'bar' }, {
repeat: {
every: 1000,
startDate: new Date(),
tz: "America/Los_Angles",
},
});
Then clicking on the delayed
tab in the UI I get the following error
RangeError: Invalid time value
at bundle.js:30
at delay (bundle.js:30)
at to (bundle.js:22)
at Gl (bundle.js:22)
at gu (bundle.js:22)
at mu (bundle.js:22)
at ru (bundle.js:22)
at bundle.js:22
at t.unstable_runWithPriority (bundle.js:30)
at La (bundle.js:22)
How can we integrate the UI to nestjs project where queues are alreay created and maanged by @nest/bull package?
Reported by sentry, I'm not sure on which specific conditions this happens:
TypeError: Cannot read property 'id' of null File "/usr/src/app/node_modules/bull-board/routes/getDataForQeues.js", line 24, col 13, in formatJob id: job.id, ?, in Array.map File "/usr/src/app/node_modules/bull-board/routes/getDataForQeues.js", line 67, col 20, in null.<anonymous> jobs: jobs.map(formatJob), ?, in runMicrotasks File "internal/process/task_queues.js", line 93, col 5, in processTicksAndRejections ?, in async Promise.all File "/usr/src/app/node_modules/bull-board/routes/getDataForQeues.js", line 54, col 18, in async getDataForQeues const counts = await Promise.all( File "/usr/src/app/node_modules/bull-board/routes/queues.js", line 5, col 5, in async handler await getDataForQeues({
We can start with a button to clean all the jobs in the delayed status, can probably be expanded to all the statuses
Hey folks! first of all, I hope to all of you having a good and stay healthy time on this covid-19 hard times.
I don't know if this could be considered as an issue but I was reading the source code and found that there a few extra definitions that are missing on the README
router.put('/queues/:queueName/retry', wrapAsync(retryAll))
router.put('/queues/:queueName/:id/retry', wrapAsync(retryJob))
router.put('/queues/:queueName/:id/promote', wrapAsync(promoteJob))
router.put('/queues/:queueName/clean/:queueStatus', wrapAsync(cleanAll))
so want to ask if there is an official doc or something to understand what are those routes and which role are playing on the board module
besides that, I didn't found in README regarding my server being hit at this route GET /queues/queues/?
and the server is replying with 304 π€
if anyone can help me will be very appreciated
This would be a breaking change, basically removing the createQueues
method and instead of keeping track of the queues internally, we would rather accept queues as a parameter (probably to UI
or a config method).
Major benefit would be that users have more control on how to use Bull on their own and maybe even supporting integration with other libs
We're using Bull Board on a project, and it's been really helpful to visualize what's happening in our queue. However, we're finding that the completed
and failed
counts don't seem to reflect the current status of the queue. Our code gets all kinds of completed
and failed
events, which we use to drive our app. But we see nothing in the UI:
I do find that failed
events in Bull don't always have the right err
, which I've filed upstream in OptimalBits/bull#1326. But this looks like a Bull Board issue vs Bull.
bull-board currently uses vorpal
as a dependency, which was last updated in August of 2018.
There are three high-risk security vulnerabilities in a dependency of vorpal relating to lodash which were fixed in 2017, see them using npm audit
Path for all, bull-repl > vorpal > inquirer > lodash
It seems to me the alternatives are to use instead either:
I am not familiar with Vorpal or Oclif which would we prefer? This security issue really must be addressed and since the only way to do it is to change dependencies, as the one it's in is dead, there needs to be some discussed about what CLI bull-board is going to in-future use.
The value returned by your process function will be stored in the jobs object and can be accessed later on, for example in a listener for the completed event
Is the object / value that is returned by the consumer available in the dashboard? I'm seeing the object that is passed to the consumer, and the options β but I don't actually see the value that is returned.
Hey, thanks for this great dashboard,
i am not sure, but after a job finishes , the progress bar doesnt go to 100% it gust go to finished tab,
but the percentege progress still 0,
also this is my code :-
myFirstQueue.add('normal', { name: 'Hello', });
myFirstQueue.process('normal', (job, done) => { for (let i=0;i<=1000;i++){ file.write('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.'); } done(); });
how could i make the process take time , so i can see the progess in the dashboard,
would appreciate any help,
and thanks for this great work
if i added more than one queue would the UI create another tabes for the new queue,
ar all queues insert in the same tabs ?
It would be nice to have detailed under the Delayed
jobs the information for when it's going to run again, if Bull can provide it
Also, very nice lib π
Board shows the data, but not the job name, can we get it to show this?
TypeError: queue.client.info is not a function
at getStats (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/getDataForQeues.js:12:22)
at getDataForQeues (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/getDataForQeues.js:71:23)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async handler (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/queues.js:5:5)
TypeError: queue.client.info is not a function
at getStats (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/getDataForQeues.js:12:22)
at getDataForQeues (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/getDataForQeues.js:71:23)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async handler (/Users/macbookpro/Documents/data/node_modules/bull-board/routes/queues.js:5:5)
//////////////////////////////
const { Queue, Worker, QueueEvents } = require('bullmq');
var cluster = require('cluster');
const { setQueues, UI } = require('bull-board')
const app = require('express')()
const port = 3010
var numWorkers = 8;
var queue = new Queue("Paint");
const queueEvents = new QueueEvents('Paint');
setQueues([queue])
if(cluster.isMaster){
for (var i = 0; i < numWorkers; i++) {
cluster.fork();
}
cluster.on('online', function(worker) {
// Lets create a few jobs for the queue workers
for(var i=0; i<500; i++){
queue.add('cars', { color: 'blue' });
};
});
cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died');
});
}else{
const worker = new Worker('Paint', async job => {
if (job.name === 'cars') {
await console.log(job.data.color);
}
});
queueEvents.on('waiting', ({ jobId }) => {
console.log(A job with ID ${jobId} is waiting
);
});
queueEvents.on('active', ({ jobId, prev }) => {
console.log(`Job ${jobId} is now active; previous status was ${prev}`);
});
queueEvents.on('completed', ({ jobId, returnvalue }) => {
console.log(`${jobId} has completed and returned ${returnvalue}`);
});
queueEvents.on('failed', ({ jobId, failedReason }) => {
console.log(`${jobId} has failed with reason ${failedReason}`);
});
}
app.use('/ui', UI)
app.listen(port, () => console.log(Example app listening at http://localhost:${port}
))
Is it possible to add the result (json) of a completed job into the dashboard to see what the output was?
A website will allow for a more structured way of showing documentation
We're currently declaring the types for redis-info
manually at src/declarations.d.ts
, but recently we've got access to more complete types through the @types/redis-info
package.
We should change to use the types in DefinetilyTyped and update some of the types we declare locally to be defined in terms of them.
I'm trying to run bullboard but he just returning me a blank page with HTML code and i dunno how to solved it, can you guys help me:
This is the code i'm trying to run:
// bullboard.js
import {UI, setQueues} from 'bull-board'
import Queue from '../services/queueService'
setQueues(Queue.queues.map((queue) => queue.bull))
export default UI
// setup-app.js
import cors from '../middlewares/cors'
import jsonParser from '../middlewares/json-parser'
import rawParser from '../middlewares/raw-parser'
import contentType from '../middlewares/content-type'
import bodyParser from '../middlewares/body-parser'
import logger from '../middlewares/logger'
import bullboard from '../middlewares/bullboard'
import voyager from '../middlewares/voyager'
export default (app) => {
app.disable('x-powered-by')
app.use(cors)
app.use(jsonParser)
app.use(rawParser)
app.use(contentType)
app.use(bodyParser())
app.use(logger)
app.use(bullboard)
app.use('/voyager', voyager)
}
// queueService.js
import Queue from 'bull'
import env from 'env-cat'
import * as jobs from '../jobs'
import * as logger from '../core/logger'
const queues = Object.values(jobs).map((job) => ({
bull: new Queue(job.key, env.get('REDIS_URL')),
name: job.key,
handle: job.handle,
}))
export default {
queues,
add(name, data) {
const queue = this.queues.find((queue) => queue.name === name)
return queue.bull.add(data)
},
process() {
return this.queues.forEach((queue) => {
queue.bull.process(queue.handle)
queue.bull.on('failed', (job, err) => {
logger.info('Job failed', queue.key)
logger.info(err)
})
})
},
}
// jobs.js
export {default as Mail} from './Mail'
export {default as Task} from './Task'
export {default as Pipeline} from './Pipeline'
export {default as SegmentUpdate} from './SegmentUpdate'
export {default as SegmentDelete} from './SegmentDelete'
Node version: 12.16.1
NPM Version: 6.13.7
I was trying to run bullboard with Apollo Server, but somehow is not showing anymore the dashboard, when i try to access the is returning this to me.
I was starting the bullboard with a middleware and adding the queue with this code, after i debug aparently is normally working.
And adding to my application with this code.
Node version: v10.15.2
Bullboard version: ^0.5.0
Bull version: ^3.12.1
I was really excited about this project when I saw it, but I had trouble getting it to work. We have Bull queues running in production in a Docker Swarm with dozens of Fastify worker containers and several different redis endpoints. I tried using fastify.use()
but it's not compatible I don't think.
But this kinda sucks anyways β we would have to embed the bull-board and all its dependencies (express, react, react-dom, ramda, etc) directly into our server images. I was hoping this would be more of a standalone server that you can point to a redis process and give it a prefix and it'll find your queues for you (via KEYS bull:*
every so often). That would mean just adding one more container instead of embedding an instance of it to every worker image.
I think you already have the basis of it here, an UI + Express server, what do you think of modifying it to work without any required integration into your node app that is running Bull?
are you guys planning on supporting BullMQ 4 ?
Hello
If you will try to setQueue(bullInstance) to the board and then bullInstance.close() => it will produce error "Error: Connection is closed.", please make it just removing it from dashboard
When i tried to use bull-board, nothing happens.
With no queues i have "no stats to display" page, but if I add some existing queue like
import { createQueues } from 'bull-board'
const queues = createQueues(myConfig)
export const myQueue = queues.add('MyQueue')
And I try to use it, there is empty white page on '/admin/queues' and I have following errors in browser's console
TypeError: Expected a finite number, got number: NaN
at e.exports (bundle.js:1)
at $e (bundle.js:30)
at Ai (bundle.js:22)
at Vl (bundle.js:22)
at Fl (bundle.js:22)
at Ll (bundle.js:22)
at Rl (bundle.js:22)
at bundle.js:22
at t.unstable_runWithPriority (bundle.js:30)
at ca (bundle.js:22)
updated: I used bull-board api.
Integration tests, a suggestion is to use Dockest to spin up a redis instance so it would be possible to add/remove jobs in queues and test the code
Hello!
Love the screenshot look and tried setting this up for a redis hosted on Heroku.
It's very hard because there are no error messages anywhere and the documentation seems incomplete.
How do I know if my credentials are correct?
How do I specify queues?
When it "fails" it just ends up pending forever and then fails with MaxRetries, is that intended?
After reading around this is what I ended up with
const { createQueues, UI } = require('bull-board');
const app = require('express')();
const redisConfig = {
redis: {
port: process.env.REDIS_PORT,
host: process.env.REDIS_HOST,
password: process.env.REDIS_PASSWORD,
},
}
const queues = createQueues(redisConfig);
queues.add('posts');
app.use('/admin/queues', UI)
app.listen( 3000, () => {
console.log( `Server up and running on port 3000` );
});
I'd gladly help with PRs but right now I don't even know where to start π
eslint was set up with the minimal requirements for prettier, but it still doesn't have any rules.
we could at least have eslint:recommended
in the extends
list and make it work with both React and Node env (routes/ and ui/ folders)
Pipeline doesn't run on forked PRs.
Explore Azure Pipelines.
Otherwise, a suggestion would be using Travis
import { router } from 'bull-board';
app.use('/ui', router)
trace:
at Function.render (/Users/dev/platform/node_modules/express/lib/application.js:580:17) at ServerResponse.render (/Users/dev/platform/node_modules/express/lib/response.js:1012:7) at exports.entryPoint (/Users/dev/platform/node_modules/bull-board/src/routes/index.ts:8:7) at Layer.handle [as handle_request] (/Users/dev/platform/node_modules/express/lib/router/layer.js:95:5) at next (/Users/dev/platform/node_modules/express/lib/router/route.js:137:13) at Route.dispatch (/Users/dev/platform/node_modules/express/lib/router/route.js:112:3) at Layer.handle [as handle_request] (/Users/dev/platform/node_modules/express/lib/router/layer.js:95:5) at /Users/dev/platform/node_modules/express/lib/router/index.js:281:22 at Function.process_params (/Users/dev/platform/node_modules/express/lib/router/index.js:335:12) at next (/Users/dev/platform/node_modules/express/lib/router/index.js:275:10)
There are alredy two retry buttons. It would be nice to have some buttons to clean up all failed jobs and single jobs.
Just trying your nice board! Well done π
Is it possible to see job logs logged with job.log()
?
It would be nice to know what was logged during jobs runs.
Cheers! π»
Currently the Bull-board is not providing the needed info.
For example when a job fails we retry it X amount of times depending on the endpoint and this returns βjob stalled more than allowable limitβ
when it fails. Before in Kueue this would give the data object which contained the reason for failing.
Currently it's only possible to see the complete message by calling βjobs/failed/retrieve/0/99999999β
endpoint.
For this I would like the DATA and OPS object to be added to the failed job or an other solution would be to add a link to retrieve the complete message.
Solution one extend the array of queue(ui/components/Queue.js 63) :
failed: ['id', 'failedReason','data', 'timestamps', 'progress', 'attempts', 'retry'],
I want to use it in an application built by koa2οΌbut I don't get it.
my code like:
const Koa = require('koa');
const app = new Koa();
const { UI } = require('bull-board');
app.use((ctx, next) => {
if(ctx.path.startWith('/monitor')) {
if (ctx.status === 404 || ctx.status === '404') {
delete ctx.res.statusCode
}
ctx.respond = false;
UI(ctx.req, ctx.res);
} else {
next();
}
});
This is from Can we mount express app into koa?
GET /queues
returns
{"status":500,"message":"Cannot use 'in' operator to search for 'redis_version' in undefined"}
when I return new Redis.Cluster(...)
in createClient
while creating Bull.Queue
Hello.
First of all, thank you for making a nice and clean interface for bull. I use it in my project that syncs data between databases. I do the work using batches, so I never know the max value of things that need processing in order to calculate the percentage. So if i do job.progress(item.count)
where item.count is for example 4235 items, the frontend interface will go crazy because the green progressbar will be style="width: 4235%"
.
What im proposing is a simple check, something along the lines of:
if (job.progress > 100) // Don't render class="progress-bar", style="width: 4235%" and the % sign
progress: ({ job }) => {
switch (type(job.progress)) {
case 'Object':
return (
<Highlight className="json">
{JSON.stringify(job.progress, null, 2)}
</Highlight>
)
case 'Number':
if (job.progress > 100) {
return (
<div className="progress-wrapper"> {job.progress}</div>
)
}
return (
<div className="progress-wrapper">
<div
className="progress-bar"
style={{
width: `${job.progress}%`,
}}
>
{job.progress}
%
</div>
</div>
)
default:
return '--'
}
},
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.