Comments (11)
@markalanevans correct. Our hangfire is only keeping 10K job records as history, but the tag records are only being cleaned up after 24hrs. This was destroying our system with Busy errors, as there were millions of these tag records :) To get us through till our next deployment window tonight, I'm manually deleting them every 30min or so. And tonight we will be disabling till we get a better solution.
from hangfire.tags.
Good question. It’s been a while since I’ve touched that code, but I think the Redis provider itself passes an expiry date when setting values. Redis itself will clean up keys at the specified expiry date.
I have to admit that I’m not entirely sure, but I’ll have to dove in a bit more to find out.
Are you currently experiencing issues with this behavior?
from hangfire.tags.
We haven't installed it yet. We just came across the project and started to evaluate it and while reading the docs that was highlighted as a potential issue.
If you happen to have a quick answer that's great, if not we will leave that as an open question on our end to test.
Thank you very much for making this.
from hangfire.tags.
We are using this project and wondering about the same issue. Is there any harm in us manually cleaning up the tags folder so it only keeps a few thousand? Is there any harm to hangfire proper, if we delete a tag that is tied to jobs that haven't been cleaned up by hangfire?
from hangfire.tags.
@matthewjcooper so is it not auto cleaning up tags that are tied to expired jobs?
from hangfire.tags.
So, I've checked a few things here. The "issue" is that Redis does not support an expiry value on a single value in a set of values.
What we could do instead, is to register a special set which contains keys and use the expire timestamp as scores. We would also need a special worker process that runs sometimes to clean up the expired keys.
An alternative is doing this manually, like I think @matthewjcooper does.
from hangfire.tags.
Is there any harm in us manually cleaning up the tags folder so it only keeps a few thousand? Is there any harm to hangfire proper, if we delete a tag that is tied to jobs that haven't been cleaned up by hangfire?
No and no.
from hangfire.tags.
I've just created a branch (redis-expire) which contains some cleanup code for tags in Redis.
This will:
a) Use a "global" SET where the KeyValue combination of the tag is a value, and the expiry date is the score (in seconds since Epoch). Tags that do not expire are not in this list.
b) Added an ExpireTagWatcher which runs hourly and at the start of the application, and will cleanup the global SET in step a, but also cleanup every tag which is expired according to the global SET in step a.
c) Set the current date in tags:expire_min_ticks when started for the first time. It will use this value as the minimum date when filtering tags that should be removed. This means that tags from BEFORE this change WILL NOT be cleaned up!
@matthewjcooper Is it possible that you test this branch? It's not on nuget, so you need to compile it yourself.
from hangfire.tags.
@erwin-faceit would it be possible to build a pre-release NuGet package for your change for testing? I'd be willing to test it, I only have VS2022 and can't build due to the 4.5 Net build dependency.
from hangfire.tags.
Here you go!
https://www.nuget.org/packages/Hangfire.Tags/1.8.4-beta.1
https://www.nuget.org/packages/Hangfire.Tags.Pro.Redis/1.9.1-beta.1
https://www.nuget.org/packages/Hangfire.Tags.Redis.StackExchange/1.9.1-beta.1
from hangfire.tags.
Our internal tests are working for these changes, so I'll merge the code to the master branch, and release accordingly.
from hangfire.tags.
Related Issues (20)
- Hangfire tags has huge memory food print in redis storage HOT 3
- How to retrieve jobs from hangfire with a given tag id? HOT 3
- Searching by tag throws exception when tag matches existing jobId HOT 1
- Hangfire 1.8 Support HOT 16
- Possibility to delete jobs based to tags HOT 1
- MongoDB support HOT 4
- IdExtensions.Clean makes it difficult to grok a large number of tags HOT 8
- Could not load file or assembly 'Hangfire.PostgreSql, Version=1.9.5.0 HOT 1
- Security Vulnerability in Hangfire.Tags.SqlServer CVE-2022-41064 HOT 2
- Question on Tag Amount Limits and Duplicate Tags HOT 2
- Tags expiration does not use Redis key prefix HOT 1
- The functions QueueCommand and AddCommand cannot be found error HOT 6
- Hangfire.Redis.StackExchange HOT 2
- RedisStorageOptions no longer defined in Hangfire.Redis.StackExchange HOT 5
- NullReferenceException with Hangfire.Tags when scheduling a background job HOT 3
- How to have default tags HOT 1
- Missing Logs on UI when TAG is enabled HOT 3
- Unable to Search with Tags after Upgrading to Hangfire Version 1.8.12 and FaceIT.Hangfire.Tags.SqlServer to version 1.9.0-beta.3 HOT 4
- Provide access to JobStorage.RemoveTags HOT 3
- Wrong dapper version in dependencies? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hangfire.tags.