Giter VIP home page Giter VIP logo

hangfire.tags's Introduction

Hangfire.Tags

Build status NuGet MIT License Donate

Inspired by the lack of searching and grouping, Hangfire.Tags provides a way to search and group different jobs.

sidemenu dashboard

Contributers

  • Yong Liu: Special thanks for all the pull requests you've created. Thank you so much!
  • Adam Taylor: MySql support

Features

  • 100% Safe: no Hangfire-managed data is ever updated, hence there's no risk to corrupt it.
  • Attributes: supports [Tag("{0}")] syntax for creating a default set of tags when creating a job.
  • Extensions: has extension methods on PerformContext, but also on string (for instance for adding tags to a jobid).
  • Clean up: uses Hangfire sets, which are cleaned when jobs are removed.
  • Filtering: allows filtering of tags based on tags and states, this makes it easy to requeue failed jobs with a certain tag.
  • Searching: allows you to search for tags
  • Storages: has an storage for SQL Server, MySql, PostgreSql and initial Redis support
  • Dark and light mode: supports the new dark and light mode support of Hangfire

Setup

In .NET Core's Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    services.AddHangfire(config =>
    {
        config.UseSqlServerStorage("connectionSting");
        // config.UseTagsWithPostgreSql();
        // config.UseTagsWithMySql();
        // config.UseTagsWithRedis();
        config.UseTagsWithSql();
    });
}

Otherwise,

GlobalConfiguration.Configuration
    .UseSqlServerStorage("connectionSting")
    //.UseTagsWithPostgreSql()
    //.UseTagsWithMySql()
    //.UseTagsWithRedis();
    .UseTagsWithSql();

NOTE: If you have Dashboard and Server running separately, you'll need to call UseTags(), UseTagsWithSql(), UseTagsWithPostgreSql(), UseTagsWithMySql() or UseTagsWithRedis() on both.

Sql Options

If you have a custom HangFire schema in your database, you'll need to pass your sql options to your storage method. For example:

        var tagsOptions = new TagsOptions() { TagsListStyle = TagsListStyle.Dropdown };
        var hangfireSqlOptions = new SqlServerStorageOptions
        {
            SchemaName = "MyCustomHangFireSchema",
        };
        services.AddHangfire(hangfireConfig => hangfireConfig
            .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
            .UseColouredConsoleLogProvider()
            .UseSimpleAssemblyNameTypeSerializer()
            .UseRecommendedSerializerSettings()
            .UseSqlServerStorage("dbConnection", hangfireSqlOptions)
            .UseTagsWithSql(tagsOptions, hangfireSqlOptions)
        );

Additional options

As usual, you may provide additional options for UseTags() method.

Here's what you can configure:

  • TagColor/DarkTagColor - default background color for the tags
  • TextColor/TextColor - default text color of the tags
  • Tags interface - you can specify an autocomplete tags search (Yong Liu)
  • MaxLength - the maximum length of the tags, automatically set to 100 for SQL Server

NOTE: After you initially add Hangfire.Tags (or change the options above) you may need to clear browser cache, as generated CSS/JS can be cached by browser.

Providers

In order to properly cleanup tags for expired jobs, an extension is required for the default storage providers. Right now, there are three providers: for SQL server, for PostgreSQL and for MySql.

Tags

Hangfire.Tags provides extension methods on PerformContext object, hence you'll need to add it as a job argument.

NOTE: Like IJobCancellationToken, PerformContext is a special argument type which Hangfire will substitute automatically. You should pass null when enqueuing a job.

Now you can add a tag:

public void TaskMethod(PerformContext context)
{
    context.AddTag("Hello, world!");
}

which results in the tag hello-world.

You can also add tags using attributes, either on the class, or on the method (or both!)

[Tag("TaskMethod")]
public void TaskMethod(PerformContext context)
{
    ....
}

Search tags

In the Dashboard, when clicking on Jobs, you'll see a new menu item, called Tags. By default this page will show you all defined tags in the system. Clicking on a tag will show a list of all jobs with that tag attached.

The default view for showing the tags is a so called tagcloud. If you prefer an autocomplete dropdown list, you can specify that using the options:

var options = new TagsOptions()
{
   TagsListStyle = TagsListStyle.Dropdown
};
config.UseTagsWithSql(options);

The result will look like this: tagsearch

License

Copyright (c) 2018 2Face-IT B.V.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

hangfire.tags's People

Contributors

1bberto avatar dependabot[bot] avatar erwin-faceit avatar erwinbovendeur avatar granicus422 avatar harleedavidson avatar lucaskepa avatar pdevito3 avatar r-win avatar schmittflorent avatar vitaliibushylo avatar yongliu-mdsol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

hangfire.tags's Issues

System.IO.FileNotFoundException for RecurringJob

If I create a repeating task that calls a method with a tag, this error crashes and the task is not created at all. Although the table [Set] tag values are written.

System.IO.FileNotFoundException
Could not load file or assembly 'System.Data.SqlClient, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

System.IO.FileNotFoundException: Could not load file or assembly 'System.Data.SqlClient, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
at Hangfire.Tags.SqlServer.SqlTagsTransaction.PersistSetValue(String key, String value)
at Hangfire.Tags.Storage.TagExpirationTransaction.Persist(String jobid)
at Hangfire.Tags.States.TagsCleanupStateFilter.OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
at Hangfire.States.StateMachine.InvokeOnStateApplied(Tuple`2 x)

Latest Build failed

@erwin-faceit Looks like the latest build failed. Something wrong with the API key. Also looks like it needs the version # for the main package bumped to 1.6.4. Thanks!

Tags expiration does not use Redis key prefix

Hi,

We configured Hangfire to use Redis for tags storage, and we have an issue where tags are not being expired.

This is how we configure Hangfire:

builder.Services.AddHangfire((sp,opts) =>
{
    var rs =  sp.GetService<IRedisService>();

    opts.UseRedisStorage(rs.data_apps_redis.Configuration, new Hangfire.Pro.Redis.RedisStorageOptions()
    {
        Prefix = $"hf:",
        MaxStateHistoryLength = 100,
        MaxSucceededListLength = EnvironmentStuff.is_development()?1000: 100000
    });
    opts.SetDataCompatibilityLevel(CompatibilityLevel.Version_170);
    opts.UseSimpleAssemblyNameTypeSerializer();
    opts.UseTagsWithRedis(new TagsOptions()
    {
        TagsListStyle = TagsListStyle.Dropdown
    });
    opts.UseConsole(new ConsoleOptions()
    {
        FollowJobRetentionPolicy = true,
        PollInterval = 5000
    });
});

The Redis key hf:tags:expiry contains values such as tags:locktaken|d1d819eb-18f8-48cc-828b-decdc341032e. When I look at the contents of the key hf:tags:locktaken, it contains values that should have been expired because the timestamp stored in hf:tags:expiry is several hours ago, and the expired tags watcher is supposed to run hourly.

I suspect that tags are not expired because is because keys in hf:tags:expiry does not contain the hf: prefix. Am I missing a configuration step?

We use the following versions:

  • Hangfire: 1.7.34
  • Hangfire.Pro: 2.3.2
  • Hangfire.Pro.Redis: 2.8.20
  • FaceIT.Hangfire.Tags.Pro.Redis: 1.9.1

Thanks for your help.

Suggestion: Include Enqueue At for Scheduled Jobs

When you view Scheduled jobs from the standard Hangfire UI, you get an extra column "Enqueue At" which shows when the job will run. It would be awesome if the tag search feature could also show that Enqueue At when showing Scheduled jobs.

Not all tags are searchable

Hi!
When I try to add a tag using a date time format but in string like 25/05/2020 my tag is added in this format 25052020, but this is not a problem. The main problem is that when i try to search using dropdown or the click it doesnt appear but it's been added since the tag number increases.

The following code is this one

internal virtual void SetBackgroundTaskContextTags(string[] tags)
        {
            var currentDateTag = DateFunctions.ConvertDateToStringFormat(DateTime.Now, "d");
            var tagsWithCurrentDate = tags.Append(currentDateTag);
            BackgroundContext.AddTags(tagsWithCurrentDate);
        }

Any thoughts why this happen?

Null Reference Exception after increasing aws redis instance size.

This exception started happening after upsizing our aws redis instance from a t3.micro to t3.small. Due this error, Hangfire Background jobs couldn't be created.

Work around: Deploy with tags plugin disabled. Reenable tags and redeploy.

[22:55:59.686 ERR][***] Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware: An unhandled exception has occurred while executing the request. {EventId={Id=1, Name="UnhandledException"}, RequestPath="/api/Job/RecreateDefault", ConnectionId="0HMHFLLIL4HD6", ExceptionDetail={["HResult"]=-2146233088, ["Message"]="Background job creation failed. See inner exception for details.", ["Source"]="Hangfire.Core", ["InnerException"]={["Type"]="System.NullReferenceException", ["Data"]={["OriginalStackTrace"]=" at Hangfire.Tags.States.CreateJobFilter.OnCreated(CreatedContext filterContext)\n at Hangfire.Client.BackgroundJobFactory.InvokeOnCreated(Tuple2 x)"}, ["HResult"]=-2147467261, ["Message"]="Object reference not set to an instance of an object.", ["Source"]="Hangfire.Tags"}, ["Type"]="Hangfire.BackgroundJobClientException"}}`

I'm not a 100% sure, but it might be due to this line:

.Union(filterContext.BackgroundJob.Job.Type?.GetCustomAttributes<TagAttribute>() ?? Enumerable.Empty<TagAttribute>())

Exceprion with Npgsql.NpgsqlConnection

image

Object of type 'System.Action`1[Npgsql.NpgsqlConnection]' cannot be converted to type 'System.Action`1[System.Data.IDbConnection]'.

System.ArgumentException: Object of type 'System.Action`1[Npgsql.NpgsqlConnection]' cannot be converted to type 'System.Action`1[System.Data.IDbConnection]'.
   at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
   at System.Reflection.MethodBase.CheckArguments(StackAllocedArguments& stackArgs, ReadOnlySpan`1 parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Hangfire.Tags.PostgreSql.PostgreSqlTagsTransaction.QueueCommand(Action`1 action)
   at Hangfire.Tags.PostgreSql.PostgreSqlTagsTransaction.ExpireSetValue(String key, String value, TimeSpan expireIn)
   at Hangfire.Tags.Storage.TagExpirationTransaction.Expire(String jobid, TimeSpan expireIn)
   at Hangfire.Tags.States.TagsCleanupStateFilter.OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
   at Hangfire.States.StateMachine.InvokeOnStateApplied(Tuple`2 x)

Have a exception on job finishing.

Hangfire 1.7.28
Hangfire.Tags.PostgreSql, Version=1.8.1.0
Hangfire.Tags, Version=1.7.1.0

Job fails Enqueue when passing Null for PerformContext

Background

We are seeing a null reference exception when calling Enqueue and passing Null for PerformContext. Our Enqueue call looks like this:

BackgroundJob.Enqueue(() => ProcessFile(file.Name, fileContents, Config.CustomConfigurationEntries, /*PerformContext*/null));
2019-02-07 16:19:06.8422|ERROR|Background job creation failed. See inner exception for details.
Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Hangfire.Tags.States.CreateJobFilter.OnCreated(CreatedContext filterContext)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   --- End of inner exception stack trace ---
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at FileProcessor.EnqueueFiles() 

Version Information

Hangfire: 1.7 Beta 2
Hangfire.Tags: 0.6.0

Tag Page with error 500

I couldn't see the tag page, since it throws a 500 error. I have all dependencies updated but i can't see no page.

The startup code is

services.AddHangfire(configuration =>
            {
                configuration.UseSerializerSettings(new JsonSerializerSettings()
                {
                    ContractResolver = new DefaultContractResolver(),
                    ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
                    TypeNameHandling = TypeNameHandling.Auto
                });
                configuration.UseFilter(new AutomaticRetryAttribute { Attempts = 0 });
                configuration.UsePostgreSqlStorage(WebServicesConfigurations.ConnectionString,
                    new PostgreSqlStorageOptions()
                    {
                        PrepareSchemaIfNecessary = true,
                        SchemaName = BackgroundJobConfigurations.DatabaseSchema
                    });
                configuration.UseConsole(new ConsoleOptions
                {
                    ExpireIn = TimeSpan.FromDays(2.0)
                });
                configuration.UseTagsWithPostgreSql(new TagsOptions()
                {
                    TagsListStyle = TagsListStyle.Dropdown,
                    Storage = new PostgreSqlTagsServiceStorage(new PostgreSqlStorageOptions()
                    {
                        PrepareSchemaIfNecessary = true,
                        SchemaName = BackgroundJobConfigurations.DatabaseTagsSchema
                    })
                });
                configuration.UseDarkDashboard();
            });

Attribute [Tag] behaves different than [DisplayName] in case of overrides

Hello,

I prepared minimal example of the issue here

When [Tag] attribute is placed on overriding method - it is ignored. However [DisplayName] follows overridden method attribute.
I think that Hangfire.Tags behavior should be aligned with the Hangfire behavior.

[Tag("base-class")]
public class BaseService
{
    [Tag("base-method")]
    [DisplayName("base-method")]
    public virtual void Run()
    {
    }
}
[Tag("overriding-class")]
public class OverridingService : BaseService 
{
    [Tag("overriding-method")]
    [DisplayName("overriding-method")]
    public override void Run() 
    {
        base.Run();
    }
}

image
image

Note1: I tested it also using PostgreSql Storage so it doesn't seem to be storage related.
Note2: There is other issue present in my example url (not related to this one)

Suggestion: Multi-tag searching/refinement

Since you are so amazingly quick to implement suggestions, here is another one :)

Just to give some background, we operate on a scale of about 50,000 jobs per week. At any point in time we have around 15-20k jobs in the Scheduled status, going from a few minutes to months in the future. Also I configured to hold on to completed jobs for up to 30 days for basic troubleshooting. That means in the complete job status we have typically around 1.5m jobs available for searching. Hence the use of Hangfire Tags is immense to help searching! (Before I discovered this I was using a bunch of handcrafted queries directly on the DB tables/json values to find and sort jobs)

To get a starting point, I made every job type (method name) a separate tag, and then every parameter value as a unique prefixed tag. For example, 'employeeid:123', or 'companycode:abcc' (yeah I realized afterward the ':' gets stripped out but it's okay). That way not only does every parameter have a unique searchable tag, but I can tell two int parameter value types apart. Unfortunately after only a week of production usage, I already have 41k tags, but I have not seen any performance issue thankfully! I also use the dropdown method instead of tag cloud.

Now to the question. When I am searching by tag, I often want to look for a combination of method, and one or more parameter values. Would it be possible for the tag search interface to allow entering multiple tags at a time, and search for the combination of all jobs having all entered tags? And maybe to go with that, when you are on the search results by tag, could there be a way to further refine the search, by entering more tags, but filtered only for jobs that match the existing tag filter, if that makes sense?

Thanks a bunch for all your efforts!!

Edit I don't know if there is a DM ability in Github, but I'd be willing to make a donation to your efforts for this request, let me know. Thanks again!

The specified storage is not suitable for use with tags

Hi,
I setup my app like you in sample core app:

        public void ConfigureServices(IServiceCollection services)
        {
            var sqlServerOptions = new SqlServerStorageOptions()
            {
                PrepareSchemaIfNecessary = false
            };

            services.AddHangfire(x =>
            {
                x.UseSqlServerStorage(Configuration["ConnectionString"], sqlServerOptions);
                x.UseTags();
                x.UseConsole();
            });
            services.AddHangfireServer();
        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            app.UseHangfireDashboard();
            GlobalJobFilters.Filters.Add(new ProlongExpirationTimeAttribute());
        }

But in UseTags I have error "The specified storage is not suitable for use with tags" when run app

Orphaned Tags Cleanup

Hi :)

I have been running smoothly with Tags here for a number of months now. I'm sorry I haven't followed up on the other requests, I should be able to now that we are out of our busy season. Thanks again for your efforts!

I'm noticing now I still have around 550K tags, even though a few months back, I removed one of the tag sources (email id) which was adding a bunch of useless tags (my own fault). The vast majority of those tags were one-time use and associated with jobs that have long since finished, expired, and been deleted. I believe I have our successful jobs configured to delete after 30 days.

Is there an easy way to check for all tags which have no associated job and clean them up out of the database? Is there some configuration as well that would automatically cleanup any tags when a job is cleaned up?

Thanks!

Edit: It is causing a bit of performance issue using the Tags screen is why I ask. It is taking around 10 seconds to open tags screen.

Could not load file or assembly 'Hangfire.PostgreSql, Version=1.9.5.0

Hi - I am having an issue with FaceIT.Hangfire.Tags.PostgreSql package v1.9.7-beta.2 and Hangfire.PostgreSq v1.20.0.
When this line gets executed:

app.MapHangfireDashboardWithAuthorizationPolicy(string.Empty);

I get a FileNotFoundException:

Could not load file or assembly 'Hangfire.PostgreSql, Version=1.9.5.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

I see that v1.9.5 is a dependency in the FaceIT.Hangfire.Tags.PostgreSql library, but I thought referencing v1.20.0 would override that?
I created a sample app that will throw the exception when you run it: https://github.com/LeorGreenberger/HangfireTagsIssue

upgrade Hangfire.MySqlStorage 2.0.3 Reference exception

upgrade Hangfire.MySqlStorage 2.0.3 Reference exception
System.TypeLoadException: Could not load type 'MySql.Data.MySqlClient.MySqlConnection' from assembly 'MySqlConnector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d33d3e53aa5f8c92'.
at Hangfire.Tags.MySql.MySqlTagsTransaction.ExpireSetValue(String key, String value, TimeSpan expireIn)

.net framework

nuget 没有.net framework版本的,可以添加一个不

Support for Redis?

This looks like a super-helpful extension. Are you considering adding support for other providers?

IdExtensions.Clean makes it difficult to grok a large number of tags

IdExtensions.Clean lower cases words and removes all punctuation except -. This makes it difficult to grok a large collection of tags. Keeping CamelCase names would be a big help and being able to use punctuation to categorize tags would be helpful. For example, we'd like to be able to "tenant: {tenantId}"

What is the purpose of this seemingly aggressive clean? Is there an option to not clean the names?

Thanks,
Drew

Hangfire tags has huge memory food print in redis storage

Hi,
in our scenarios (we use the redis stroage) the tagged jobs has a high memory consumption, in comparison to the plain jobs without tags. The keys are also kept in redis for a long time.
What is the reason for this?

For example:
In Redis storage the complete hangfire storage has a size of approx 30 MB and the "Tags" folder uses approx 20 MB of them.
image

Proposal for UI

Hello

Next Execution column does not show any data currently, though it might me meaningless to show. I propose to show there Total Duration.

Search for jobs with tags programatically

Hello! I've been looking for a way to tag/group jobs in the application we have written, and this extension seems right up the alley!

However, I am hoping there is a way for us to programmatically use the searching API provided to search for jobs that either contain a tag, or have a tag that starts with a specific phrase, etc.

The reasoning behind this, is we have a system that needs to ensure that all jobs that were triggered from a specific FormID (our own tracking mechanism) are all finished or cleared out before allowing someone to continue the workflow. The tags will be generated dynamically upon job enqueuing.

Does the extension have a way to query/count/etc jobs with specific tags or parts of tag names from a programmatic side and not just the dashboard itself? I noticed most classes are marked internal, so i'm unsure if this is available out of the box.

Thanks for your time and work!

Searching by tag throws exception when tag matches existing jobId

Consider an existing job with the ID 125462.
Searching by tag with the value 125462 (or using the URI ".../hangfire/tags/search/125462") returns the following error message:
{"Message":"Exception has been thrown by the target of an invocation.\r\nError converting data type nvarchar to bigint."}

Question on Tag Amount Limits and Duplicate Tags

Just a question on if there is a limit to number of total tags. I know the search is a drop down but my use case has an upper limit of like 50k tags so I am wondering how this dropdown will react to that many tags.

Also, what happens when we add the same tag multiple times in a single job? I assume nothing but wanted to clarify.

Provider for Redis / Expired Jobs

Hey in your notes, you mentioned that to cleanup tags from expired jobs a provider was needed, but I don't see one for Redis. Does that mean that right now if we use redis, we will end up with references to jobs that are expired?

TablePrefix bug

I set TablePrefix=“_hangfire” ,then click Jobs->Tags menu ,throw MySqlException: Table 'qcdataprocessor.hangfire_set' doesn't exist。
i use this code repository:
https://github.com/yuzd/Hangfire.HttpJob/blob/master/Hangfire.Tags.Mysql/Hangfire.Tags.Mysql.csproj

this is code:
//hangfire
services.AddHangfire(op =>
{
op.UseStorage(new MySqlStorage(Configuration["ConnectionStrings:DataProcessor"], new MySqlStorageOptions
{
TransactionIsolationLevel = System.Data.IsolationLevel.ReadCommitted,
QueuePollInterval = TimeSpan.FromSeconds(15),
JobExpirationCheckInterval = TimeSpan.FromHours(1),
CountersAggregateInterval = TimeSpan.FromMinutes(5),
PrepareSchemaIfNecessary = true,
DashboardJobListLimit = 50000,
TransactionTimeout = TimeSpan.FromMinutes(1),
TablePrefix = "_hangfire",
}));
op.UseTagsWithMysql();
});

this is detail error :
An unhandled exception occurred while processing the request.
MySqlException: Table 'qcdataprocessor.hangfire_set' doesn't exist
MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in ResultSet.cs, line 146

MySqlException: Table 'qcdataprocessor.hangfire_set' doesn't exist
MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet(ResultSet resultSet) in MySqlDataReader.cs, line 74

TargetInvocationException: Exception has been thrown by the target of an invocation.
System.RuntimeMethodHandle.InvokeMethod(object target, object[] arguments, Signature sig, bool constructor, bool wrapExceptions)

Stack Query Cookies Headers
MySqlException: Table 'qcdataprocessor.hangfire_set' doesn't exist
MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in ResultSet.cs

Show raw exception details
MySqlException: Table 'qcdataprocessor.hangfire_set' doesn't exist
MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet(ResultSet resultSet) in MySqlDataReader.cs
MySql.Data.MySqlClient.MySqlDataReader.ReadFirstResultSetAsync(IOBehavior ioBehavior) in MySqlDataReader.cs
MySql.Data.MySqlClient.MySqlDataReader.CreateAsync(MySqlCommand command, CommandBehavior behavior, ResultSetProtocol resultSetProtocol, IOBehavior ioBehavior) in MySqlDataReader.cs
MySqlConnector.Core.TextCommandExecutor.ExecuteReaderAsync(string commandText, MySqlParameterCollection parameterCollection, CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in TextCommandExecutor.cs
MySql.Data.MySqlClient.MySqlCommand.ExecuteScalarAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in MySqlCommand.cs
MySql.Data.MySqlClient.MySqlCommand.ExecuteScalar() in MySqlCommand.cs
Dapper.SqlMapper.ExecuteScalarImpl(IDbConnection cnn, ref CommandDefinition command) in SqlMapper.cs
Dapper.SqlMapper.ExecuteScalar(IDbConnection cnn, string sql, object param, IDbTransaction transaction, Nullable commandTimeout, Nullable commandType) in SqlMapper.cs
Hangfire.Tags.Mysql.MysqlTagsServiceStorage+<>c__DisplayClass6_0.b__0(DbConnection connection)
Hangfire.MySql.Core.MySqlStorage+<>c__DisplayClass20_0.b__0(MySqlConnection connection)
Hangfire.MySql.Core.MySqlStorage.UseConnection(Func<MySqlConnection, T> func)
Hangfire.MySql.Core.MySqlStorage.UseTransaction(Func<MySqlConnection, T> func, Nullable isolationLevel)
Hangfire.MySql.Core.Monitoring.MySqlMonitoringApi.UseConnection(Func<MySqlConnection, T> action)

Show raw exception details
TargetInvocationException: Exception has been thrown by the target of an invocation.
System.RuntimeMethodHandle.InvokeMethod(object target, object[] arguments, Signature sig, bool constructor, bool wrapExceptions)
System.Reflection.RuntimeMethodInfo.Invoke(object obj, BindingFlags invokeAttr, Binder binder, object[] parameters, CultureInfo culture)
System.Reflection.MethodBase.Invoke(object obj, object[] parameters)
Hangfire.Tags.Mysql.MysqlTagsMonitoringApi.UseConnection(Func<DbConnection, T> action)
Hangfire.Tags.Mysql.MysqlTagsServiceStorage.SearchWeightedTags(string tag, string setKey)
Hangfire.Tags.Storage.TagsStorage.SearchWeightedTags(string tag)
Hangfire.Tags.Dashboard.Pages.TagsSearchPage.Execute()
Hangfire.Dashboard.RazorPage.TransformText(string body)
Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context)
Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Show raw exception details

'could not serialize access due to concurrent update' with Postgre storage

First of all, thank you for developing this package. It is an almost perfect library to meet our business needs. Unfortunately, I encountered concurrency update issues happening at job creation time.

My config:

PostgreSQL 10 (running locally on my dev machine)
Hangfire 1.7.19
Hangfire.Postgre 1.8.1
Hangfire.Tags.PostgreSql 1.7.1

Exception

Here is the exception:

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
 ---> Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to concurrent update
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteNonQuery()
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader) in /_/Dapper/SqlMapper.cs:line 2822
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in /_/Dapper/SqlMapper.cs:line 572
   at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType) in /_/Dapper/SqlMapper.cs:line 443
   at Hangfire.PostgreSql.PostgreSqlWriteOnlyTransaction.<>c__DisplayClass18_0.<AddToSet>b__0(NpgsqlConnection con)
   at Hangfire.PostgreSql.PostgreSqlWriteOnlyTransaction.Commit()
   at Hangfire.Tags.Storage.TagsStorage.AddTags(String jobid, IEnumerable`1 tags)
   at Hangfire.Tags.HangfireExtensions.AddTags(String jobid, String[] tags)
   at Hangfire.Tags.States.CreateJobFilter.OnCreated(CreatedContext filterContext)
   at Hangfire.Client.BackgroundJobFactory.InvokeOnCreated(Tuple`2 x)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.EmptyProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
  Exception data:
    Severity: ERROR
    SqlState: 40001
    MessageText: could not serialize access due to concurrent update
    File: nodeModifyTable.c
    Line: 1012
    Routine: ExecUpdate
   --- End of inner exception stack trace ---
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at LINE REMOVED HERE

My use case:

The tags I use:

public class MyJobs
{
    [Tag("MyJobOperation", "{0}", "{1}", "relatedObject-{2}")]
    Task RunMyJobAsync(
      SomeEnum someEnum, //enum (about 10 values)
      string someJobType someJobType, //job type string (we can have about 10 unique values here)
      int relatedObjectId //An ID from our database
    );
}

Then, I have several recurring jobs that are responsible for creating RunMyJobAsync jobs for different relatedObjectIds. The recurring jobs can be run in parallel.

Here is the pseudo-code:

//Recurring job 1:

var someEnum = SomeEnum.Value1;
var someJobType = "MyFirstJobType";

for(int i=1, i<1000;i++)
{
    BackgroundJob.Enqueue<MyJobs>(a=>a.RunMyJob(someEnum, someJobType, i));
}

//Recurring job 2: running at the same time as Recurring job 1
var someEnum2 = SomeEnum.Value2; //The difference is here
var someJobType = "MyFirstJobType";

for(int i=1, i<1000;i++)
{
    BackgroundJob.Enqueue<MyJobs>(a=>a.RunMyJob(someEnum2, someJobType, i));
}

Reasons?

I think it makes sense that I get a concurrency update error here due to the fact the Hangfire.Postgre uses RepetableRead isolation level for its transactions. I check postgresql docs (1.3.2.2. Repeatable Read Isolation Level section), and here what it says:

Applications using this level must be prepared to retry transactions due to serialization failures.

I could potentially add a try-catch in my code to get it resolved, but it does not look like the best way to do it.

I think, that Hangfire.Tags should provide some kind of a retry strategy to deal with concurrency updates. Neither it provides one now, nor it allows to plugin a custom retry strategy. There is a public interface ITagsStorage, but there is no chance to provide a custom implementation for that because everywhere TagsStorage class is used directly with a new keyword.

I would be happy if we could include a retry strategy directly into TagsStorage.AddTags method, or at least give an opportunity to provide custom ITagsStorage implementation so that I could implement a retry strategy myself.

Any ideas about how to better fix my issue are really appreciated. Thanks!

Cannot use tags on multiple dashboards

Hello,
I'm hosting multiple dashboards in a single .NET Core 3.1 web application. Hangfire servers are hosted separately. Dashboards use different SQL storages.
Dashboards show the correct number of tags for each of them.
image
When I click on the Tags tab I get the following error.
image

I tried to configure multiple SQL storages in GlobalConfiguration. The exception was gone but both dashboards display tags from the latest storage that I registered.

Please advise how to configure tags for multiple dashboards in a single app for distributed environment.

Thank you!

SqlException when tag is too long

When a tag is used which exceeds the column size, an exception occurs.

Allow for a maximum length to be configured, and cut off longer tags.

How to allow .net 4.5 for Hangfire.Tags.SqlServer

Hello!

I know that the standard Hangfire.Tags allows for .net 4.5 - but I noticed that the Hangfire.Tags.SqlServer does not allow for .net 4.5.

As I don't have the option to upgrade the .net 4.5 version for reasons out of my hands, is there any information one can give to be able to enable that project to be usable in a .net 4.5 environment?

Invalid object name 'HangFire.Set'.

I use SchemaName, in both, dashboard and server, but the error try to find the "HangFire" schema and ignore the SchemaName in the settings

How to retrieve jobs from hangfire with a given tag id?

Does anyone know how can I retrieve jobs from Hangfire with a given tag id after using this extension? For instance, using a Where lambda expression or similar. I need to find out the status of all jobs with a certain tag.

(apologies in advance if this is not the right forum to ask)

NullReference Exception on Job Creation

Hi,
I've just updated the nuget package and I experience a nullreference exception on job creation.

This really seems to be the possible NRE you talk about in fd5dc19

  • Could you provide an update to fix it?
  • Is there a possibility to workaround this?

Best regards,

Suggestion: Allow clicking Tag on job view

When viewing an existing job in the Hangfire UI, it shows the list of tags associated with that job at the top. It would be really helpful if you could maybe click on the tag, and it would take you to the tag search interface, filtered for the tag you clicked on. Quick and easy way to find other jobs associated with that tag.

Tags are lost after some time

I've been looking at my failed queue and noticed that some jobs are there but when I click to view them they have no Tags and are not visible under the Tag to which they should.

  • Scrolling down 3000 failed jobs I find 30 jobs which should have the tags X, Y and Z
  • Viewing Tags and looking at my three tags I see one failed job (executed 9 hrs ago) which has all 3 tags.

Is there an expiration date to the tags and if so is it customizable, and shouldn't the tags be visible if the job can still be previewed in the failed queue explorer?

Version: 1.6.3

duplicate key value violates unique constraint "set_key_value_key"

While working on #35, I started off creating a unit-test to reproduce the error. 70% of the time, that test fails with the error described in #35. However, sometimes there is another exception saying Npgsql.PostgresException 23505: duplicate key value violates unique constraint "set_key_value_key"

Full stack trace:


Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.

Hangfire.BackgroundJobClientException
Background job creation failed. See inner exception for details.
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at Hangfire.BackgroundJobClientExtensions.Create[T](IBackgroundJobClient client, Expression`1 methodCall, IState state)
   at Hangfire.Tags.PostgreSql.Tests.Hangfire.Tags.PostgreSql.Tests.Integration_ConcurrencyTests.<>c__DisplayClass3_0.<AddTags_ShouldNotFail_WithConcurrencyError>g__CreateJobs|1(JobType jobType, CancellationToken token) in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\tests\Hangfire.Tags.PostgreSql.Tests\Integration_ConcurrencyTests.cs:line 65
   at Hangfire.Tags.PostgreSql.Tests.Hangfire.Tags.PostgreSql.Tests.Integration_ConcurrencyTests.<>c__DisplayClass3_3.<AddTags_ShouldNotFail_WithConcurrencyError>b__4() in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\tests\Hangfire.Tags.PostgreSql.Tests\Integration_ConcurrencyTests.cs:line 75
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at Hangfire.Tags.PostgreSql.Tests.Hangfire.Tags.PostgreSql.Tests.Integration_ConcurrencyTests.AddTags_ShouldNotFail_WithConcurrencyError() in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\tests\Hangfire.Tags.PostgreSql.Tests\Integration_ConcurrencyTests.cs:line 80
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
--- End of stack trace from previous location where exception was thrown ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

Npgsql.PostgresException
23505: duplicate key value violates unique constraint "set_key_value_key"
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteNonQuery()
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader) in /_/Dapper/SqlMapper.cs:line 2813
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in /_/Dapper/SqlMapper.cs:line 572
   at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType) in /_/Dapper/SqlMapper.cs:line 443
   at Hangfire.PostgreSql.PostgreSqlWriteOnlyTransaction.<>c__DisplayClass18_0.<AddToSet>b__0(NpgsqlConnection con)
   at Hangfire.PostgreSql.PostgreSqlWriteOnlyTransaction.Commit()
   at Hangfire.Tags.Storage.TagsStorage.AddTags(String jobid, IEnumerable`1 tags) in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\src\Hangfire.Tags\Storage\TagsStorage.cs:line 116
   at Hangfire.Tags.HangfireExtensions.AddTags(String jobid, String[] tags) in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\src\Hangfire.Tags\Extensions\HangfireExtensions.cs:line 19
   at Hangfire.Tags.HangfireExtensions.AddTags(String jobid, IEnumerable`1 tags) in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\src\Hangfire.Tags\Extensions\HangfireExtensions.cs:line 12
   at Hangfire.Tags.States.CreateJobFilter.OnCreated(CreatedContext filterContext) in D:\_Projects\Hangfire\Hangfire.Tags.Fraz\src\Hangfire.Tags\States\CreateJobFilter.cs:line 40
   at Hangfire.Client.BackgroundJobFactory.InvokeOnCreated(Tuple`2 x)
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.EmptyProfiler.InvokeMeasured[TInstance,TResult](TInstance instance, Func`2 action, String message)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_1.<CreateWithFilters>b__2()
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_1.<CreateWithFilters>b__2()
   at Hangfire.Client.BackgroundJobFactory.CreateWithFilters(CreateContext context, IEnumerable`1 filters)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)



Test failed after have created 1887 jobs

Environment

  1. Hangfire.Tags 1.7.2
  2. Hangfire.Tags.PostgreSql 1.8.0
  3. Hangfire.PostgreSql 1.8.1
  4. Hangfire.AspNetCore 1.7.19
  5. .NET Core 3.1
  6. Postgres 13 (latest docker images)
  7. I7-10700K (16 cores)

Steps to reproduce

  1. Checkout commit: Frazi1@b0705b6
  2. docker run --name postgres-hangfire -e POSTGRES_PASSWORD=test -p 5432:5432 -d postgres
  3. IMPORTANT Create a fresh DB instance for each test run. The error happens more often on a fresh DB.
  4. Adjust the connection string in testsettings.json file, if needed
  5. Run AddTags_ShouldNotFail_WithConcurrencyError unit-test

Actual behavior

The unit test fails with either exception from #35, or with duplicate key value violates unique constraint "set_key_value_key" error.

You may have to re-run the unit test several times (make sure you run it on a fresh DB instance each time) to get this very exception.

I haven't tried researching this error yet. @erwin-faceit Have you seen it before? Any ideas? Maybe it is another subject for being handled by the retry policy from #35.

The functions QueueCommand and AddCommand cannot be found error

After migration to Hangfire 1.8.6 tasks stopped running getting this error below. We use Hangfire.Tags 1.8.5

System.ArgumentException: The functions QueueCommand and AddCommand cannot be found.
at Hangfire.Tags.SqlServer.SqlTagsTransaction..ctor(SqlServerStorageOptions options, IWriteOnlyTransaction transaction)
at Hangfire.Tags.SqlServer.SqlTagsServiceStorage.GetTransaction(IWriteOnlyTransaction transaction)
at Hangfire.Tags.Storage.TagExpirationTransaction.Persist(String jobid)
at Hangfire.Tags.States.TagsCleanupStateFilter.OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
at Hangfire.States.StateMachine.InvokeOnStateApplied(Tuple`2 x)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.