Now that we know the fundamental architecture from Part 1 and how to create jobs using the client from Part 2, we’ll create a Hangfire server to process our jobs.
Set-up and dependency installs
Our server application needs the same dependencies as the client application we created in Part 2:
dotnet add package Hangfire.Core
dotnet add package Hangfire.SqlServer
dotnet add package Microsoft.Data.SqlClientConfiguring Hangfire
The configuration is almost the same as in Part 2. We instruct Hangfire to use the same database as in our client application so it can see the jobs we create with the client. To get some additional output from Hangfire server, we also configure the colored console log provider. Please do not use this log provider in production as it will negatively impact performance. Use one of the other log providers like NLog or Serilog in production.
using Hangfire;
GlobalConfiguration.Configuration
.UseSqlServerStorage(@"
Server=.;
Database=MyApp;
Integrated Security=True;
TrustServerCertificate=True;
")
.UseColouredConsoleLogProvider();Starting two servers for processing
To simplify our following investigation, we’ll create two server instances within our console application. In a real application, you’d have one Hangfire server per physical server/VM/AppServicePlan. The code also configures names for our servers and reduces the worker count per instance because the default is too much for our sample: Math.Min(Environment.ProcessorCount * 5, 20).
var serverOptions = new BackgroundJobServerOptions[]
{
new BackgroundJobServerOptions
{
ServerName = "#1",
WorkerCount = Environment.ProcessorCount/2
},
new BackgroundJobServerOptions
{
ServerName = "#2",
WorkerCount = Environment.ProcessorCount/2
}
};
using(var s1 = new BackgroundJobServer(serverOptions[0]))
using(var s2 = new BackgroundJobServer(serverOptions[1]))
{
/*
This code is here to make our main thread wait for Ctrl+c to be pressed
before shutting down and disposing the servers.
*/
TaskCompletionSource cancelKeyPressed = new TaskCompletionSource();
Console.CancelKeyPress += (_, consoleCancelEventArgs) => {
consoleCancelEventArgs.Cancel = true;
cancelKeyPressed.SetResult();
};
await cancelKeyPressed.Task;
Console.WriteLine("Exiting…");
}
When we run this console application, we should see some output, showing both our servers started successfully:
...
[INFO] (...) Using the following options for Hangfire Server:
Worker count: 4
Listening queues: 'default'
Shutdown timeout: 00:00:15
Schedule polling interval: 00:00:15
...
[INFO] (...) Using the following options for Hangfire Server:
Worker count: 4
Listening queues: 'default'
Shutdown timeout: 00:00:15
Schedule polling interval: 00:00:15
[INFO] (...) Server #1:23232:1a145898 successfully announced in 247.1951 ms
[INFO] (...) Server #2:23232:649b5f5f successfully announced in 247.2125 ms
...
[INFO] (...) Server #2:23232:649b5f5f all the dispatchers started
[INFO] (...) Server #1:23232:1a145898 all the dispatchers started
In our MyApp database, the servers should be visible too:
USE [MyApp]
SELECT * FROM [HangFire].[Server] [s]
Id Data LastHeartbeat
#1... {"WorkerCount":4, ...} 2024-06-03 10:21:10.563
#2... {"WorkerCount":4, ...} 2024-06-03 10:21:10.563
Stopping the server
Gracefully stopping our application and the Hangfire servers also works by pressing Ctrl+c in the terminal. (VSCode: make sure you configure "console": "integratedTerminal" in launch.json because the Debug Console does not forward Ctrl+c.)
Exiting...
[INFO] (...) Server #2:8220:c56ade91 caught stopping signal...
[INFO] (...) Server #2:8220:c56ade91 All dispatchers stopped
[INFO] (...) Server #2:8220:c56ade91 successfully reported itself as stopped in 4.1553 ms
[INFO] (...) Server #2:8220:c56ade91 has been stopped in total 48.315 ms
[INFO] (...) Server #1:8220:fa21dd79 caught stopping signal...
[INFO] (...) Server #1:8220:fa21dd79 All dispatchers stopped
[INFO] (...) Server #1:8220:fa21dd79 successfully reported itself as stopped in 1.1618 ms
[INFO] (...) Server #1:8220:fa21dd79 has been stopped in total 29.4216 ms
Looking in the database reveals that the servers deleted themselves from the [HangFire].[Server] table. If you shut the application down forcefully, the servers will not delete themselves and appear as available. If you start the app again it will look like four servers are available. The heartbeats of those zombie servers will not be updated, however, and after some time (~5 Minutes) they will be removed from the table by one of the live servers. This event will be logged:
[INFO] (Hangfire.Server.ServerWatchdog) 2 servers were removed due to timeout
Creating Jobs for our servers
Using our knowledge from Part 2, let’s create some new jobs for our servers to process. I’ll create a new Jobs static class in the client console application containing two jobs:
public static class Jobs
{
public static async Task HelloJob(string name){
for(uint i = 0; i < 5 ; ++i)
{
await Task.Delay(5);
Console.WriteLine("Hello " + name + "! " + i);
}
}
public static Task BrokenJob(){
throw new NotImplementedException();
}
}
Then in the Program.cs file, we’ll enqueue some jobs to execute on our server:
BackgroundJob.Enqueue(() => Jobs.HelloJob("Michael"));
BackgroundJob.Enqueue(() => Jobs.HelloJob("James"));
BackgroundJob.Enqueue(() => Jobs.HelloJob("Thomas"));
BackgroundJob.Enqueue(() => Jobs.BrokenJob());
Now I’ll start the server application first, then the client application aaaaaand… the server application crashed. The client application finished without errors though.
2024-06-03 01:39:13 [WARN] (Hangfire.AutomaticRetryAttribute) Failed to process the job '3': an exception occurred. Retry attempt 3 of 10 will be performed in 00:00:34.
System.IO.FileNotFoundException: Could not load file or assembly 'HangfireClient, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'HangfireClient, Culture=neutral, PublicKeyToken=null'...
So our server assembly is trying to load the client assembly HangfireClient at runtime but fails. Why does it try to load the client assembly? Easy: the code we want the server to execute lives in our Jobs class which is part of the client assembly. To solve this, the code we want to execute must be present in an assembly that is accessible by both the client application and the server application. So we’ll create a new classlib project to house our Jobs class and add a dependency to this assembly in both our client and server applications.
dotnet new classlib -n Jobs
dotnet add .\Hangfire\ reference .\Jobs\
dotnet add .\HangfireClient\ reference .\Jobs\Now restart the server application and see if it works. We still get the same error. This is because the invocation information the client wrote into the database is still referencing the wrong assembly. So We’ll delete the old jobs from the database, start the server and then the client. Now we get the output we expected! The Hello job prints to the console and the broken job fails every time and is scheduled to be retried after some time:
Hello Michael! 0
Hello James! 0
Hello Thomas! 0
Hello James! 1
Hello Michael! 1
Hello Thomas! 1
Hello James! 2
Hello Michael! 2
2024-06-03 02:08:34 [WARN] (Hangfire.AutomaticRetryAttribute) Failed to process the job '8': an exception occurred. Retry attempt 1 of 10 will be performed in 00:00:26.
System.NotImplementedException: The method or operation is not implemented.
at Jobs.BrokenJob() in C:\Users\oberauer_m\source\repos\Hangfire\Jobs\Jobs.cs:line 11
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Hello Thomas! 2
Hello Michael! 3
Hello James! 3
Hello Michael! 4
Hello James! 4
Hello Thomas! 3
Hello Thomas! 4
2024-06-03 02:09:07 [WARN] (Hangfire.AutomaticRetryAttribute) Failed to process the job '8': an exception occurred. Retry attempt 2 of 10 will be performed in 00:00:56.
System.NotImplementedException: The method or operation is not implemented.
at Jobs.BrokenJob() in C:\Users\oberauer_m\source\repos\Hangfire\Jobs\Jobs.cs:line 11
at InvokeStub_Jobs.BrokenJob(Object, Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
As you can see, it’s quite hard to get an overview of what is processed, which jobs exist and the state they are in. To simplify this, Hangfire includes a web UI implemented as OWIN middleware that simplifies this process tremendously: the Hangfire Dashboard. Join me in Part 4 where we’ll create an ASP.NET Core web application to host the Hangfire Dashboard.
Photo by Mark Hindle on Unsplash