Reactive handling of secret key rotation in .Net Core Applications using Polly.

Secret key rotation is a common process in production applications where organizations do a periodic or sometimes adhoc key rotation of their azure services like Cosmos DB, Storage Accounts, Event Hubs etc. for the security reasons.

When the new rotated keys are updated to KeyVault, the consuming apps start failing as the secrets used are not valid anymore, so either we must restart our apps or programmers implement long polling to KeyVault to do a periodic secret refresh. Both approaches are not efficient as the first one requires manual intervention or some sort of custom app that can trigger an app restart when keys get rotated and in the second one if you poll too frequently then it will add to unnecessary KeyVault cost and if you do it with a longer duration then your app can have a downtime.

Considering this problem we have a GitHub repository in azure samples as a .Net Core Web API in which retry policies of Polly library are used to pull the updated KeyVault secrets reactively and update the corresponding service classes to refresh the connections. In the sample code this approach is demonstrated against two azure services (cosmos and storage account) but you can extend this to other services as well.

Defining retry policies for the corresponding services:

Here is the implementation

Cosmos Retry Policy:

This retry policy watches for CosmosException and filters that by unauthorized status code, this will happen when your existing key is no more valid after key rotation, in this case it will fetch the latest secret key from key vault and refresh the cosmos connection by calling the reconnect method from cosmos service.

// Get the latest secret from Keyvault for cosmos key
var cosmosKeySecret = await keyVaultClient.GetSecretAsync(configuration["KeyVaultUrl"], 
Constants.CosmosKey).ConfigureAwait(false);
// Refresh cosmos connection
cosmosDbService.Reconnect(new Uri(configuration[Constants.CosmosUrl]), cosmosKeySecret.Value, configuration[Constants.CosmosDatabase], configuration[Constants.CosmosCollection]);

Blob Retry Policy

This retry policy watches for RequestFailedException and filters it by unauthorized or forbidden error codes and then attempts to refresh the storage service connection.

// Get the latest secret from keyvault for blob connection.
var blobConnectionSecret = await keyVaultClient.GetSecretAsync(configuration["KeyVaultUrl"], Constants.BlobConnection).ConfigureAwait(false);
//Refresh storage account connection.
blobStorageService.RefreshBlobServiceClient(blobConnectionSecret.Value);

Dependency Injection Extension for KeyRotation retry policies:

In CreateHostbuilder method of program.cs class you configure KeyVault and then register all the required services:

Here is a sample Web API controller which uses those retry policies.

At line # 34 and #53 you call the corresponding service methods with their respective retry policy handlers, which will be responsible for reactively refreshing the service connections when keys get rotated.

You can download the complete sample code from azure samples GitHub repository.