Log Analytics Workspace Retention Reporting Options (Part 1)

Hi all,

This is the first of two posts that I will be doing on how you can report on the Retention settings of an Azure Log Analytics workspace. In the second post I will provide a sample Workbook for displaying the settings.

It is often that during my conversations with customers about Azure Monitor, Azure Security Center and Azure Sentinel, the topic of data retention comes up. In most cases discussing the default global settings of:

• 31 days for Log Analytics
• 90 days for Application Insights
• 90 days for an Azure Sentinel linked workspace
• and the maximum retention time of 730 days

are sufficient but then there are those occasions when a customer wants to retain certain data types for either a longer or shorter period of time, because either the data becomes stale and therefore not of value or they are thinking about cost optimisation.

Note: the above global defaults do not apply to Free Pricing tier, which has a retention of 7 days.

The process to change the data retention period is part of the Azure Monitor documentation in the Usage and Cost section. Changing the data retention period using the Azure portal is a global change across all data types. The ability to set retention by data type has been available since October 2019 and can be changed by utilising the Azure Resource Manager REST API.

Using this method it is possible to set different retention settings for individual data types from 30 to 730 days. I should note that both Usage and AzureActivity data types are retained for a minimum of 90 days by default and these cannot be set any lower.

So now that we know that it is possible to set individual settings, how do we go about setting it? The documentation provides a link to an OSS tool – ARMClient but I decided to take a slightly different approach and used Postman as it allowed me to save individual requests and then come back to them at a later date.

After downloading and installing the Postman client, I used this great blog post from Jon Gallant to configure Postman to work with Azure AD. It also provides some examples to make sure that everything is working correctly.

One of the things that I like about using Postman is that I can set variables for an environment and then reuse them when constructing the REST API calls. Although not shown in the screenshot below, I created variables for each of my workspaces and then inter-changed the variable as I needed.

With that bit done, it was now a case of creating the API requests applicable to the Azure Monitor Log Analytics workspace. The Azure Monitor documentation provides some example code for a GET request to list the retention for all the tables in a workspace:

GET /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables?api-version=2017-04-26-preview

Getting the current settings

Rather than walk through the creation of a new request, below are some screenshots showing what a GET request looks like in Postman (note: I have utilised the variables in the construction of the GET request, so it now looks like this:

{{resource}}/subscriptions/{{subscriptionId}}/resourceGroups/{{ala-workspace}}/Tables?api-version=2017-04-26-preview

Params section:

Headers section:

Once these have been set, clicking the SEND button will connect to the workspace and return the current settings as shown below:

The screenshot above shows that I have already updated the retention settings for the ConfigurationData table to 30 days.

Setting the Retention

To change the setting(s) I created a new PUT request and the settings in the Params and Headers sections are the same as the GET request but now I needed to add the necessary code to the Body section to actually set the retention period – see below:

And like the GET request, clicking the SEND button connected to the workspace and updated the setting for the chosen table. In the screenshot below, I changed the setting for the ConfigurationData table to 60 days.

 

In summary, once you have Postman set up to work with Azure AD and you get your environment variables configured, it is a very simple and straightforward process to:

• check the existing settings across all tables or individual tables,
• but you can easily update the retention periods to suit your needs whether it is from a cost optimisation point of view or you simply don’t want to retain specific data types.

 

In the next post I will use the same REST API calls but will display the results in a workbook.