Data storage challenges and changes for Dataverse data

Many administrators of Microsoft Dynamics 365 Online and Microsoft Power Platform have been challenged by managing the storage of their data verse data. Sometimes, especially in this world of citizen developers, it feels that a cloud over-capacity issue can come out of nowhere!

Originally, we only had to manage the storage. As long as we stayed within our capacity, life was sweet.  Then the storage was split into three areas:

• Database
• File
• Log

Each of these areas is independent and may require the purchase of additional space. It is not possible to “borrow” space from one area to meet the needs of another. Interestingly, although the areas are independent, different environments within one tenant are not.

Best practice is to not allow any of the storage areas to go above 80% full.

Most production systems will use all three of these storage capacities. Typically sandbox, training UAT and test environment use far less of the file and log capacity.

The database capacity stores most of the data from dataverse – note that it is most, but not all. The file capacity stores email attachments and other attachments linked to records. The log capacity stores the audit log files.

Interestingly, the default amount of storage is greater for files than database - the default storage capacity for database is 10GB per tenant and 0.25GB per user, while the default storage capacity for file capacity is 20GB per tenant and 2GB per user.

If you use a lot of attached files - and images on forms are attached files, even though they may appear to be simple data - it will be cheaper to configure your environment to use SharePoint storage. Dataverse storage is currently $30/GB/Month, compared to SharePoint costing $0.2/GB/Month.

Understanding what is stored where is important if you need to manage capacity. Once you exceed your available capacity on any of the areas, some actions within your environment will become slower. It will be impossible to back up or restore an environment once you have run out of storage.

Currently all of the text of an email is stored in the database storage capacity.

The change that is coming

However, this is changing to store the email description field, i.e. the body of the email, which may also contain embedded images in the file capacity. This means that like the other data stored in the file capacity, the email description field will be stored in unstructured blob storage.

The data migration from Dataverse database capacity storage to Azure Blob storage is expected to start in May 2023. This data migration will take place as a background process. The initial data movement for existing customers is expected to last for about 6 to 12 weeks, and possibly even longer depending on the size of the data. After the initial data movement, any remaining migration is a continuous process. All email descriptions older than 12 months will be moved into Azure Blob storage automatically. Newer emails will not be moved until they become 12 months old.

This data migration will be transparent to you with the exception that you will see a reduction in the size of the ActivityPointer table after the migration process is fully completed. In Power Platform admin center within the Capacity report, a new email line item will be available in File usage. The end result is an increase in the overall File storage consumption and reduction in the database storage consumed. However, there will overall be a reduction in the storage used because the increase in file storage will probably be smaller than the data that is removed from the ActivityPointer table because of compression of file data.

Apart from the change in storage capacity, there will be little that users or administrators will notice with this change. However, there is a change in how you can search or sort on the email description field. Once this data is moved to file, ie Azure Blob storage, users will not be able to search in email bodies or filter on email description data. Users will be able to include the email description field in the output of an Advanced Find.

This change may mean that you want to ensure that you users are using the regarding field of emails (and other activities) correctly, ie specifically linking to the relevant record, rather than linking everything to an account and then searching on specific words in the body of an email.