Data Migration, Management, and Reporting Continuity for a UK Professional Membership Organisation
.png){kind=small}
Data Migration, Management, and Reporting Continuity for a UK Professional Membership Organisation
VE3 led the data migration and management workstream for a UK professional membership organisation holding approximately 900 GB of CRM data, over four billion records, across four structurally distinct on-premises databases. The scope covered pre-migration data profiling and optimisation, database-specific migration strategies, validated data movement to Dataverse and Azure SQL, and a reporting continuity approach to preserve 163 active SSRS reports from the first day of go-live.
The data estate comprised the core CRM database, a master configuration database managing GUID consistency, an 86 GB CRM Additions database with complex stored procedures, and a read-only legacy CONCEPT database. Each had different technical dependencies, different cloud destinations, and different sequencing constraints, no single migration approach applied across all four.
Three tables - Attachment (229 GB), ActivityPointerBase (163 GB), and AuditBase (81 GB) - accounted for over 55% of the total database. A direct lift-and-shift would have resulted in substantially higher ongoing Dataverse storage costs, much of it for data past its operational retention value.
SSRS is not natively supported in Dynamics 365 Online, placing 163 production reports immediately at risk. Recreating all 163 from scratch was cost-prohibitive; a strategy was needed that preserved full reporting functionality from go-live day without any period of downtime.
The configuration database - maintaining GUID consistency across all CRM environments - was managed entirely through 149 manually executed batch files with no version control, no automation, and no audit trail. Migrating incorrectly risked breaking GUID references across every platform customisation.
The 86 GB Additions database contained complex stored procedures, functions, and views supporting event management and scheduling. Its complexity made direct Dataverse migration technically unviable within the project timeline, requiring an alternative hosting strategy.
The scale of the dataset meant a single full-volume migration attempt without incremental validation would make it impossible to isolate defects before they propagated. The organisation required clear rollback capability and confirmed data integrity at every stage before proceeding to the next.
A data profiling exercise quantified volume distribution and identified archiving opportunities. A pre-migration pruning sprint then safely archived and deleted records per the organisation's retention policies and Fair Processing Notices. 229 GB of historical file attachments were relocated to SharePoint and Azure Blob Storage, accessible via Dataverse virtual tables but removed from premium cloud storage, reducing projected Dataverse costs by over £15,400 per month.
The core CRM database was migrated directly into Dataverse via Microsoft LCS OP2OL tooling. The master configuration database was deprecated and consolidated into the development environment. Its 149 batch files were replaced with Azure DevOps CI/CD pipelines. The Additions database was migrated to a dedicated Azure SQL instance, preserving all stored procedures without rewriting. The CONCEPT database was migrated to a separate Azure SQL instance within the analytics layer, continuing to serve historical reports unchanged.
Data migration was validated in progressive tranches - 1%, 5%, 25%, 50%, then 100%, across a minimum of three complete mock migrations in the staging environment before production cutover. Issues identified at 1% were resolved before 5% was attempted, ensuring defects were never discovered at full scale for the first time.
Rather than recreating 163 reports, VE3 established a dedicated Azure SQL reporting database receiving a continuously refreshed mirror of Dataverse data via automated pipelines. All reports were repointed to this source with minimal change, maintaining full reporting functionality from the first day of go-live with no downtime and no retraining required.
The 149 manually executed configuration batch files were replaced entirely with automated Azure DevOps CI/CD pipelines. Configuration changes now flow from development to UAT and production through version-controlled, auditable pipeline runs, eliminating manual error risk and introducing deployment discipline for the first time.
Throughout the programme, Simego DataSync Studio maintained continuous delta synchronisation between the on-premises CRM and the cloud environment. This kept the integration team's development target current throughout the programme and ensured the final production cutover required only a minimal delta migration, reducing the cutover window and the risk of data drift.
.png)
Migrating a 900 GB, four-database CRM estate to the cloud is not a single project - it is four parallel engineering problems, each with its own risk profile. VE3's differentiated approach, combined with pre-migration optimisation and incremental validation at every stage, delivered a cloud data estate that is intact, accurate, leaner, and more cost-efficient than what it replaced, with full operational continuity from go-live day.
.png)
