Designing your data migration properly is key to its success. But what steps should you take to ensure this outcome?
So, your data migration project has reached a stage where a new technology platform has been selected and you’ve chosen implementation partners. Next, it’s time to look at the design of your data migration. Getting the design right will ensure your migration runs smoothly, and it all starts with a decision about your approach.
Big Bang or Phased Migration?
Generally, there are two extremes when carrying out a migration. Either you select a date and plan to extract all your data from your legacy systems, prevent all updates to those systems and run your migration code to transform your legacy data and load it into your new platform. We refer to this as the ‘big bang’ approach, as it means that as soon as the data is loaded, you make your new application available to users. It’s a simple approach – a clean break and a fresh start, but it’s not always possible and it carries more of a risk of delay, as everything must be complete before hitting ‘go’ on your migration.
Alternatively, there’s always the phased approach. In this strategy, you migrate portions of your legacy data and business processes to your new application. This would require parallel running of both your new and legacy application, but it works well if you have distinct, exclusive alcoves of data and processes that don’t draw from common objects or feature any dependencies. These data segments can be carved up and moved, and the teams responsible for them can be deployed to the new platform. Move onto the next segment and department until your migration is complete and you can initiate the legacy system’s decommissioning.
However, in our experience this virtually never happens. It’s rare to have distinct business functions with no overlap in your legacy data which makes a phased approach much more complex. You now must consider some form of data synchronization or cut-over facility where common data and objects are kept in sync. This can get complex as the number of shared fields increases. This is generally the more expensive approach, due to the increase in design and development needed.
Once you’ve decided on your approach, you’re ready for the real design work!
Data Migration Mapping
The whole reason the practice of data migration exists is because your new application will generally be substantially different to your old one. If it isn’t, then you should ask yourself some questions about if you really needed a new one. For this step though, you must plot a course, which requires a map!
When creating a data migration map, there are four aims to consider:
- Showing the direction of travel for data
This shows where a piece of data originates from and where you plan on sending it in your new platform. This is what most people consider the fundamental aspect of mapping.
- Detailing the transformation that needs to take place
Your DQR (Data Quality Rules) database is the vehicle to help here. Not all your transformations will be set as DQRs, but the majority should be. If you don’t have a transformation in your DQRs, unless it’s a simple one-to-one mapping, then you should raise a new DQR at that point so there is a record of what transformations need to take place.
- Orchestrating the transformation
Many of the transformations may need other transformations to have taken place first before they can proceed. So as part of your design you need to consider this as part of an orchestration step. For example, a contact needs to exist for it to be a campaign member or opportunity. During mapping design, you should start to look at objects and fields to identify the order in which they may need to be transformed as they are a dependency for a later transformation. And if this wasn’t complicated enough, you may even find there are system generated values in your new platform that are dependencies for your transformations.
- Identifying the verification rules to reconcile the data
You also need to consider counting the data you’re planning to move. Is it all accounted for and how will you know? If you have data being moved from one place to several places in the new system, with potentially some data being dropped, ensure you add in some simple logic to your map that your developers can use to check their code when they come to write it.
Gap Analysis
The gap analysis is a great tool as part of your migration arsenal. Its purpose is to review all the data that is being left behind by your directional plan and it works by answering two questions:
- What data should populate these unpopulated fields in our new platform?
Having worked through your mapping, it’s probable that you will be faced with multiple fields in your new platform that don’t appear to have any legacy data available to populate them. That’s OK and it’s why you run the design phase like this. Your job is to raise all these as DQRs. That way they will be triaged, and the subject matter expert and SI will provide an overview of what data they expected would be in there and if it already exists, where it can be located and what transformation rules should be applied to it.
- What legacy data hasn’t been accounted for in the new platform?
This question can highlight any areas where your requirements definition wasn’t up to scratch, meaning more requirements being added to your platforms at eleventh hour. Though in fairness to the Business Analysists out there, it’s impossible to capture and cater for every requirement in a project of this size and scope.
In fact, there’s likely to be lots of redundant legacy data identified by the gap analysis that isn’t needed any more. If you suspect that it may be needed in the future, it can be placed in a data warehouse or data lake for future use.
In Summary
In our experience, deciding on an approach for your migration, then following the above steps of mapping and gap analysis leads to a more seamless migration process. Euler CEO Rob Jones has compiled his years of experience with data migrations into a fascinating white paper. Here, he details the A-Z process of complex data migration projects. Click here to view it.
