Arkivum 101: Data Migration into an Archive

There is a perception (rightly or wrongly) that migrating large datasets between systems is tough. This includes beliefs that lots of effort and resources are required, it’s easy to make mistakes which result in losing data (i.e. risks to data integrity) and it’s difficult to resolve any issues if they happen. This is compounded by the fact it usually feels easier to leave the data where it is to be worried about later down the line. 

When it comes to retaining data for long periods of time, this approach is not only unfeasible, but is putting that data at risk. 

If required to retain data for decades or longer, it is unavoidable that you will need to migrate that data at some point. Systems will change, formats will evolve, and business requirements will shift, all requiring data to be moved.  

It therefore follows that data must be kept so that it is portable, to adjust to an ever-changing technology and business landscape. This, I would argue, is a paradigm shift in perception for many. 

Why Migrate Now? 

As discussed, it can feel easier to leave data where it is and deal with it later… “Why do today what I can put off until tomorrow?” 

Well in fact, there’s actually quite a few good reasons to act today and not leave things for tomorrow; 

• Staff and those involved with the related projects/work are familiar with the data/datasets involved. This not only will make any migration efforts more efficient, but they’re also less likely to make mistakes and that familiarity makes it’s easier to rectify any mistakes that do occur.  

• Those same staff and stakeholders also know and are familiar with the technology/systems, offering the same benefits as the above point. Additionally they can be confident that they will be accessible; the longer that data is left in situ, there is an increased risk of vendor lock in, or that those systems/the data has become obsolete. 

• It can be harder to justify the work and any related expense further down the line, potentially delaying the work indefinitely and thus putting that data at risk in future and over the long term.  

 In this section I’ve assumed that you are familiar with the risks and challenges associated with retaining data long term within source systems (i.e. leaving the data where it is!). If you’d like to find out more about this topic, we’ve covered why source systems aren’t fit for purpose for long term data retention in this post here. 

Building Blocks for a Successful Data Migration 

I hope I have provided a strong case for why it’s important to migrate data from a source system into a long term repository soon after the conclusion of the project or activity which generated that data. 

So what do I/you need in place to make that migration as simple and efficient as it can be, while also ensuring that data integrity is maintained throughout? In this section I’ll be talking about migration in the context of moving data into a system like Arkivum’s, but these apply for any data migration projects who may have. 

Successful migration requires three main pillars: 

• Process: it’s crucial to have a clear understanding of the end-to-end process for the migration. Key questions to help define this include how are you getting the data (and who from), where is it being held in transit, how do you plan to check it before upload (both for errors and missing files), is it properly organised for archival, where is it going and how are you getting it there. Once you have answered these types of questions, it’s important to create supporting processes to enable a smooth migration. 

• People: as mentioned previously, having staff familiar with the data and systems involved can make migration efforts much easier. Coupled with this, at a minimum it’s important to ensure that you have some experience or expertise in managing migrations, either internally or from an external supplier. If this experience and expertise is limited, it is worth considering running a couple of practice migrations before committing. This though would take more time and resource. 

• Technology: the final pillar is having the right technology to support the processes you’ve created. These can range from having the ability to automatically check all files for any data integrity issues to being able to evidence a full chain of custody throughout the migration. It might sound obvious, but it’s important to ensure that any technology used has been assessed to be fit for purpose in supporting those processes. 

Successful data migration requires elements of all three pillars. This is much in the same way as three-legged stool analogy for successful digital preservation (created by Anne R. Kenney and Nancy Y. McGovern). I covered this in another blog post on “8 Tips for successfully keeping digital records for 25 years”. 

How can Arkivum Support Your Data Migration? 

Throughout this post I’ve talked about migration efforts in general terms of transferring data into an archive. I now want to dedicate a section on how Arkivum supports customers to migrate data into our solution.  

We aim to make it as simple as possible for customers, by offering two routes; 

1. Manage the process yourself: We provide customers with everything that they need to migrate data themselves into the Arkivum solution. This includes training and guidance documentation, as well tools to support the process (for example you can take a look at our recent release of IngestIQ). This is in addition to standard checks conducted when all data is ingested into the Arkivum solution (e.g. data integrity checks, virus scanning etc.) 
2. Use the Arkivum migration service: we offer a migration service to manage the end-to-end data transfer from source system to archive. Our team of experts will work with you and both your internal and external stakeholders, to move your data from where it is, to where it needs to be. The objective of this service is to deliver an efficient, safe and secure migration while reducing the level of customer effort required, and providing any necessary oversight. 

 Which route you choose will largely depend on your internal resource and expertise when it comes to not only data migration, but records management and archiving. This is something that the team would be more than happy to discuss with you in more detail, or you can visit our migration service page to find out more.  

Don’t Put Off What You Should Do Today 

It may feel easier to do nothing when it comes to migrating valuable data into a long term archive, but it’s something which shouldn’t be put off for too long. Leaving data within source systems can put that data at risk in the long term, and migration challenges and risks increase the longer it is left where it is. 

Don’t put off what you should do today.