Arkivum 101: Why Data Sprawl is a Problem and How to Overcome It

With the rapid adoption of B2B technology in the last 5-10 years, many organisations find themselves with a myriad of platforms and systems. There is perhaps no better example than the rate of adoption of technology throughout the clinical trial process, with sponsors now using up to and beyond 10 different systems throughout a single clinical trial phase.

Many of these systems generate and/or store large amounts of data, giving rise to an exponential level of data sprawl. Data silos with various business owners, proprietary and non-proprietary systems (and data formats) and lack of joined up strategic planning creates numerous challenges.

These challenges only grow when retaining data for long periods of time. In this post we’ll consider some of the greatest risks of uncontrolled data sprawl, and how a consolidated approach to long term retention (and archiving) is quickly becoming a business imperative.

Why is Data Sprawl a Problem?

While data sprawl presents many challenges for organisations, I want to focus on the top four in the context of long term data retention:

• Difficult to actively safeguard and preserve data: in order to guarantee data is accessible, available and legible (in line with industry good practice e.g. ALCOA+) data must be actively managed and maintained throughout the entire retention period. With data dispersed across multiple locations, this becomes extremely time consuming, costly and in some cases impossible.
• Maintaining and monitoring access: with increasing staff turnover rates and organisational change (both internally and externally focused) it is difficult to manage, monitor and maintain access to systems for decades or longer. The task becomes much harder if those processes need to be individually managed across multiple systems. This can give rise to inconsistent practices, slow or delayed access and audit trail gaps.
• Cost of legacy/archived systems: although it may seem easier to leave data where it is, if that data is not currently in use, then you are likely paying much more individually for separate systems (i.e. paying for “archived” data in 10 systems vs. one consolidated repository). It is also likely that you no longer require the full functionality provided by that system, and therefore still paying for functionality you no longer use.
• Risk of vendor lock-in: having multiple systems provided by various third-party vendors increases the risk of vendor lock-in, particularly if those systems have been designed to make it difficult to export data from them. This challenge only increases in difficulty with time as vendor relationships change, newer systems are released/updated or even the supplier themselves are acquired. If retaining data for decades or longer, it is likely unavoidable that you will need to move that data at some point during that period. When that need does arrive, you don’t want to find that data locked in that system.

The above challenges can be largely overcome or at least significantly reduced if data is consolidated into a single long term repository. I’ll explore why this is in the next section.

A Consolidated Digital Archive

Consolidating data into a single long term repository (or archive) makes it much easier to manage records and data. Preserving data, providing access, monitoring usage and everything else that is required of retained data is much simpler to achieve (and less risky) if conducted in one system vs. ten (or more). In short, every challenge outlined in the previous section is much easier to manage in one system instead of many.

And by that same logic, there are also significant savings to be made for organisations by paying for one system vs. many. This does not even take into account the internal resource cost for managing ten systems vs. one.

Additionally, when looking for a long term home for your data, it is much easier to assess the suitability of one system, instead of assessing each source system individually. This importantly includes how portable the data is from that system, significantly reducing the risk of lock-in, and enabling you to move that data on in future if required.

Consolidation over Sprawl = Compliance

At the beginning of this blog post I referenced the level of adoption of technology within the clinical trial space; for organisations required to comply with long term retention regulations, I would argue a consolidated archive is essential.

Complying with regulations such as EU CTR (which requires sponsors to records and data are accessible, available and legible +25 years) and alignment with industry good practice (ALCOA+) is significantly easier if managed within one system. Not only can that system be selected to provide the capability to retain records, but only one corresponding set of processes (SOP) is required. Add in the possible cost and resource savings, there is an extremely strong case for consolidation.

To finish, I want to mention a final upside of consolidation; the value of intelligently leveraging large datasets. With the rise of AI and other powerful analytical tools, consolidating rich and well curated records and data offers potentially huge rewards. These new tools though, are only as good as the data they are built on, and so to fully realise their potential, consolidation is crucial.

If you’d like to discuss any of the topics I’ve raised in this post, do not hesitate to contact me (and the wider Arkivum team) via our contact us page.