Archiving GxP ‘Dynamic Data’

The Challenges of Dynamic Data

‘Dynamic Data’ presents plenty of interesting digital preservation challenges, especially in the context of long-term retention under GxP regulations and guidelines.  Dynamic data means “electronic records that allow for an interactive relationship between the user and the record content” (more on the definitions of dynamic data a little bit later).

The key thing is the ‘interactive relationship’ between user and data, which means the ability for a user to query, search, filter, visualise and analyse the underlying data so it is presented in a meaningful way and most importantly can be used answer questions that the user has of the data.

In GxP archiving contexts, these questions could be whether clinical data has been contemporaneously captured with consistent chronology, whether statistics data is complete and correctly coded, whether environmental conditions have been properly maintained during manufacturing, whether CAPAs have been managed in a timely way, whether staff have been correctly trained, whether access to systems has been controlled and many, many, more.

The ‘dynamic data’ that supports these questions can be very varied, for example data coming from site loggers, lab instruments, audit trails, clinical EDC, supporting records in a CTMS, TMF, LIMS, QMS, LMS and more.  The format of the data can be varied too, for example spreadsheets, statistical data, log files, relational databases and data in various industry formats, both open and proprietary.

The one thing in common is that converting the data into a static format just won’t cut it, for example ‘printing to PDF’ or presenting the data in its raw form with no way to interact with it.  The user needs to be able to engage with the data, visualise the data, ask questions of the data.  This is what dynamic means.

In some cases, data will be in a standardised format with readily available viewers or analysis tools so its dynamic nature can be maintained.  DICOM imaging is an example.  In some cases, the data is so specific and so closely linked to an application that there is little choice but to keep both the original data and original software together.  Proprietary instruments are an example.  But there is also a large class of applications where the data exists somewhere ‘inside the application’ in the form of tables, databases, log files and the like.  The application itself provides the way to query the data through a UI, for example by generating reports, charts, graphs and dashboards.

It’s this class of application that creates a very real and very large headache.  When maintaining dynamic data for long periods of time, for example 25 years or more, trying to retain the original data in the original systems with the original user interfaces typically won’t work.  Even if it were possible to keep all those legacy systems running for decades, the numbers of systems would build up and up and up over time. Someone would need to know how to use hundreds if not thousands of different user interfaces to access and interpret the original data. I wouldn’t want to be the archivist saddled with that problem!

Migrating legacy data into new live systems isn’t an easy solution either.  Repeated migrations will be needed over multi-decade timescales.  Repeatedly carrying legacy data over to new systems adds complexity, risk and cost.   And then storing old data in live systems means a need to keep detailed track of what is where and the need for tight access control to ensure legacy data can’t be altered in any way.

Rather than be a harbinger of doom, there are solutions to this problem!  One approach is to apply business intelligence and data analytics techniques directly to archived data within the archiving environment.  Bring the dynamics to the data if you like.

But before we get to that, what do the regulators expect?

What the GxP Regulations and Guidelines Say

The regulations and guidelines describe the dynamic data problem in different ways. Sometimes dynamic data is explicitly referenced.  For example, OECD GLP No 21 says “Records in dynamic format, such as electronic records, allow an interactive relationship between the user and the record content.” and “Information that is captured in a dynamic state should remain available in that state.”

The FDA guidelines for CGMP has a similar definition and provides some examples such as “electronic records from certain types of laboratory instruments—whether stand-alone or networked—are dynamic, and a printout or a static record does not preserve the dynamic record format that is part of the complete original record.” and lists many types of data that will likely fall into this category, for example “laboratory records, which includes but is not limited to notebooks, worksheets, graphs, charts, spectra, and other types of data from laboratory instruments”.

The MHRA goes further and talks about the requirements for dynamic data as part of Data Integrity across all GxP areas: “Data must be retained in a dynamic form where this is critical to its integrity or later verification.”.

But in other cases, dynamic data is implied rather than explicitly stated.  For example, under ICH E6(R3) essential records for GCP includes “Data and relevant metadata (including documentation of data corrections) in the data acquisition tools” and “trial participants’ medical/health records/notes/charts; data provided/entered by trial participants (e.g., electronic patient-reported outcomes (ePROs)); healthcare professionals’ records from pharmacies, laboratories and other facilities involved in the clinical trial; and data from automated instruments, such as wearables and sensors.”.

This data needs to remain readable: “The sponsor and investigator/institution should retain the essential records in a way that ensures that they remain complete, readable and readily available and are directly accessible upon request by regulatory authorities, monitors and auditors.”  Readable in this context means preserving Data Integrity and that can only be done if data that is dynamic remains dynamic.

Bringing The Dynamic to The Data

It would be great if there was a way to archive dynamic data from multiple sources in a manageable way that maintains its dynamic nature over time.

Suppose we could consolidate dynamic data from lots of different sources directly into a single archive environment where it can be queried and analysed in a consistent and flexible way.   Ideally, we would like to do this in a way that allows a wide range of questions to be asked of a wide range of data types, but using a single user interface that only needs to be learnt once.  One user interface, one process, one system to validate.  No recommissioning of old systems and no export and migration of data out of the archive into external tools.

This is where business intelligence and data analytics tools come in.  These tools are designed to allow users to interrogate lots of data sources in a range of ways and to build reports, charts graphs and other visualisations.  These tools have been developed over decades to allow users to look for trends or anomalies in data, to slice and dice data, to generate reports and charts and graphs.   They are built to answer just the sort of questions that an organisation or inspector might have of GxP dynamic data.

So what we want is to integrate and data analytics and business intelligence into an archive environment. This allows ‘in-situ’ analytics of archived data without the hassle and complexity of having to export the data and load it into separate systems.  Data remains dynamic and the organisation remains compliant.

And this is exactly the approach we’ve taken with the Arkivum solution. If you’d like to know more, then please join our webinar on Making GxP data dynamic within the Arkivum system!