Knowing that need to get the data right.

Data Testing is the crucial art of testing that data is valid, correct and complete. It may be that the data is moving from one system to another as part of a migration, or the original collection and storage of the data, or perhaps interpretation and reporting on this data. Data has a lifecycle and we test data throughout all parts of its life including its ultimate archival and/or deletion.

  • Data migration is normally performed as part of system transformation. Testing these involves ensuring the extract process is successful, the mapping to business domain objects is complete and accurate and interpretation, mapping and loading into the target system. Once all of this is complete, the old capability needs to work in the new system. New processes may need to be verified as operable end to end.

  • Data is easily corrupted, duplicated and misattributed. When these events occur, discovery is seldom automatic, usually requiring human discovery, either through consistency checks or automation.

  • Databases, data files and unstructured files all can have problems- such as missing data, inconsistent formatting and constraints. This type of testing involves reporting on issues found, making recommendations on remediation and perhaps even correcting issues found.

  • Extract Transform and Load is the standard method for data migration. Testers play a very active role testing the extract process, often scripting much of the extract process and testing that the transform is working correctly.

  • When ETL isn't used, frequently transforms happen by streaming data from one API, transforming in situ then loading into the target system. This process needs to be tested much like with ETL, except it is done during transform, whereby issues found are logged so they can be corrected with the process and automation later.

Our Approach

  • Test data quality and integrity at the beginning of the data lifecycle.

  • Implement automated checks for data consistency and accuracy.

  • Comply with data protection regulations like GDPR during testing.

  • Verify that data transformations produce expected results without loss or corruption.

  • Implement ongoing data quality assessments to catch issues early.