Testing Artificial Intelligence

Artificial Intelligence is constantly being redefined, annually and sometimes even more frequently. Our offering list in this space is constantly evolving- however we see 3 seperate core use cases for AI in business:

  1. Introduction of Large Language Models (such as ChatGPT, CoPilot etc.)

  2. Modular/bespoke libraries such as those produced and provided by Azure, AWS and GCP

  3. Open Source / Bespoke (from the ground up) models designed and built for specific problems.

  • LLM's come in all shapes and sizes. Depending on what you are building or employing, we can help you test for bias, guardrails, accuracy and relevance and supporting infrastructure such as auth and access control systems.

  • Azure, Amazon Web Services and Google Cloud Platform all provide API bound AI services that can be used in isolation or as chained AI solutions. We test these in many different ways depending on the module and what it is intended to do. Some only require data and expectation validation while others may require testing to 10 decimal places in precision.

  • AI solutions do not exist in a vacuum, they operate in complex, hetrogenous environments. Our testers will test that the AI system correct behaves with peered and downstream/upstream systems to ensure that the outcomes that are expected are realised.

  • If you are building your own AI solution, we can help test your implementation to ensure your chosen algorithms are producing the correct output so predictions and inferences derived from your data have a high trustworthiness.

  • PTP can help you by reviewing and determining any challenges in your AI policies. We can identify situations where your IP (or other valuable data) could leak out of your organisation, determine loop holes that will allow employees or partners to behave poorly. This is an emergent field and will constantly be changing over the next few years, but is here to stay.

Our Approach

  • Set specific goals for model accuracy, fairness, and reliability.

  • Use varied and representative data sets to evaluate model performance.

  • Continuously monitor AI models for changes in performance over time.

  • Make AI models transparent and understandable to stakeholders.

  • Regularly test for and mitigate bias in AI models.