/
AI PRODUCT EVALUATION & LLM TESTING AGENCY
AI Product Evaluation
We're the reason your AI product monetizes & delivers outputs that increasingly get better.
Why teams choose PH1
/01
Supercharge your next AI & CX strategy
We've worked with the most innovative orgs in the world and will bring their lessons and methods to your project.
/02
Answer the questions that AI can't
Even though it's easier than ever to build, the pressure to build the right thing is hotter than ever. That's what we answer.
/03
Solve the limitations of evals & telemetry
Our work doesn't compete with evals & telemetry, it makes them more effective by gaining clarity on failure modes & golden data sets.
The research & strategy partner for teams building intelligent experiences
/
FOUNDED IN 2012 /FROM STARTUPS TO THE FORTUNE 500
TRUSTED BY 300+ COMPANIES
/
AI EVALS & LLM TESTING CONSULTANCY
We're the forward-deployed AI strategist who gets measured against outcomes.
Product, digital, and AI leaders hire us to:
Benchmark an AI product from the customer’s side.
Set the AI strategy for a product or business unit.
Guide a major AI transformation programme.
Define and fine-tune outputs that customers will pay for.
Measure AI adoption and the value it created.
Add clarity to evals and telemetry that missing what matters.
/
EVALS · GOLDEN DATA SETS · FAILURE MODES · TRUST
Prove your AI investment & AI strategy are working and what to improve.
Engineering evals measure output against test cases, which is necessary and not sufficient. What decides whether an AI product succeeds is what happens when a person with their own goal uses that output inside a real workflow. We run behavioural science and model evaluation together, so we can tell you whether the problem sits in the model, the experience or the go-to-market. PH1 has deep expertise across AI evals, trust and confidence research, golden data sets, failure mode research and adoption measurement.
Everything we build is yours to keep running: eval sets, golden data sets and failure catalogues, delivered for your pipelines rather than for a slide.
Golden Dataset Research
An explicit definition of a successful output
AI Failure Mode Research
The exact prompts and outputs that risk your brand
AI Adoption & ROI Measurement
Value created, workflows changed, investment benchmarked
Measurement Frameworks
The KPIs that prove an initiative worked
Feature Impact Evaluation
Whether the release changed anything that matters
AI Evals & Output Fine-tuning
Multi-turn evals built on real customer tasks
AI Trust & Confidence Research
The barriers limiting adoption, weighted by segment
Generative AI Product Design
Design AI features people trust enough to rely on
AI Adoption Strategy
The roadmap for scaling adoption and value
Product Impact Podcast
Secrets to unlocking the value of AI
Our podcast, started in 2023, is the ultimate hub for learning from the leading voices in AI. We focus on delivering essential lessons for product teams determined to stay ahead of their competition.
/
BLOG
Latest insights
digital-transformation
/
Jul 9, 2026
Search: The Digital Transformation Project Quietly Hurting Your Customers
Broken site and enterprise search quietly erodes perception, acquisition, retention, and revenue. See why it's the transformation fix to prioritize now.
digital-transformation
/
May 23, 2026
AI Will Destroy University Website Traffic. There's Only One Path Forward.
Google's AI is replacing your website in the applicant journey. Here's how university marketing teams should respond before it's too late.
/
BLOG
Latest insights
digital-transformation
/
Jul 9, 2026
Search: The Digital Transformation Project Quietly Hurting Your Customers
Broken site and enterprise search quietly erodes perception, acquisition, retention, and revenue. See why it's the transformation fix to prioritize now.
digital-transformation
/
May 23, 2026
AI Will Destroy University Website Traffic. There's Only One Path Forward.
Google's AI is replacing your website in the applicant journey. Here's how university marketing teams should respond before it's too late.
/
BLOG
Latest insights
digital-transformation
/
Jul 9, 2026
Search: The Digital Transformation Project Quietly Hurting Your Customers
Broken site and enterprise search quietly erodes perception, acquisition, retention, and revenue. See why it's the transformation fix to prioritize now.
digital-transformation
/
May 23, 2026
AI Will Destroy University Website Traffic. There's Only One Path Forward.
Google's AI is replacing your website in the applicant journey. Here's how university marketing teams should respond before it's too late.
/
FAQ
Frequently Asked Questions
What is AI product evaluation and how does it improve my product?
AI product evaluation is structured testing of whether an AI feature works the way people actually use it. It measures full workflows rather than single prompts and scores output against a defined standard of correct. It improves the product by returning a prioritised list of what to fix, ranked by what each problem costs you, plus an eval set that catches the same problem recurring after you have fixed it.
What is the difference between model evals and product evals, and do I need both?
Model evals measure output against test cases: accuracy, latency, cost, refusal rates. Product evals measure what happens when a person with their own goal and their own tolerance for error uses that output inside a real workflow. You need both. The first tells you the model performed; the second tells you the customer got what they came for, and most failures live in the handoff between them.
What is a golden dataset and why does my AI product need one?
A golden dataset is an explicit, documented definition of what a successful output looks like for each use case, assembled as data you test against. Your product needs one because without it quality arguments are a matter of taste and no eval score means anything. It has to be built with your product team, your engineers and your customers, not by the model team alone.
What is LLM-as-a-judge and can it replace testing with real users?
LLM-as-a-judge uses a model to score another model’s output at volume. It works once you have defined what good looks like, and we use it. It cannot replace user testing, because it cannot produce that definition and it cannot see the moment a person quietly stops trusting the output. A model judging a model returns the consensus view of quality, which is the same view your competitors get from the same models.
What is AI failure mode analysis and which failures actually threaten my brand?
Failure mode analysis finds the specific prompts, inputs and outputs where an AI product fails, through adversarial and edge-case testing across real workflows. The ones that threaten your brand are rarely the rarest. They are the failures that happen in front of customers, in high-volume moments, in ways people screenshot. We catalogue and rank by frequency and brand risk so guardrails get built for those first.
How do I test my AI product with real users before launch?
Multi-turn sessions across the whole workflow, with participants pursuing their own goals rather than scripted tasks. Before launch is the right time, because evaluation then changes what you build, while evaluation afterwards produces a list of changes that are expensive once the architecture has set. Participants also tell an external researcher things they will not say to the team that built the feature.
How do I measure AI adoption when licence usage already looks healthy?
By benchmarking against workflows rather than against licence counts. Licence utilisation looks healthy in plenty of organisations where nothing about the work has changed. We measure which teams and tools are genuinely in use, which workflows changed, which did not, and where adoption stalled, which is the gap between a seat being paid for and a job being done differently.
How do I measure ROI on our AI investment and prove it to finance?
By defining the outcome before measuring the usage. We establish what the investment was meant to change, measure value created against what was spent, and report which workflows moved. The output is written for the people who approved the budget rather than for a technical audience, which is usually the difference between a number finance accepts and one it questions.
How do I benchmark our AI product against competitors?
Identical tasks and the same participants across every product in the study, scored on output quality, workflow completion and user confidence. Comparable by construction rather than by assertion, with the methodology documented so your team can rerun it next year and show movement.
Should we evaluate our AI in-house or hire a partner?
In-house evaluation is necessary and has one structural weakness: your team is measuring something it built, and users are reliably kinder about a feature to the person who shipped it. An external partner gets the version they will not say internally, and brings the customer-definition step most internal programmes skip. The two work best together.
What should I look for when choosing an AI evaluation partner?
Three things. Whether they do behavioural research as well as model evaluation, since most failures sit in the handoff between them. Whether you keep the eval sets and data they build. And whether their knowledge is current, because this field changes monthly. We publish on it ourselves through the Product Impact podcast and our AI value research, which is how we stay in the conversation rather than remembering a project from two years ago.
Can you evaluate an unreleased AI feature without it leaking?
Yes. Sessions run under NDA, participants are screened and briefed without your name attached where that matters, and an unreleased capability can be evaluated without your brand on it. Being external is what makes that possible.
What AI products has PH1 evaluated?
AI website strategy for GMR, delivered through DotDotDash, reframing an AI-powered website brief into a qualified-lead strategy with a three-phase roadmap. Evaluation of AI features for an emerging device category at Microsoft. AI product evaluation for Search Atlas. AI product evaluation, prototyping and benchmarking for The Weather Network.
How long does an AI product evaluation take and what does it cost?
A diagnostic can start within a week and runs about a week, because it needs your product and your team rather than recruited participants. Full eval programmes with customer sessions start within two to three weeks. If the project is a rush, we can get you initial insights within 10 days. Cost scales with use cases, workflows and segments, and each expertise page carries its own range.
/
FAQ
Frequently Asked Questions
What is AI product evaluation and how does it improve my product?
AI product evaluation is structured testing of whether an AI feature works the way people actually use it. It measures full workflows rather than single prompts and scores output against a defined standard of correct. It improves the product by returning a prioritised list of what to fix, ranked by what each problem costs you, plus an eval set that catches the same problem recurring after you have fixed it.
What is the difference between model evals and product evals, and do I need both?
Model evals measure output against test cases: accuracy, latency, cost, refusal rates. Product evals measure what happens when a person with their own goal and their own tolerance for error uses that output inside a real workflow. You need both. The first tells you the model performed; the second tells you the customer got what they came for, and most failures live in the handoff between them.
What is a golden dataset and why does my AI product need one?
A golden dataset is an explicit, documented definition of what a successful output looks like for each use case, assembled as data you test against. Your product needs one because without it quality arguments are a matter of taste and no eval score means anything. It has to be built with your product team, your engineers and your customers, not by the model team alone.
What is LLM-as-a-judge and can it replace testing with real users?
LLM-as-a-judge uses a model to score another model’s output at volume. It works once you have defined what good looks like, and we use it. It cannot replace user testing, because it cannot produce that definition and it cannot see the moment a person quietly stops trusting the output. A model judging a model returns the consensus view of quality, which is the same view your competitors get from the same models.
What is AI failure mode analysis and which failures actually threaten my brand?
Failure mode analysis finds the specific prompts, inputs and outputs where an AI product fails, through adversarial and edge-case testing across real workflows. The ones that threaten your brand are rarely the rarest. They are the failures that happen in front of customers, in high-volume moments, in ways people screenshot. We catalogue and rank by frequency and brand risk so guardrails get built for those first.
How do I test my AI product with real users before launch?
Multi-turn sessions across the whole workflow, with participants pursuing their own goals rather than scripted tasks. Before launch is the right time, because evaluation then changes what you build, while evaluation afterwards produces a list of changes that are expensive once the architecture has set. Participants also tell an external researcher things they will not say to the team that built the feature.
How do I measure AI adoption when licence usage already looks healthy?
By benchmarking against workflows rather than against licence counts. Licence utilisation looks healthy in plenty of organisations where nothing about the work has changed. We measure which teams and tools are genuinely in use, which workflows changed, which did not, and where adoption stalled, which is the gap between a seat being paid for and a job being done differently.
How do I measure ROI on our AI investment and prove it to finance?
By defining the outcome before measuring the usage. We establish what the investment was meant to change, measure value created against what was spent, and report which workflows moved. The output is written for the people who approved the budget rather than for a technical audience, which is usually the difference between a number finance accepts and one it questions.
How do I benchmark our AI product against competitors?
Identical tasks and the same participants across every product in the study, scored on output quality, workflow completion and user confidence. Comparable by construction rather than by assertion, with the methodology documented so your team can rerun it next year and show movement.
Should we evaluate our AI in-house or hire a partner?
In-house evaluation is necessary and has one structural weakness: your team is measuring something it built, and users are reliably kinder about a feature to the person who shipped it. An external partner gets the version they will not say internally, and brings the customer-definition step most internal programmes skip. The two work best together.
What should I look for when choosing an AI evaluation partner?
Three things. Whether they do behavioural research as well as model evaluation, since most failures sit in the handoff between them. Whether you keep the eval sets and data they build. And whether their knowledge is current, because this field changes monthly. We publish on it ourselves through the Product Impact podcast and our AI value research, which is how we stay in the conversation rather than remembering a project from two years ago.
Can you evaluate an unreleased AI feature without it leaking?
Yes. Sessions run under NDA, participants are screened and briefed without your name attached where that matters, and an unreleased capability can be evaluated without your brand on it. Being external is what makes that possible.
What AI products has PH1 evaluated?
AI website strategy for GMR, delivered through DotDotDash, reframing an AI-powered website brief into a qualified-lead strategy with a three-phase roadmap. Evaluation of AI features for an emerging device category at Microsoft. AI product evaluation for Search Atlas. AI product evaluation, prototyping and benchmarking for The Weather Network.
How long does an AI product evaluation take and what does it cost?
A diagnostic can start within a week and runs about a week, because it needs your product and your team rather than recruited participants. Full eval programmes with customer sessions start within two to three weeks. If the project is a rush, we can get you initial insights within 10 days. Cost scales with use cases, workflows and segments, and each expertise page carries its own range.
/
FAQ
Frequently Asked Questions
What is AI product evaluation and how does it improve my product?
AI product evaluation is structured testing of whether an AI feature works the way people actually use it. It measures full workflows rather than single prompts and scores output against a defined standard of correct. It improves the product by returning a prioritised list of what to fix, ranked by what each problem costs you, plus an eval set that catches the same problem recurring after you have fixed it.
What is the difference between model evals and product evals, and do I need both?
Model evals measure output against test cases: accuracy, latency, cost, refusal rates. Product evals measure what happens when a person with their own goal and their own tolerance for error uses that output inside a real workflow. You need both. The first tells you the model performed; the second tells you the customer got what they came for, and most failures live in the handoff between them.
What is a golden dataset and why does my AI product need one?
A golden dataset is an explicit, documented definition of what a successful output looks like for each use case, assembled as data you test against. Your product needs one because without it quality arguments are a matter of taste and no eval score means anything. It has to be built with your product team, your engineers and your customers, not by the model team alone.
What is LLM-as-a-judge and can it replace testing with real users?
LLM-as-a-judge uses a model to score another model’s output at volume. It works once you have defined what good looks like, and we use it. It cannot replace user testing, because it cannot produce that definition and it cannot see the moment a person quietly stops trusting the output. A model judging a model returns the consensus view of quality, which is the same view your competitors get from the same models.
What is AI failure mode analysis and which failures actually threaten my brand?
Failure mode analysis finds the specific prompts, inputs and outputs where an AI product fails, through adversarial and edge-case testing across real workflows. The ones that threaten your brand are rarely the rarest. They are the failures that happen in front of customers, in high-volume moments, in ways people screenshot. We catalogue and rank by frequency and brand risk so guardrails get built for those first.
How do I test my AI product with real users before launch?
Multi-turn sessions across the whole workflow, with participants pursuing their own goals rather than scripted tasks. Before launch is the right time, because evaluation then changes what you build, while evaluation afterwards produces a list of changes that are expensive once the architecture has set. Participants also tell an external researcher things they will not say to the team that built the feature.
How do I measure AI adoption when licence usage already looks healthy?
By benchmarking against workflows rather than against licence counts. Licence utilisation looks healthy in plenty of organisations where nothing about the work has changed. We measure which teams and tools are genuinely in use, which workflows changed, which did not, and where adoption stalled, which is the gap between a seat being paid for and a job being done differently.
How do I measure ROI on our AI investment and prove it to finance?
By defining the outcome before measuring the usage. We establish what the investment was meant to change, measure value created against what was spent, and report which workflows moved. The output is written for the people who approved the budget rather than for a technical audience, which is usually the difference between a number finance accepts and one it questions.
How do I benchmark our AI product against competitors?
Identical tasks and the same participants across every product in the study, scored on output quality, workflow completion and user confidence. Comparable by construction rather than by assertion, with the methodology documented so your team can rerun it next year and show movement.
Should we evaluate our AI in-house or hire a partner?
In-house evaluation is necessary and has one structural weakness: your team is measuring something it built, and users are reliably kinder about a feature to the person who shipped it. An external partner gets the version they will not say internally, and brings the customer-definition step most internal programmes skip. The two work best together.
What should I look for when choosing an AI evaluation partner?
Three things. Whether they do behavioural research as well as model evaluation, since most failures sit in the handoff between them. Whether you keep the eval sets and data they build. And whether their knowledge is current, because this field changes monthly. We publish on it ourselves through the Product Impact podcast and our AI value research, which is how we stay in the conversation rather than remembering a project from two years ago.
Can you evaluate an unreleased AI feature without it leaking?
Yes. Sessions run under NDA, participants are screened and briefed without your name attached where that matters, and an unreleased capability can be evaluated without your brand on it. Being external is what makes that possible.
What AI products has PH1 evaluated?
AI website strategy for GMR, delivered through DotDotDash, reframing an AI-powered website brief into a qualified-lead strategy with a three-phase roadmap. Evaluation of AI features for an emerging device category at Microsoft. AI product evaluation for Search Atlas. AI product evaluation, prototyping and benchmarking for The Weather Network.
How long does an AI product evaluation take and what does it cost?
A diagnostic can start within a week and runs about a week, because it needs your product and your team rather than recruited participants. Full eval programmes with customer sessions start within two to three weeks. If the project is a rush, we can get you initial insights within 10 days. Cost scales with use cases, workflows and segments, and each expertise page carries its own range.