Planning under uncertainty
AI-guided design of efficient clinical trials with BATON
Naim Rashid, PhD
Associate Professor, Department of Biostatistics, Gillings School of Global Public Health Member, Lineberger Comprehensive Cancer Center · UNC Chapel Hill
NextGen TriBix Symposium · November 17, 2026
Pull-from-if-needed: The FDA sotorasib review describes the postmarketing comparison.
Q&A pockets: What is AI-guided here? A statistical model guides which candidate design to simulate next, before the trial enrolls.
Delivery: Open with “A cancer drug was approved at 960 mg. What would you want to know before deciding whether 240 mg was a better choice?” Pause for a moment. “I’ll come back to that drug. First, let’s look at the choices early trials have to make.”
1 The stakes
What do we need to learn before committing?
2 The design problem
3 BATON
4 BATON in practice
5 Looking to the future
Drug development usually follows three phases before approval
Phase I
First trials in people. Is it safe, and at what dose?
→
Phase II
Is there a sign that it works?
→
Phase III
Often the pivotal trial. Is there convincing evidence of benefit?
→
Approval
Fewer candidates continue. Schematic.
Phases can overlap or be combined.
Where a Phase II–III gap exists, it averages 17 months.
Tufts CSDD · non-expedited approvals, 2008 to 2018 (Getz 2021)
Say aloud: “Drug development is commonly organized into three phases before approval. Phase I is the first trials in people. They are small and ask whether the drug is safe and at what dose. Phase II asks whether there is a sign that it works. Phase III typically provides pivotal evidence, the main evidence used to judge whether a treatment works. It also adds information on safety. Approval decisions depend on the total evidence. The phases can overlap or be combined. The design question is what evidence we need before making each decision.”
Pull-from-if-needed: The 10.5 years on slide 5 runs from the start of phase I to approval. BIO 2021 splits it as 2.3 years in phase I, 3.6 in phase II, 3.3 in phase III and 1.3 in regulatory review (VERIFIED row of tribix-opening-facts.md).
Pull-from-if-needed (the gap): Tufts Center for the Study of Drug Development, reported by Getz, Applied Clinical Trials, April 2021 . Among non-expedited drugs and biologics approved from 2008 to 2018, 82% had a gap between Phase II completion and the start of the pivotal Phase III trial, and that gap averaged 17 months. The other 18% overlapped. The mean was 12 months for 2008 to 2013 and 21.6 months for 2014 to 2018. These are means, no median is reported, and the source is a trade publication. The same analysis could not measure the Phase I to Phase II gap reliably, so the slide gives none.
Q&A pockets: Is it always three separate phases? ICH E8(R1), Section 4.3 describes phases as an organizing convention and explains that they can overlap or be combined. Section 5 returns to how the final claim determines what earlier studies need to learn.
Pull-from-if-needed (the band): The narrowing band only says that fewer candidates continue at each step. Its widths carry no proportions. If a rate is wanted, use the BIO 2021 figure in the slide 5 notes.
Delivery: One beat per box. Then say “Start with Phase I. Where those first trials happen is getting attention right now.”
The U.S. is concerned about its lead in early clinical research
Pull-from-if-needed: HHS’s Operation TrialBlazer roadmap is the source for the US leadership concern and the relative-share statement. The underlying series has not been independently verified. The Hitchen et al. chart counts adult solid-tumor Phase I trials with an Australian site in two years. See the verified entries in tribix-opening-facts.md. Page 6 of the HHS roadmap discusses alternative trial designs and recruitment.
Q&A pockets: Why do sponsors go abroad? Industry testimony to Congress cites cost and speed to a first signal. These slides describe two observations and do not estimate their causes.
Delivery: About 35 seconds. Attribute the U.S. concern to HHS and give the Australian counts as a separate observation. Then say “That’s one pressure on the earliest stage. Zoom out to the full path from the first trial in people to approval.”
Developing one new therapy is slow and expensive
10.5 years from Phase I to approval, based on successful transitions
$1.1 billion median cost per approved drug, failed candidates included (2018 dollars)
Each year of development is a year patients wait.
BIO, Informa, QLS Advisors 2021 · Wouters, McKee, Luyten, JAMA 2020, corrected 2022
Sources (VERIFIED rows of tribix-opening-facts.md): (1) Biotechnology Innovation Organization, Informa Pharma Intelligence, QLS Advisors. Clinical Development Success Rates and Contributing Factors 2011 to 2020. February 2021. https://go.bio.org/rs/490-EHZ-999/images/ClinicalDevelopmentSuccessRates2011_2020.pdf. Mean 10.5 years from phase I to approval, based on 6,151 successful phase transitions, 2011 to 2020 (2.3 years phase I, 3.6 phase II, 3.3 phase III, 1.3 regulatory). Oncology 10.3 years. Trade-association report, not peer reviewed. (2) Wouters OJ, McKee M, Luyten J. Estimated Research and Development Investment Needed to Bring a New Medicine to Market, 2009-2018. JAMA. 2020;323(9):844-853. doi:10.1001/jama.2020.1166, corrected 20 September 2022 (JAMA. 2022;328(11):1110). Median capitalized R&D investment $1141.7 million (95% CI 888.1 to 1480.8), 2018 dollars, failed trials included, 63 of 355 drugs approved 2009 to 2018 with public R&D data.
Say aloud: “A drug that makes it from its first trial in people to approval takes about ten and a half years on average. Each approved drug costs a median of about 1.1 billion dollars once the failures are counted.”
Pull-from-if-needed: Use 1.14 billion if pressed, never the 985 million in the uncorrected abstract. For cancer drugs the same paper gives a median of about 2.77 billion, from a small subgroup with a wide interval. If a success rate is wanted, about 5% of oncology drugs entering phase I reach approval (5.3%, BIO 2021). These are the public concerns behind current federal attention to clinical development, time, cost, early trials moving abroad, and patients waiting. The previous slide names Operation TrialBlazer.
Q&A pockets: Does BATON shorten this? No result says so. This talk is about one part of the problem, the design of each trial, and the close says which part. Those cost numbers are contested. Yes. The sample skews to smaller firms and to products with public filings.
Delivery: Say each number with its denominator. Then say “Even after a drug has made it through the early studies, the biggest test can still be ahead.”
Many problems become apparent only in late-stage trials
54%
failed in late-stage development 640 therapeutics that entered pivotal trials in 1998–2008
Most often the drug did not show enough benefit (57% of failures).
Hwang et al., JAMA Internal Medicine 2016
What can we do in the early trials to make a late failure less likely?
Late failures have many causes. Earlier-phase trials can also affect later-phase trials in ways that are hard to predict.
Pull-from-if-needed: Hwang et al. tracked 640 therapeutics entering phase 3 or other pivotal trials from 1998 through 2008. The study reported inadequate efficacy as the reason for 57% of the failures.
Q&A pockets: Is this a current rate? It describes this historical cohort. What do the dots represent? Each dot represents one percentage point.
Delivery: Give the historical denominator and the result. Some late failures are the right conclusion. Name the other causes in one sentence (the wrong target, the wrong patients, the wrong dose, an early endpoint that did not predict the later one), then say that early trial design is the one this talk addresses. Now ask “What can we do in the early trials to make a late failure less likely? Let’s go back to the 960 mg decision from the opening.”
Early decisions become downstream commitments
Sotorasib · KRAS G12C lung cancer
Phase I 960 mg Highest tested
→
Accelerated approval 960 mg
→
Postapproval comparison
960 mg vs 240 mg
FDA-required
Progression-free survival
5.4 months
5.6 months
Severe side effects (grade 3+)
35.6%
19.2%
Tumor response
32.7%
24.8%
The highest dose patients tolerate, or the lowest dose with full benefit?
Pull-from-if-needed: The FDA review and FDA authors’ retrospective describe the accelerated approval and dose comparison. The randomized study reports the progression-free survival and tumor response estimates. The ODAC briefing, Table 29 reports grade 3 or higher treatment-related adverse events (37 of 104 at 960 mg and 20 of 104 at 240 mg). The Phase II comparison was descriptive and was not powered for formal hypothesis testing. Collins et al. counted a broad range of postmarketing requirements for additional dosing information, including interactions, dosing in specific groups, and dose comparisons.
Pull-from-if-needed (the two dose targets): The maximum tolerated dose is the highest dose with acceptable side effects, the usual target of a traditional Phase I escalation. The optimal biological dose is the lowest dose that gives the full benefit. Sotorasib’s 960 mg was the highest of four doses tested (180, 360, 720 and 960 mg). The Phase I report recorded no dose-limiting toxic effects, so do not call 960 mg the maximum tolerated dose (Hong et al., NEJM 2020 ). FDA’s Project Optimus is the public statement of the concern. The link to Phase III is a rationale and not a measured effect. Too high a dose brings side effects, dose reductions and patients stopping treatment, which may blunt the benefit a pivotal trial sees. No study I know of shows that choosing the optimal biological dose raises Phase III success.
Q&A pockets: Did the comparison lead FDA to change the dose? The FDA authors report that the totality of the evidence did not lead to a dose change.
Delivery: Name sotorasib as the drug from the opening. Show the Phase I dose and the accelerated approval, which lets FDA approve earlier based on an outcome expected to predict clinical benefit. Ask “What evidence would you want before carrying that dose into a larger trial?” Give the room a moment, then reveal the comparison requirement. Patients would be assigned to the two doses by chance. On the next click, show the study results. On the last click, name the two dose targets and say that the dose a program carries forward may affect what Phase III can show. Define progression-free survival as the time until the cancer worsens or the patient dies. The observed medians were close. The lower dose had fewer severe treatment-related adverse events, and the higher dose had a numerically higher tumor response rate. The study did not establish equivalence. Land on the timing: the balance between activity and side effects was still an open question at approval. Close with “What would we learn about 240 mg by enrolling more patients at 960 mg?” If time permits, state that 78 of 132 novel oncology drugs approved from 2012 to 2022 had a postmarketing requirement to collect additional dosing information.
How best to design early trials for later success is not yet clear
More patients at 960 mg cannot show how 240 mg compares.
Different measures
Separate designs Each phase has its own targets.
Too many possible plans Patients per phase, and when to move on.
More patients later may not answer what the early trial never asked.
Say aloud (the open question, PI 2026-11-14): “So why not just design early trials with the later ones in mind? Because it is not yet clear how best to do it. Early trials are small and read out short-term signals, tumor shrinkage and side effects, while later trials are judged on long-term outcomes such as survival, so the link from an early signal to late success is uncertain. Each phase is designed on its own, to its own targets, often by different teams. And the number of ways to divide patients and decisions across phases is too large for a formula. We come back to this at the end.”
Pull-from-if-needed: The schematic replaces the first box’s sentence, “Early signals are short-term. Later trials are judged on longer-term outcomes.” The trial sizes in it are not to scale. Sotorasib on the previous slide is one example of the first reason. The dose came from phase I, which reads short-term signals, and the later question was about longer-term benefit.
Q&A pockets: Has anyone solved this? Say that it is an open problem and that this talk does not claim a solution. No numbers here.
Delivery: Start with the question from the previous slide. More patients at 960 mg can improve our estimate for that dose. To learn how it compares with 240 mg, we may need to reopen the comparison. Say that more patients later cannot answer a question the early trial never asked, then say “If we want earlier trials to answer better questions, we need a way to test how their rules will behave before enrolling anyone.”
Better early trials are possible. Designing them is the hard part.
So far
10.5 years
from Phase I to approval
54%
of late-stage candidates failed
960 mg
chosen early, compared with 240 mg only after approval
Trials that adapt as the data arrive can learn more from fewer patients.
One bottleneck is finding an efficient trial design that still meets the evidence requirements. That is a search problem.
Delivery (section close): Recap the three numbers in one breath. Development takes about ten and a half years, more than half of late-stage candidates failed, and sotorasib’s dose was chosen early and compared only after approval. Then turn. Say aloud first: “We can change how a trial is designed. Each change gives us more choices about when to look, what to measure, and what evidence is enough to act.”
Say once, here: “When I say efficient, I mean efficient in patients, not faster computation.”
Say aloud (the plain glosses): “Adaptive designs can stop a trial early when the drug is not working, so fewer patients receive it. Platform designs test several drugs or subgroups under one protocol. Seamless designs move from screening to confirmation without starting a new trial. Each can improve how a team uses patients or time, depending on the rules and the setting.” The slide no longer lists these three designs, so name them aloud here. Say “Suppose we want to stop early when a treatment looks unpromising. When should we look, and how much evidence would be enough to stop? Each choice changes how the trial behaves. We can simulate one set of rules. Then we have to decide which set to try next.”
Pull-from-if-needed: Adaptive, platform and seamless designs are not new. The 2019 FDA adaptive-design guidance describes how sponsors can use simulations to evaluate complex adaptive designs, including their statistical operating characteristics. The bottleneck is the calibration work, which is what the rest of the talk is about.
Q&A pockets: If they are better, why are they not used everywhere? Design time and the expertise needed to calibrate them are part of the answer. Do not claim they are the only reason.
Delivery: Pause on the search question, then say “Let’s start with one Phase II decision. Has the treatment shown enough promise to continue?”
1 The stakes
2 The design problem
Is there enough evidence to advance the treatment?
3 BATON
4 BATON in practice
5 Looking to the future
A clinical trial is an experiment built for a decision
? Benchmark Treatment
The question
Does the treatment beat a benchmark or a control?
Single-arm all get the treatment Randomized treatment control
How patients are used
Everyone gets the treatment, or patients are assigned to treatment or control.
Does not work GO Works STOP
Two errors
Advance a treatment that fails, or drop one that works.
Two choices dominate: how many patients, and what evidence is enough to take the treatment to the next phase.
Say aloud (moved off the slide for the 60-word max sample size): “Traditional development is organized in phases: choose a dose, look for a signal, then confirm it. Each phase is a trial with its own rule for go or stop.” Then the caption: for the calibration problems in this talk, two choices dominate.
Glosses at first use: benchmark (the response rate the treatment has to beat, fixed before the trial), single-arm (everyone gets the study treatment; we compare with a prespecified benchmark), randomized (patients are assigned to treatment or control), objective (the quantity we try to make small, here patients). PFS, when it comes up: “time until the cancer worsens or the patient dies.”
Delivery: Start with “Most of the BATON examples today are Phase II screening designs. The decision is whether the treatment has shown enough promise to study further. We’ll follow one EVOLVE breast cancer cohort.” Lead with the two decisions at the bottom. Carry over the example from slide 9. “If we wanted to stop early, how many patients would we enroll before deciding, and what would count as enough evidence to continue?”
An adaptive trial can act on an interim result
When should we look, and what evidence is enough to keep enrolling?
Split from the merged slide (CompMed 5, now on its own again). Order: the figure, then the closing line, which is the CompMed 5 title.
Say aloud (the patient contrast, PI request 2026-11-14): “In a traditional fixed trial, every planned patient is enrolled and the results are looked at once, at the end. If the treatment is not working, every one of those patients still receives it, and we learn that only at the end. An adaptive trial looks at the accumulating results at planned points, called interim looks, and can stop early for futility, meaning the treatment looks unlikely to work. Then fewer patients receive a treatment that is not working, the answer can come sooner, and patients and resources are free for the next option.”
Say aloud (the qualifier, required): “Fixed designs are simpler and well understood. The adaptive benefit holds only if the rules are well chosen, and choosing them well is what the rest of this talk is about.”
Say aloud on the closing line: “The design is Bayesian because those decisions rest on posterior probabilities under a model and a prior.” Then, as the bridge to the next slide: “Maximum N, look timing, futility and efficacy thresholds, priors, margins. Every one is a setting, and every one changes how the trial behaves.”
Pull-from-if-needed: Where the benefit is counted. Expected patients enrolled when the treatment is inactive, on the EVOLVE objective slides (slides 26 and 27), is this slide’s benefit as a number. The 50 and 10 patients in the figure are schematic, not a real design. The precise contrast, per the paper: unlike classical group-sequential designs (Jennison and Turnbull), complex Bayesian adaptive trials generally lack closed-form operating characteristics (Berry 2010; Thall and Simon 1994). The flexibility comes from posterior-probability monitoring, and the cost is that neither formulas nor enumeration reach the behavior.
Q&A pocket: Are adaptive designs always better? No. They help only when the stopping rules are well chosen, which is the design problem this talk is about.
Delivery: 85 seconds. Figure first, top row then bottom row. Say that this example has one treatment group and that a look is a planned review of efficacy data. Futility means there is too little evidence of benefit to justify continuing under the rule.
A good design has to meet four requirements at once
Must meet
1
Rarely advance a treatment that fails
Type I error at most 10%
2
Usually advance one that works
Power at least 80%
Then improve
Feasible means meeting 1 and 2. Calibration then improves 3 and 4.
Delivery: These are the four requirements from the Lineberger symposium talk, reworded. The first two are fixed targets and every usable design has to meet them, which is what feasible means in the rest of the talk. The last two are what we try to improve among the feasible designs. Do not name EVOLVE yet, it is introduced two slides on.
Split from the merged slide (CompMed 6 title and boxes, with the CompMed 5 count of choices). The slide connects the two go rates to the EVOLVE screening example on the next slide.
Say aloud: Both conditions are required. A Phase II design in this EVOLVE example is feasible only if its power is at least 80% and its type I error is at most 10%. Say that the null scenario uses the benchmark response rate. The alternative scenario uses the prespecified response rate that would make the treatment worth pursuing. Type I error is the chance of a go under the null. Power is the chance of a go at that alternative. Then read the caption. Calibration means tuning the rules to meet both targets. These are planning targets for the EVOLVE screening example.
Say aloud (the CompMed 6 table, not shown): “A typical trial needs one design with a few parameters, and power and type I error often come from a formula. A platform needs many subtrial designs with many interacting parameters, and power and type I error come only from simulation. And we had to design repeatedly, on a clock.”
Q&A pocket: Bayesian rule, frequentist error rates? The monitoring rule is Bayesian (posterior probabilities drive stopping). The FDA describes how simulation can demonstrate the error rates of a complex adaptive design. Calibration is the work of tuning the rule to meet the targets.
Q&A pocket: Why 0.10 here? These are screening subtrials. The targets are specific to this example and depend on the trial’s purpose.
Delivery: 60 seconds. Do not read the boxes, define the two terms in one breath.
Simulation tests one design in two scenarios
One design, held fixed in every simulated trial
Fixed: settings, rules, model and assumptions. Only the treatment effect changes.
Ineffective (null)
target: at most 10%
2,100 of 50,000 say go
4.2% type I error
Effective (alternative)
target: at least 80%
41,000 of 50,000 say go
82% power
Illustrative numbers
Both targets met, so this design is feasible.
Pull-from-if-needed: This is the H0-Optimal design from EVOLVE’s triple-negative breast cancer example. The BATON 0.4.0 verification file reports 703 and 41,069 go decisions in 50,000 simulations per scenario and 11.22226 expected patients under the inactive scenario. See manuscript Table 3 and results/retune_xphil/final_summary_designs.csv, row R01. The FDA adaptive-design guidance, section VI.A describes using simulation to check operating characteristics, including error rates and power.
Q&A pockets: Why these two scenarios? The first estimates how often an inactive treatment would pass. The second estimates how often a treatment with the prespecified benefit would pass. What does 11.2 mean? Some simulated trials stop early and some enroll longer, so the average can be fractional.
Pull-from-if-needed (illustrative numbers, PI 2026-11-16): The 4.2% and 82% on this slide are illustrative and are labelled so on screen. They are not the EVOLVE design. The verified EVOLVE design appears on slide 26 with 1.4% type I error (703 of 50,000), 82.1% power (41,069 of 50,000) and 11.2 expected patients when ineffective. Do not describe the slide 14 numbers as results.
Delivery: Say what is held fixed before reading the bars. The candidate design, its decision rules, the analysis model and the trial assumptions (accrual, the outcome distribution, the benchmark) do not change. We run 50,000 simulated trials under the null, where the treatment is ineffective, and 50,000 under the alternative, where it has the prespecified benefit. Each bar is the share of 50,000 simulated trials that end in a go decision, and the dashed line is the target. After the bottom line, say “Now repeat this for every subtrial and every rule we might try.” Return to the EVOLVE Phase II cohort introduced at the start of the section. Use the same two scenarios from slide 13. Read the ineffective row first, then the effective row. Point out that the simulation assumes the effect in each row and estimates how often the trial reaches a go decision. Explain that this is one candidate, with a rule fixed before simulation. Reveal the enrollment outcome and ask whether another acceptable design might enroll fewer patients. Use the next slide to show why EVOLVE had to repeat this exercise. The verified EVOLVE design appears on slide 26.
One EVOLVE protocol can host 10 to 15 subtrials
EVOLVE is a platform trial. Each biomarker-defined group of patients gets its own subtrial.
Each subtrial needs its own decision rules, with eight or more interacting settings.
Pull-from-if-needed: The BATON 0.4.0 manuscript describes plans for 10 to 15 biomarker-defined subtrials. Each cohort is a group of patients studied in its own subtrial. The three cards give examples of trial architecture. The settings counts (8, 8 and 13) are the numbers of design parameters calibrated for each architecture in the BATON 0.4.0 manuscript and supplement, shown in backup B13.
Q&A pockets: Why can’t we reuse one design? Cohorts can differ in eligibility, expected outcomes, and available control data. Does classification uncertainty always lower power? It depends on who is misclassified and which decisions use the result. Check power, false-go rate, and enrollment under plausible scenarios.
Delivery: Say what EVOLVE is before the count. It is one master protocol in metastatic breast cancer, and each biomarker-defined group of patients is studied in its own subtrial. Point to the three design types and the number of settings each one has. Examples of settings are the max sample size, the timing of interim looks, and the thresholds for stopping. Then say “Every one of those subtrials needs its own rules. I started by tuning one of them by hand.”
I could not verify that a hand-tuned design met the power target
Eight settings in the EVOLVE calibration example
The outcome is progression-free survival, analyzed with a Bayesian piecewise exponential model. Interim reviews can stop the trial if benefit looks unlikely.
Max sample size
Spacing of interim looks
Events needed before a decision
Futility threshold
Efficacy threshold
Margin over the benchmark
Strength of the prior
Resolution of the survival model
Changing one setting can change which values work for the others.
Target power: at least 80%
Several more subtrials were waiting. How could we search so many settings systematically?
Pull-from-if-needed: Tell this as experience, without numbers.
Q&A pocket: How long did it take, and how close did you get? “Long enough that more subtrials were waiting. I did not keep a record I can cite, so I will not give a number.”
Pull-from-if-needed (the eight settings): These are the calibrated single-arm parameters listed in backup B3, in plain words. The minimum-events gate is the number of progression events needed before a decision counts. The hazard intervals set the resolution of the piecewise exponential model. The single-arm and randomized EVOLVE designs reported in the BATON 0.4.0 manuscript stop early for futility only and decide efficacy at the end. The seamless design stops in both directions (backup B6).
Delivery: Show the eight interacting settings in the documented EVOLVE calibration example. Name a few that the audience already knows, such as the max sample size, timing of looks, and stopping thresholds. Then tell the manual attempt as a separate recollection: “I started by tuning one subtrial by hand. I couldn’t verify that the design met our power target, and several more subtrials were waiting.” Ask how we could search systematically.
A grid search can produce 100 million candidate designs
Think hyperparameter tuning for trial rules. Which candidate is worth the next simulation?
Say aloud: “Ten options for each of eight settings is already 100 million designs, and each one has to be tested on thousands of simulated trials. Some settings meet both statistical targets. Within that group, we look for designs that enroll fewer patients. A grid has to walk the whole haystack. Random search throws darts at it.” Then ask the question at the bottom.
Pull-from-if-needed (grid arithmetic): With three values per setting, a grid has 9 designs for two settings, 6,561 for eight, and 1,594,323 for thirteen. The haystack and the zoomed map are schematic. Slide 24 shows the measured comparison on an eight-setting EVOLVE between-arm task (Scenario B).
Pull-from-if-needed (matched-budget evidence): BATON 0.4.0 manuscript Table 5. Three Simon scenarios with exact answers, 15 runs per method at matched budgets. Random search’s median relative gap in expected enrollment under the inactive scenario was 16.4%, and it found the exact best in 1 of 15 runs. BATON’s side of the same table comes in the notes for slide 23.
Q&A pocket: Didn’t the paper once report a random-search tally? Yes, a count of how many random designs met the targets. It was dropped because it described the search box, not a method, and the matched-budget comparison above replaced it. What about a grid at BATON’s budget? Manuscript Section 4.4 and Table 5 give a comparison on Scenario B, a separate between-arm design problem. Slide 24 shows the measured three-level grid and the BATON comparison runs on the same Scenario B task.
Delivery: 70 seconds. Haystack first, then the zoom. Call the grid size illustrative. Point to the two axes: the evidence thresholds to declare success or to stop for futility. One cell pairs one value from each axis while the other six settings stay fixed. Green combinations meet both targets. Outlined combinations also have lower expected enrollment. Say “In machine learning, you try a set of hyperparameters and see how it performs. Here the settings are trial rules, such as when to look at the data and how much evidence is enough to stop. Simulate many trials with those rules to see whether the design meets the power and false-go targets.” Land on “Which candidate is worth the next simulation?” Advance to the same map with BATON’s search path.
1 The stakes
2 The design problem
3 BATON
How do we search intelligently?
4 BATON in practice
5 Looking to the future
BATON in one picture
BATON stands for Bayesian Adaptive Trial Optimization. It applies Bayesian optimization to trial rules.
You supply a trial simulator · an analysis model · a performance evaluator · targets and an objectiveStart from a known design or specify your own trial
Space-filling start
Futility threshold Efficacy threshold
Fit model →
Model-guided search
Futility threshold Efficacy threshold
Two settings shown. Each dot is one design tested in many simulated trials.
Meets both targets Candidate to verify
Simulate → Fit → Choose, and repeat. Then check the best design with fresh simulations.
Say aloud: “Start with designs spread across the space. Simulate those designs and fit a model to the results. Then let the model guide which designs we test next.”
Delivery: 80 seconds. Point to the starting sample and follow the jumps through steps four and five. The diamond marks a candidate for verification. Reveal the measured grid comparison. Both numbers are allocated compute for the same EVOLVE design task. The grid tried combinations systematically. BATON used what it had already learned to choose which designs to test next.
Pull-from-if-needed: BATON stands for Bayesian Adaptive Trial Optimization. A team can start from a known design type or specify its own trial by connecting a simulator, an analysis model, and code for calculating trial performance. EVOLVE took the second route, since no existing calibration tool covered the designs it needed. The planned BATON Workbench would help teams connect their own designs, reproduce a run, and review how the design was selected and checked.
Q&A pockets: What is the approximation in the diagram? The surrogate is a prediction model fitted to completed simulations. The team supplies the trial’s analysis model. BATON fits a separate model to guide its search.
Delivery: Expand BATON as Bayesian Adaptive Trial Optimization. Say “The team supplies how to simulate the trial, how to analyze each simulated run, and how to measure performance across runs.” Define operating characteristics as how a design behaves when we repeat the trial under specified assumptions. Point to BATON’s prediction model, also called a surrogate. The trial analysis model measures evidence within one simulated trial. BATON’s separate model predicts performance for designs it has yet to try. Point to the space-filling start first. It is the left square from the previous slide and it runs once, before any model is fitted. Then follow the loop. The model learns from every design simulated so far, chooses the next design, and that design is simulated. The return line goes from that test back to the model. At the end of the search, reveal the independent check. The team can review the selected rules and the verification results. Keeping the same inputs and run settings lets the team reproduce the run. If a classifier drives routing or stopping, simulate plausible calls and apply the actual decision rules to them. Before enrollment, BATON chooses which design to simulate next. During enrollment, the trial follows the rules the team approved. One sentence on the planned Workbench if time permits.
Pull-from-if-needed (verification, from the former verification slide, now backup B14): Selection-induced optimism occurs when a noisy estimate helped a design win the search. Independent Monte Carlo runs check the selected design under the simulated scenarios. Reported operating characteristics come from verification.
Say aloud on the last click: “The design that looked best during the search may have benefited from simulation noise. So before BATON reports a design, fresh simulations check that it still meets both targets.”
A nonlinear model predicts how untested designs might behave
X Design settings
Y Simulated outcomes
Initial sample
.3 .9 32 6
72% 3% 16.2
.2 .8 36 5
88% 13% 19.5
Guided search
.3 .8 30 5
77% 8% 14.6
.3 .8 32 5
simulate →
82% 8% 15.1
Next design
.3 .8 31 5
Pending
x1 x2 x3 x4
Power Type I error Expected enrollment
Simulate → Fit Y from X → Choose
Illustrative values
Pull-from-if-needed: BATON uses Gaussian process regression to fit a nonlinear relationship between sampled design parameters and their simulation-estimated operating characteristics. The figure illustrates power as one output and one setting as the input. The actual search uses several design parameters and fits models for the quantities it needs to predict. The uncertainty helps it decide which designs to simulate next.
Q&A pockets: What are operating characteristics? They describe how the trial behaves when repeated under specified assumptions. Power, type I error, and expected enrollment are examples.
Delivery: Show how one row of settings (X) pairs with one row of simulation results (Y). On the first click, reveal the new design. On the second, show the simulation results for that row. On the third, show how the fit predicts power for untested settings, with a band indicating uncertainty. Point to the 80% target. On the fourth, reveal the dashed row. BATON chooses the next design by weighing potential improvement and feasibility. Its outcomes are pending until that design is simulated.
BATON balances potential improvement with feasibility
Which candidate is worth the next simulation?
Test the orange point. It could reduce enrollment and still meet both targets.
Simulate it, update the predictions, and choose again.
Pull-from-if-needed: This is a schematic of constrained expected improvement. BATON weighs potential improvement in the objective against the estimated chance of meeting the constraints. Gardner et al., Bayesian Optimization with Inequality Constraints describes the underlying method. Backup B1 has the technical details.
Q&A pockets: What can the objective be? EVOLVE used enrollment objectives. In a design problem whose simulator reports duration, the team could instead define an objective based on duration or a weighted combination of duration and enrollment. When does the search stop? It follows its stopping rules or reaches its evaluation budget. It can report the best feasible design found if one passes verification.
Delivery: About 50 seconds. Move from one setting to two. Say that each axis is a decision threshold: how much evidence is needed to stop for lack of benefit or to declare success. Feasible means meeting both statistical targets. Here the map predicts where those designs might be. Ask “Which would you simulate next, the white diamond or the orange point?” Pause, then reveal the choice. Say “We simulate the new candidate, update the model, and repeat. We’re searching for a feasible design with the smallest value of the objective we chose. When the search meets its stopping rule or uses its budget, we take the best candidate it found to fresh simulations.” In EVOLVE the objective was enrollment. A simulator that reports duration could support a duration objective or a weighted combination.
1 The stakes
2 The design problem
3 BATON
4 BATON in practice
What did it find for a real trial?
5 Looking to the future
Sanity check: a known answer tells us how close BATON gets
Simon’s two-stage design
Stop after a first group if too few respond.
Only four settings
Every possible design can be listed, so the best is known exactly.
Three ways to define best
Optimal : fewest patients on average. Minimax : smallest maximum. Admissible : a compromise.
BATON against Optimal: 3.6% more patients on average.
4 exact matches (dark points). All 12 met both targets.
Pull-from-if-needed: BATON 0.4.0 manuscript Section 4.3 and Web Appendix D.1. Each of the 12 benchmarks is one run with wide search bounds. The worst case, 13.5%, is the scenario with response rates 0.15 versus 0.35. The ten-start trace in B2 used narrower bounds on one scenario. Manuscript Table 5 gives the matched-budget comparison. Across three Simon scenarios and five starts each using exact operating characteristics, the median relative gap from the optimum was 3.5% for BATON and 16.4% for random search. That is a separate 15-run matched-budget comparison. The slide’s 3.6% is the mean relative gap across the 12 wide-bound benchmark runs.
Q&A pockets: Why is the optimum known? Simon’s two-stage designs have four integer settings (first-stage size, first-stage cutoff, maximum size, final cutoff), so they are simple enough to enumerate and check by exact binomial calculation. Simon, 1989 . What happens on EVOLVE? For the more complex trial designs, estimates come from fresh simulations. The benchmark checks the search against a known answer.
Delivery: Give the context first. A Simon two-stage design has four settings, the size of the first group, the response cutoff for stopping after it, the maximum size, and the final response cutoff. With so few settings we can loop over every possible design and know exactly which is best. Best has three standard meanings. Optimal has the fewest expected patients if the treatment is ineffective, Minimax has the smallest maximum size, and Admissible is a compromise between them. These three names come back on the EVOLVE slides. The comparison here is against the Optimal design. Set up the comparison before revealing the plot. Simon’s two-stage design enrolls a first group and can stop if too few patients respond. In this design family, we can check each candidate by exact calculation and find the one with the lowest expected enrollment under the null that meets both statistical targets. Ask how close BATON gets to that independently known answer. Each point is a scenario. The x-axis is the exact Simon design and the y-axis is BATON, both evaluated under the same null scenario. The dashed line is equality. Four designs match, all 12 meet both targets, and the average relative extra enrollment is 3.6%. The largest gap is about two expected patients. EVOLVE’s more complex designs use fresh simulations for verification. Turn from the benchmark to the decision EVOLVE had to make before the cohort opened.
EVOLVE designs have too many settings to try every one
Simon: 4 settings. Every design can be listed.
EVOLVE (the breast cancer platform trial): 8 to 13 settings. Listing every design is not possible.
One randomized EVOLVE subtrial, 8 settings. The score blends max sample size and expected enrollment. Lower is better.
BATON
model-guided search
51.3
score
76 core-hours
Grid, same effort
2 values per setting, 256 designs
101.3
score
matched to BATON
Grid, finer
3 values per setting, 6,561 designs
70.5
score
3,594 core-hours
A coarse grid does worse. A finer grid costs about 47 times more and still does worse. It is the hyperparameter tuning problem.
Pull-from-if-needed (source): BATON 0.4.0 manuscript Section 4.4 and Web Appendix D.5.4. On the Scenario B Admissible between-arm problem, the three-level grid attempted 6,561 evaluations and used 3,593.9 allocated core-hours. BATON comparison runs used a median 76.3 allocated core-hours, including search and certification. The grid selected a certified design with loss 70.45, versus BATON’s mean of 51.28. Core-hours measure allocated computing resources, for example 16 cores for one hour is 16 core-hours. The search pictures are schematic. A randomized, space-filling design supplies the initial sample when eligible prior evaluations are unavailable, and some EVOLVE runs reused earlier evaluations.
Q&A pocket: Is it faster than other optimizers? On this example the fine grid used much more compute. Under matched search budgets in Table 5, BATON achieved lower mean certified loss at overlapping compute spend. BATON used no tuning runs in that comparison. Is the AI a neural network? BATON uses a statistical model of the map, explained on the next slides.
Pull-from-if-needed (the three numbers): BATON 0.4.0 manuscript Table 5, Section 4.4 and Web Appendix D.5.3, Scenario B, the between-arm Admissible problem with eight active settings. The score is the certified weighted loss, a blend of maximum sample size and expected enrollment, and lower is better. BATON’s 51.28 is the mean over five runs, with a median of 76.3 allocated core-hours per run. The two-value grid evaluated all 256 designs within a budget of 282 evaluations per run, matched to BATON, and its mean certified loss was 101.25 over five runs (range 99.68 to 107.55). The three-value grid is a single run of 6,561 designs that used 3,593.9 allocated core-hours and certified at 70.45. The manuscript claims no computational saving against the other optimizers at matched budgets, so keep the claim to these grids.
Delivery: About 60 seconds. Remind the room that EVOLVE is the breast cancer platform trial from Section 2. Contrast with the sanity check. Simon’s design has four settings, so we could list every design. An EVOLVE design has eight to thirteen, so we cannot. If we try anyway, a grid with the same effort as BATON gives a design about twice as bad on this score, and a finer grid needs about 47 times the computing and is still worse. This is the same problem as tuning hyperparameters in machine learning by grid search.
EVOLVE had to settle each subtrial’s max sample size in advance
The team needed a max sample size about three months before first enrollment.
This EVOLVE design has eight settings. We estimated its performance by simulation.
Interim schedule
The calibrated rules fixed when the interim reviews happen.
Capacity across subtrials
The steering committee weighed the max sample size against expected enrollment before choosing a design for the first subtrial.
How many slots should a subtrial reserve when an early stop could leave some unused?
Pull-from-if-needed: BATON 0.4.0 manuscript Section 5. The steering committee used the trade-off displays to weigh maximum and expected enrollment when choosing a design for the first subtrial. Slide 24 gives the measured compute comparison on the randomized cohort.
Q&A pockets: What if an interim analysis happens at a different time? The reported operating characteristics apply to the schedule that was simulated. A revised schedule needs a new check.
Delivery: Open by contrasting with the sanity check. A Simon design has four settings (the two stage sizes and the two response cutoffs), so every design can be enumerated. An EVOLVE design has the eight settings from slide 16, a progression-free survival outcome and a Bayesian model, so there is no exact answer to compare against. Then lead with the enrollment-max sample size deadline. Explain that sites had to reserve capacity before the cohort opened. The committee used maximum and expected enrollment to weigh those options. These deadlines and the trial designs the team wanted to consider motivated BATON: we needed a faster way to calibrate more ambitious designs. Briefly connect the deadline to planning interim reviews and resources across cohorts. Ask how many slots to reserve when an early stop could leave some unused. The next slide puts numbers on that choice.
Same requirements, different trials
EVOLVE’s triple-negative breast cancer (TNBC) subtrial
Max sample size (slots reserved)
32
30
Average enrollment if ineffective
11.2
13.3
Power
82.1%
86.1%
Type I error
1.4%
1.5%
Would you reserve two fewer slots for about two more patients on average if the treatment is ineffective?
Pull-from-if-needed: BATON 0.4.0 manuscript Table 3 and Section 4.1. The Optimal design (H0-Optimal in the manuscript) is first shown on this slide. Slide 14 uses illustrative numbers and is not this design. Rows R01 and R02 of final_summary_designs.csv are used for the comparison. All three verified designs cleared the 80% power floor and 10% type I error max sample size at five fresh seeds. The hand-tuning experience on slide 16 motivated this work. It was not a controlled comparison.
Q&A pockets: Why are the expected values fractional? They average over simulations. Some simulated trials stop early and others run to the max sample size. Why do powers differ? All had to clear the same floor. The searches prioritized enrollment under different criteria.
Delivery: Return to the capacity EVOLVE had to reserve, then introduce the Optimal design. Call the second design a compromise between reducing the max sample size and reducing average enrollment. Give the group a moment to choose whether to reserve two fewer slots at the cost of enrolling about two more patients on average if the treatment is inactive. Then reveal the performance rows: both designs passed the same targets in fresh simulations. The third verified design appears on the next slide.
EVOLVE’s TNBC subtrial: three objectives, three designs
What can the team prioritize?
Optimal (fewest patients if ineffective): max sample size 32, 11.2 expected
Admissible (the compromise): max sample size 30, 13.3
Minimax (smallest maximum, tied): max sample size 30, 21.4
Simon defined these for his two-stage design. BATON carries them to the wider family of adaptive trials by swapping in an objective built to mimic each one.
Admissible and Minimax tied at the lowest max sample size searched. Minimax also required eight events before an interim decision.
Physicians can choose the design that fits their priorities, from verified options.
Pull-from-if-needed: The dashed line guides the eye between the three returned designs. The Optimal criterion (H0-Optimal in the manuscript) minimizes expected enrollment when the treatment is inactive, Minimax minimizes the max sample size, and Admissible weights both. Optimal and Minimax are Simon’s 1989 criteria and Admissible is the compromise of Jung and colleagues. In Simon’s setting each can be found by enumeration, as on slide 23. Here the design is Bayesian and adaptive with a time-to-event outcome, so BATON searches for the design that best meets each criterion by simulation. The original two-stage criteria are in Simon, 1989 and Jung et al., 2004 . BATON 0.4.0 manuscript Section 4.1 reports the designs and verification numbers.
Q&A pockets: Why does Minimax sit to the right of Admissible at the same max sample size? Minimax required at least eight observed events before an interim decision, which excluded the Admissible design’s four-event rule. The Minimax design also had higher verified power. Which design went into this protocol? The manuscript says the comparison informed the steering committee’s choice for the first subtrial. Confirm this specific protocol before claiming it used the displayed design.
Delivery: Say that Optimal, Minimax and Admissible were defined for Simon’s two-stage design, where the best design can be enumerated. We carried the same ideas to the wider family of Bayesian adaptive trials by swapping the objective function, with one objective built to mimic each criterion. The impact is that physicians have a choice of design preference. Point to the orange option the room just considered. Hand tuning made it hard to verify even one candidate. BATON gave the team several verified designs to compare before committing to a max sample size. Sites could see what they would gain in reserved slots and what they might give up in expected enrollment. Both Admissible and Minimax reached the search floor of 30 patients. The Admissible option expects 13.3 patients if the treatment is inactive, compared with 11.2 for Optimal. Name the three criteria in plain language: fewer patients on average if the treatment is inactive, the lowest max sample size, and a compromise. Point to the third option and say its search required at least eight outcome events before stopping was considered. Then ask whether the same workflow could handle a different question: screening and comparison in one trial.
The same workflow calibrated a more complex trial
Randomize patients from the start
Stage 1 Screen the experimental arm against a historical benchmark.
→
Decision to proceed Apply the planned rule.
No-go or ambiguous Resolve under Stage 1 rules.
Go to Stage 2 Compare experimental and reference arms.
The seamless design as calibrated, with 13 possible settings
Swap in new rules and a simulator. Reuse search and verification.
Pull-from-if-needed: The seamless design has 13 possible settings, up to 11 active in the reported calibrations. It was used to test whether BATON could calibrate a substantially more complex architecture. BATON reused the search and verification workflow from the single-arm and randomized examples with design-specific simulation, decision rules, and targets. Manuscript Section 5 and Web Appendices A.4 and E describe the design and calibration. In stage one, participants are randomized to both arms. The experimental arm is first screened against a historical benchmark. The concurrent reference data contribute to the decision to proceed and to the later comparison. If the conversion rule does not send the trial to Stage 2, the prespecified Stage 1 rules determine the outcome.
Q&A pockets: What does this demonstrate? The framework can accept a different trial design and use the same basic calibration workflow. In the calibrated examples the randomized stage stopped at its first look in every run. Backup B13 has more detail. Does the team alter the rules after seeing data? The rule for proceeding is chosen and calibrated before enrollment.
Delivery: About 45 seconds. “We wanted to see how far the same workflow could stretch, so we tried a more complicated design. Both groups enroll from the start. We first screen the experimental group against a benchmark, then use a rule to decide whether to proceed to a direct comparison.” Reveal the second stage. “We swapped in the new trial rules, simulator, and targets. The search and verification steps were the same in outline.” Ask what the team needs to get right in the new simulator.
What if the trial assumes perfect classification?
Work in progress
Trial calculations usually assume the biomarker test classifies every patient correctly.
Power lost as the test misclassifies more patients
Patients misclassified by the test (%)
Trial outcome survival response blood marker
Same specimen
↓
Different test call
↓
Different subtrial or treatment decision
With 10% misclassified, power fell by about 8 to 11 points. The false go rate barely moved.
Simulate plausible error rates in BATON. Recheck power and type I error, then recalibrate.
Say aloud: “A biomarker is the characteristic we want to measure. The assay measures it, and a rule or classifier uses the result to decide whether the person qualifies, which cohort they enter, or whether to switch treatments. Suppose two workflows give different calls on the same specimen. If the trial uses that call, it can change who enters the cohort or which treatment they receive. We can simulate plausible error rates and see what they do to power, false positives, and enrollment before we calibrate the rules.”
Pull-from-if-needed (the figure): The bar chart is the left panel of the figure on the slide “Biomarker uncertainty propagates to screening power” in the FDA OSEL talk When Biomarkers, Assays, and Protocols Co-evolve . It shows the change in detection probability (power) from a clean assay, averaged across simulated conditions, with 95% intervals, for three endpoints (progression-free survival, objective response and circulating tumor DNA). At 10% classification error the drop is about 8 to 11 points, and at 15% about 13 to 15. The false-promotion probability moved by less than 1 point in either direction. This holds for classification error that is unrelated to site, outcome or treatment. Biomarker is used broadly here and can be any measurement, tool or model that steers the trial, including machine learning models, digital twins and AI-generated biomarkers.
Pull-from-if-needed: This example adapts the FDA OSEL talk When Biomarkers, Assays, and Protocols Co-evolve . Vary sensitivity and specificity over plausible ranges for the intended population and setting. Error rates may also differ by subgroup or time point. BATON-C studies which trial rules hold up across such scenarios.
Q&A pockets: Who checks the simulator? The trial team supplies and checks it. BATON works on simulated trials at design time. The model ranks candidate designs and the team reviews the final design.
Delivery: Connect to the preceding slide: the search can reuse the framework as soon as the team supplies a simulator for the new design. Use the example of one specimen that receives a different test call, adapted from the FDA OSEL talk When Biomarkers, Assays, and Protocols Co-evolve . If eligibility depends on the biomarker, simulate both kinds of errors using plausible rates. That changes who is enrolled and may change the evidence at the interim review. Check how the same stopping rule behaves under the plausible scenarios, then tune it if needed. Close with “Now imagine asking an AI tool to write that simulator. Which test does it assume?”
A public service announcement on AI-written simulations
If you cannot explain or defend a choice you made with AI’s help, you may be delegating too much.
Pull-from-if-needed: Adapted from the StatsUpAI talk Staying in the Driver’s Seat, slides “In your own research” and “Correct code, wrong question.” The linked companion repository has the handout, checklists, and templates.
Q&A pockets: Does BATON see patient data? This use of AI is model-guided search over simulated trials, before enrollment. The StatsUpAI example is a broader lesson about what a researcher needs to judge.
Delivery: Keep the tone light, this is a public service announcement and not a warning against using AI. Take the question from the preceding slide. Start with the benefit. An AI coding tool can speed up the implementation, and that makes custom designs practical for teams that could not have built a simulator before. This applies to BATON and to any analysis. It might leave out test errors, and the resulting power estimate can look perfectly plausible. Ask the audience what they would check first. Before trusting the estimate, trace who the code puts into each cohort. Point briefly to the StatsUpAI companion resources. Reveal the last line and offer it as a test people can apply to their own work. If you cannot explain or defend a choice you made with AI’s help, you may be delegating too much. Then say “Writing and running the code is one job. Knowing what it has to represent is ours.”
1 The stakes
2 The design problem
3 BATON
4 BATON in practice
5 Looking to the future
Can the same search design the whole program?
What does Phase III inherit?
54%
of 640 therapeutics failed in late-stage development
Phase I
20–100
Informs dose choice
→
Phase II
Up to several hundred
Tests for activity
→
Phase III
300–3,000
Tests benefit at that dose
Typical enrollment. One dot is 25 participants.
What should earlier trials learn to help Phase III confirm real benefit?
FDA trial sizes · Hwang et al. (2016) · FDA case studies (2017)
Delivery: Bring back slide 6. Of 640 therapeutics entering Phase III or other pivotal trials in 1998–2008, 344 (54%) failed in late-stage development. Among those 344, about 57% were attributed to inadequate efficacy, 17% to safety, 22% to commercial reasons, and 5% had unknown reasons (rounded). Some failures correctly rule out an ineffective or unsafe option.
Phase III inherits both a choice and the uncertainty around it. The FDA reviewed 22 selected cases where Phase III overturned promising Phase II findings. For example, torcetrapib raised HDL cholesterol in Phase II, but patients on the drug had more major cardiac events in Phase III. That is a concrete example of how the choice of what to measure affects the evidence available when deciding to advance. Short follow-up can also leave harms uncertain. These case studies illustrate specific gaps. The share of the historical 54% attributable to earlier design choices is unknown. Point to the left cards first, then reveal the jump in patient and time commitment at Phase III. The labels are FDA’s general drug development ranges. The dots illustrate trials of 50, 200, and 600 participants. Actual designs vary and the ranges overlap. The listed years refer to study duration. On the last click, ask what uncertainty is worth resolving before the larger commitment. Call back to the open question on slide 8, that more patients later may not answer what the early trial never asked. Earlier studies can help choose the dose and population and plan which outcomes to carry forward. That planning can improve the chance of valid confirmation when there is real benefit and help stop weak candidates earlier. Transition: “One decision we can change is which dose we carry forward.”
An early dose rule can shape Phase III confirmation
Illustrative example
Two doses shrink tumors about equally. The higher dose has more side effects.
A · Tolerability first Choose the dose by a toxicity limit. The higher dose advances. More patients may cut back or stop.
B · Activity and tolerability Compare the doses on both measures. The lower dose advances. It may take more patients early.
To test in simulation From Phase I onward, how often does each rule yield valid Phase III confirmation, and with how many patients?
Delivery: Recall the opening sotorasib comparison of 960 mg and 240 mg. Introduce two illustrative strategies for choosing an oncology dose to carry forward. A tolerability-led strategy can use a prespecified toxicity limit. A comparative strategy can use both early activity and tolerability, which may call for more early evidence. Programs can advance or compare multiple doses. For the next example, assume each rule sends one dose forward. Across simulated programs, track advancement of toxic and ineffective doses separately. Then track whether an acceptably safe and effective dose makes it to confirmation. See FDA oncology dosage optimization guidance .
Delivery (the example): Use a hypothetical example inspired by the opening dose question. Both doses meet the early safety limit. Early tumor shrinkage looks similar and the higher dose causes more side effects. Here, a tolerability-led escalation rule advances the higher dose. More interruptions or discontinuations in a longer trial could change the observed benefit. A strategy comparing early activity and tolerability could advance the lower dose and might need more early participants. The simulation would compare the trade-off. Count from Phase I, include programs that stop early in the denominator, and tally patients across all phases. Pause on those two outcomes before moving to PARTNER.
How PARTNER counted early patients in the final comparison
PARTNER · breast cancer trial
Control Chemotherapy alone
+
→
Selected schedule Olaparib 48 hours after chemotherapy
+
→
Other schedule Olaparib before chemotherapy
Schedule dropped · outside the primary comparison
Compare how often pathology found no invasive cancer at surgery (pathologic complete response).
Source: Abraham et al., PARTNER trial, Nature 2024 , Methods and results. In the reported germline BRCA wild-type cohort of this Phase II–III breast cancer trial, an independent committee used prespecified criteria to select the schedule that started olaparib 48 hours after carboplatin–paclitaxel. The primary comparison included patients assigned to the control or selected schedule in stages 1 and 2 alongside those assigned in stage 3. The primary endpoint was pathologic complete response at surgery. The observed rates were 52% with chemotherapy alone and 51% with the selected olaparib schedule.
Delivery: “The dose is one thing we carry forward. The evidence is another.” Blue icons mark patients who were already randomized. On the first click, show the two arms enrolling new patients in stage 3 and the schedule that was dropped. On the second click, trace the blue icons into the final column. Each early patient stays in their original arm and contributes the recorded outcome alongside patients enrolled later. Compare how often surgery found no residual invasive cancer in the retained olaparib arm versus the control arm. The selected schedule did not improve the primary endpoint in this cohort. The icons show inclusion and are schematic. For BATON-Phaseless the proposed question is how much early evidence can count and how many new patients are needed, with an analysis that accounts for selection and protects type I error. See FDA adaptive design guidance .
Extend BATON from one trial to the whole program
Proposed extension
BATON would vary patients per phase · which dose advances and when to stop · how much early evidence countsEach candidate program is simulated from Phase I to confirmation. The best is rechecked with fresh simulations.
→
Phase II
Test for activity
→
Outcomes from earlier patients count in the final analysis →
earlier patients new Phase III patients · dots are schematic
Can we confirm benefit with fewer new patients or less time, while controlling false positives?
Delivery: Return to the BATON workflow from slide 19. We would replace the single-trial simulator and evaluator with ones that simulate and score the full program. Change the amount of enrollment in each stage, how many doses can remain in play, safety and activity thresholds, and the rule for what earlier data can contribute to confirmation. For each candidate, simulate complete paths under plausible dose–efficacy and toxicity scenarios. The search uses completed results to choose what to test next. Final candidates would be checked on fresh full-program simulations. BATON-D supplies the early dose-selection piece, and BATON-C helps ask which choices hold up across plausible assumptions.
Delivery (early evidence): Take overall survival as an example of a Phase III endpoint. An early randomized trial can plan to track survival with standardized follow-up or suitable linked records. Earlier patients start contributing follow-up sooner. If their outcomes are eligible under a prespecified analysis that accounts for selection, some events may already count when Phase III begins. That could reduce new recruitment or the wait for enough events. A separate option uses an early signal such as tumor response to forecast later event accrual and plan enrollment and follow-up. Predictions guide planning. The confirmatory analysis still uses observed outcomes. As in EVOLVE, set a minimum chance of valid confirmation when a safe and effective dose exists, limit false positives, and limit advancement of unsuitable doses. Count all simulated programs from Phase I onward, including those that stop. Compare total enrollment and time only among designs meeting the evidence targets. These are potential gains for the proposed extension to evaluate. See FDA adaptive design guidance and Stallard et al. on using early endpoints in interim decisions .
Where BATON goes next
In EVOLVE, BATON found designs that met prespecified targets in simulation.
BATON-D Phase I dose finding and seamless Phase I/II research Which doses are worth advancing?
BATON-C Robustness work in progress Which trial rules still work if our assumptions are off?
BATON-Phaseless Proposed whole-program calibration Can we preserve Phase III confirmation with fewer patients overall?
Design early trials around the evidence Phase III will need.
Delivery: Return to slide 9, where the bottleneck was finding efficient trial rules that still meet the evidence requirements. “The EVOLVE examples showed that BATON could search complex trial designs and verify that the selected rules met prespecified evidence targets in simulation.” Then turn to the roadmap. BATON-D studies early dose-finding decisions, including designs that continue from Phase I into Phase II within one trial. BATON-C asks which rules hold up across plausible scenarios, such as when the treatment effect is smaller or the biomarker classifier is less accurate than assumed. BATON-Phaseless is the proposed extension to calibrate decisions together from Phase I through Phase III, including what earlier evidence can count. Return to the opening dose question. “Before we ask hundreds of patients to join Phase III, what should we learn while we still have choices?” Then end on the slide’s own sentence, “Design early trials around the evidence Phase III will need,” and stop.
Acknowledgments
EVOLVE / ADAPT
MPIs: Lisa Carey (UNC), Ian Krop (Yale), Chuck Perou (UNC), Eric Winer (Yale), Antonio Wolff (Hopkins)
EVOLVE steering committee and TBCRC site investigators
ARPA-H ADAPT program leadership
Statistics / BATON
Amber Young (Rashid Lab PhD student), who led BATON's design and validation, and co-leads BATON-C
Dinelka Nanayakkara (Rashid Lab PhD student)
Didong Li (UNC)
ARPA-H Statistics Working Group: Susan Hilsenbeck (Baylor), Nabihah Tayob (DFCI), Ruizhe Chen (Hopkins)
Rashid Lab
BATON funding · ARPA-H ADAPT (ARPA-H-MAI-24-01-03, award 140D042590009) · DoD HT9425241103100121
NIH/NCI U01 CA274298, P50 CA257911, T32 CA106209, P30 CA016086, and P50 CA058223
UNC Lineberger Comprehensive Cancer Center and its University Cancer Research Fund
Q&A routing index (TriBix numbering) - Clinical phases → slide 3. Operation TrialBlazer and trials abroad → slide 4. Development time and cost → slide 5. - Late failure and sotorasib → slides 6–8 and 33. PARTNER and early randomized evidence → slide 35. - Why efficient designs are hard to find → slides 9 and 13–17. - Fixed versus adaptive and what patients gain → slides 12 and 26–27. - EVOLVE and the hand-tuning story → slides 15–16 and 26. - Grid and random search → slides 17 and 24 and the notes for slide 23. Other optimizers → B4. - Surrogate, acquisition, and fresh simulations → slides 19–21, B14 and B1. Search initialization → B12. - Benchmark and seed details → slide 23 and B11 for all 12 scenarios, B2 and B7 for ten starts on one scenario. - Choosing among feasible designs → slides 25–27 and B3. Seamless design → slide 28 and B13. - AI, assumptions, and judgment → slides 29–31 and B5. - Bidirectional stopping → B6. Randomized cohort → B8. - Early dose selection and Phase III confirmation → slides 34–34. Whole-program calibration and borrowing → slides 35–36. Status of BATON extensions → slide 37. ADAPT background → B9–B10. - Company or translation → disclose the relationship verbally in one sentence.
References
Biotechnology Innovation Organization, Informa Pharma Intelligence, QLS Advisors. Clinical Development Success Rates and Contributing Factors 2011–2020. 2021.
Wouters OJ, McKee M, Luyten J. JAMA 2020;323(9):844–853, corrected 2022.
U.S. Department of Health and Human Services. Operation TrialBlazer roadmap. 2026.
Hitchen et al. Adult solid-tumor Phase I trials with an Australian site. 2024.
Getz K. Characterizing white space in the quest to drive development speed. Applied Clinical Trials, 2021.
Hwang et al. JAMA Internal Medicine 2016.
Hong et al. KRAS G12C inhibition with sotorasib in advanced solid tumors. NEJM 2020;383:1207–1217.
Sotorasib 960 mg and 240 mg randomized dose comparison. 2024.
FDA Oncologic Drugs Advisory Committee briefing document, sotorasib, Table 29.
FDA. Adaptive Designs for Clinical Trials of Drugs and Biologics. Guidance for industry, 2019.
Simon R. Optimal two-stage designs for phase II clinical trials. Controlled Clinical Trials 1989.
Jung SH, Lee T, Kim K, George SL. Admissible two-stage designs for phase II cancer clinical trials. Statistics in Medicine 2004.
Rashid N. When Biomarkers, Assays, and Protocols Co-evolve. FDA CDRH OSEL seminar, 2026.
FDA. Step 3: Clinical Research (trial sizes by phase).
FDA. 22 Case Studies Where Phase 2 and Phase 3 Trials Had Divergent Results. 2017.
Abraham et al. The PARTNER trial. Nature 2024.
BATON software and manuscript (in revision).
Delivery: Leave this up only if someone asks for a source. The entries are in the order the sources appear in the talk, and each links to the source.
Backup: the surrogate and the acquisition
Gaussian process models for the weighted loss and each constraint.
Proportion constraints use design-specific binomial Monte Carlo variances.
Expected constrained improvement multiplies expected objective gain by modeled feasibility.
The reported calibrations used DiceKriging GP fits.
BATON 0.4.0: The supplementary methods describe fixed, design-specific Monte Carlo noise for the proportion constraints and a fallback for the objective when a variance is unavailable. The separate repeated-design hetGP path was unused in the reported calibrations.
Q&A pocket: How is the incumbent defined under noise? Best posterior mean, not best observed value: take the evaluated designs whose observed metrics satisfy the constraints, predict the objective surrogate’s posterior mean at each, and use the minimum. Feasibility classification is a plug-in on the simulated metrics; the objective value is de-noised through the surrogate. This is the plug-in form of Picheny (2013), and the manuscript states it that way. A best-observed option exists in the code but is not the default and was not used.
Q&A pocket: Constrained EI multiplies marginal feasibility probabilities; power and type I come from the same evaluation. Independence? Independent surrogates and a product of marginals, deliberately. The noise justification: power and type I are simulated under disjoint scenarios from non-overlapping segments of the RNG stream, so their Monte Carlo errors are independent even though they share an evaluation call. The latent surfaces can still be correlated as functions of the design, the paper says so, and the approximation affects only the search: every reported design is verified by direct simulation, never through the product. (Wording caveat: the supplement says “distinct random seeds”; the code uses consecutive segments of one stream. Independent either way.)
Q&A pocket: Common random numbers? Search evaluations used distinct seeds. Later checks used a common seed set across designs. The certification seeds were held apart from search and selection.
Q&A pocket: Isn’t plug-in feasibility for the incumbent inconsistent with a smoothed objective? A real asymmetry, concede it: feasibility of the incumbent set uses observed metrics while its value uses the surrogate mean. Mitigations: the search applies a buffered (tightened) constraint threshold, and no design is reported without high-fidelity multi-seed verification.
Backup: ten starts on one Simon benchmark
All ten runs finished within 1% of the optimum under narrower search bounds. Eight matched the known design.
Pull-from-if-needed: Single benchmark scenario with response rates 0.20 versus 0.40. Runs used narrower search bounds than the twelve-scenario benchmark. See BATON 0.4.0 manuscript Section 4.3 and Web Appendix D.2. B7 gives the run summary.
Q&A pockets: What does the trace show? It shows how the best feasible candidate changed as more designs were evaluated on this scenario.
Delivery: Point to the guided-search part of the trace and the final distance from the known answer.
Backup: what the constraints and parameters actually are
Calibrated (searched) parameters, single-arm:
maximum enrollment; interim spacing; minimum-events gate before decisions count
posterior futility threshold; posterior efficacy threshold
decision margin over the historical benchmark
analysis-prior concentration; hazard-model resolution (piecewise intervals)
Seamless adds conversion and second-stage settings (13 possible, up to 11 active in the reported calibrations).
Constraints (headline scenarios):
type I error at most the ceiling (0.10 in the scenarios shown)
power at least the floor (0.80)
seamless designs add gate-behavior constraints (null conversion rate bounded, alternative conversion rate floored)
Objectives, by philosophy: expected null enrollment (H0-Optimal), maximum enrollment (Minimax), weighted blends (Admissible family), expected alternative enrollment (H1-Optimal).
Scenario A truths: null median 3.8 months, alternative 9 (median ratio 0.42), Weibull shape 1.3, accrual 2 per month; maximum enrollment searched in [30, 120]. Two Simon scenarios use a 0.05 ceiling with 0.90 power.
Backup: known failure modes and limits
Budget exhaustion : 3 of 10 reproducibility runs used the full 300-evaluation budget rather than stopping early; quote evaluations used, not “convergence.”
Approximate optimization : worst single-run benchmark gap was 13.5%. For certified designs, the constraint statements have 95% familywise confidence within each certification family at the simulated scenarios.
Correlated constraints : the acquisition multiplies marginal feasibility probabilities; latent surfaces for power and type I error can be correlated. Affects the search only; reported designs are verified directly.
No common random numbers in the loop : candidate comparisons carry full independent Monte Carlo noise; verification uses a shared seed set.
One assumed truth per calibration : every current result conditions on an assumed clinical scenario; robustness across scenarios is BATON-C, in progress.
Backup: bidirectional stopping variants
The single-arm and between-arm headline scenarios use futility-only interim stopping (interims can stop for futility; efficacy is decided at the end). The seamless design is calibrated with bidirectional stopping throughout, because its conversion gate is efficacy-driven.
The choice between stopping structures is itself a design decision upstream of the philosophy choice.
Pull-from-if-needed: The earlier version of this backup described single-arm and between-arm reruns with early efficacy stopping. The BATON 0.4.0 supplement does not report those reruns, so the bullets were removed. The seamless design’s bidirectional stopping is in supplement Web Appendix E.
Q&A pocket: Why futility-only for the headline designs? That is the EVOLVE protocol’s choice for screening subtrials. Efficacy is judged at the end.
Delivery: Backup. One sentence.
Backup: ten seeds, one design
The whole pipeline, rerun end to end at ten independent random seeds:
Runs returning a feasible design
10 / 10
Runs within 1% of Simon’s optimum
10 / 10
Runs returning Simon’s exact optimal design
8 / 10
Expected null enrollment, mean (spread)
17.77 (0.06)
Evaluations used, median
250 of a 300 budget
The other two seeds return a near-neighbor design (max sample size 29 instead of 25) whose expected enrollment differs by 0.15 patients. What reproduces is the design’s behavior, and usually the design itself.
Rerunning the search shows how much the returned design and its behavior change with the seed.
Pull-from-if-needed: Expected null enrollment takes two values across the ten seeds: 17.74 in eight runs and 17.89 in two. Simon’s exact optimum for this scenario (response rates 0.20 versus 0.40) is 17.74, so the modal BATON design matches the Simon optimum. The mean gap across all ten runs was about 0.2%. Three of ten runs used the full 300-evaluation budget; quote evaluations used, never “convergence.” These runs searched a narrower box that contained the optimum (supplement Web Appendix D.2, Web Table 7). Continuous thresholds vary across seeds while mapping onto the same integer stopping boundaries: the behavior surface is piecewise constant in the thresholds, which is why behavior reproduces exactly.
Q&A pocket: Do all ten seeds return the identical parameter vector? No, and say so plainly: eight of ten return the identical vector, which is Simon’s optimum; two return capacity 29 instead of 25. The design vector is the contract with sites and budget, so the honest claim is “one of two designs, the modal one exactly optimal,” not “one design, always.”
Delivery: Fast slide. Lead with all ten within 1% of the optimum, then 8 of 10 exactly.
Backup: the randomized-subtrial frontier
Cutting the max sample size from 103 to 91 costs about 5 extra patients on average when the treatment does not help. The next 5 cost about 7 more.
Pull-from-if-needed: BATON 0.4.0 package, manuscript Table 4 (results/retune_xphil/final_summary_designs.csv, rows R23 to R25), totals across both arms, null median 6 months against 9. The caption is arithmetic on those rows (63.9 minus 59.2, 70.8 minus 63.9).
Q&A pocket: Are these three restarts stable? The five restarts behind each design had maximum-size standard deviations of 9.7, 4.1 and 1.6 (supplement Web Appendix D.3).
Delivery: Backup for the randomized-cohort question on slide 27.
Backup: the ARPA-H ask and the N-of-1 addition
(ii) Clinical trial characteristics:
Clinical trial for metastatic cancer patients.
Clinical trial that includes multiple lines of therapy under a single protocol.
Flexible clinical trial protocol that enables changes in design (e.g., randomized, adaptive, etc.).
Strong capability to collect tumor and blood-based measurements.
Possess a treatment line with a Progression Free Survival (PFS) rate of <9 months average.
Mechanisms for biomarker evaluation.
Modular clinical trial design enabling rapid dissemination and protocol uptake by different sites.
(iii) General characteristics: key clinical needs; evolutionary design examples and rapid-enrollment plans; treatment lines; biomarker testing and integration plans.
New ask (program-manager email): integrate an "N-of-1" trial component.
excerpts from the ARPA-H solicitation and program-manager correspondence
The ask: design an adaptive multi-line trial to discover, evaluate, and implement novel biomarkers , with an additional N-of-1 subcomponent
Delivery: Three keypresses, one element at a time, in the spirit of the original PowerPoint build: solicitation excerpts, then the new-ask line, then the summary ask, then the meme as the punchline. Let each be read, not read aloud. If asked, the full source slides (complete requirement list and program-manager email) are in the pptx decks.
Backup: ADAPT technical areas and the EVOLVE team
Pull-from-if-needed: For questions about the program: three technical areas (our team leads the clinical-trial component), metastatic breast cancer, master protocol, biomarker-defined subtrials, TBCRC network.
Backup: Across 12 benchmarks: feasible in all 12, mean objective gap 3.6%
One run per scenario; ratio = BATON’s expected null enrollment / Simon’s exact optimum.
Feasible design returned
12 / 12
Exact Simon design returned
4 / 12
Mean ratio, BATON / Simon (SD)
1.036 (0.041)
Worst-case ratio
1.135
Best-case ratio
1.000
Feasibility is verified directly. Objective optimization remains approximate.
Say the full sentence: across 12 benchmark scenarios BATON returned a feasible design in all 12, and the mean objective gap was 3.6%.
Pull-from-if-needed: BATON 0.4.0 package, manuscript Section 4.3, results/validation/multiscenario_simon_summary.csv. One wide-bounds run per scenario. The worst case (ratio 1.135) is the 0.15 vs 0.35 scenario. Never compress this to “recovers the optimum” in every scenario: exact match in 4 of 12, mean gap 3.6%, worst 13.5%. The plot is backup B2.
Q&A pocket: Median? 1.025, from the same file. The paper leads with the mean.
Delivery: Backup. Read the table only if asked.
Backup: The seed sketches the map. Then BATON proposes each batch.
Initial points learn broadly; later points concentrate where the answer may improve.
Delivery: “Step one is not clever: scatter a small sample so the model has something to learn from. After that, every batch is placed by the model.” One beat per panel.
Pull-from-if-needed: In the BATON 0.4.0 runs behind the paper’s tables, each search starts from 60 or 90 space-filling designs at low fidelity, or from 3 designs carried over from a related search, then hands over to model-guided selection (results/retune_xphil/final_search_selected.csv). Points here are illustrative (synthetic surface), not real design coordinates.
Backup: Calibrating a seamless design
Stage 1 Randomize from the start. Screen the experimental arm against the historical benchmark, then apply the conversion rule. Stage 2 For trials that continue, compare the randomized arms. The reported calibrations stopped at the first Stage 2 look.
The seamless design had 13 possible settings. At most 11 were active in the reported calibrations.
Say aloud: “This is the kind of design we can now seriously consider, because calibration is tractable.” And: “The first stage lets us screen with fewer patients before committing to randomized evidence.”
Pull-from-if-needed (BATON 0.4.0 package, supplement Web Appendix E): four criteria were calibrated on the seamless design (Minimax, H0-Optimal, a balanced blend and H1-Optimal, no Admissible), maximum 76 to 132 patients, each search 13 to 30 minutes on 19 workers after a 300-design starting screen whose cost is excluded. At most 11 of the 13 settings were active. When the treatment is inactive, 2.2% to 4.0% of simulated trials moved to the randomized stage. The earlier “two objectives collapse onto one design” story belonged to a different version of the analysis and has been removed from the deck.
Q&A pockets: Is the seamless design a headline result? No, it is in the supplement. In all four designs the randomized stage stopped every trial at its first look, so treat it as a demonstration that the search can handle 13 settings, not as a recommended design.
Delivery: 80 seconds. Title, full figure (single-arm 8, between-arm 8, seamless 13 settings), then reveal the two stages and the scope of the reported calibration. Gloss “interim gate” aloud as the point where the trial decides whether to move to the randomized stage.
Backup: verifying the final design with fresh simulations
Fresh simulations check both targets before BATON reports a design.
Selection and certification (BATON 0.4.0, manuscript Methods and Web Appendix D.4): Re-simulate the 10 lowest-loss designs among those feasible in search. Select by loss among candidates that meet both targets on re-screening. Check the selected candidate at five further seeds with 10,000 simulations per seed as part of selection. Independent certification then uses fresh seeds and 50,000 trials per hypothesis for the single-arm example or 100,000 for the between-arm example. The exact one-sided Clopper–Pearson bounds use a Bonferroni adjustment across the certification family. The seamless designs have a separate family. The figure illustrates the single-arm path.
Pull-from-if-needed: Selection-induced optimism occurs when a noisy estimate helped a design win the search. Independent Monte Carlo runs check the selected design under the simulated scenarios. Reported operating characteristics come from verification.
Q&A pockets: Why run fresh simulations? A design can look good partly because the initial simulations were noisy. Fresh simulations give an independent check against the targets.
Delivery: After the search reaches its stopping rule or budget, say “The design that looked best during the search may have benefited from simulation noise. Fresh runs check whether it still meets both targets.” Fresh runs check the design under the assumptions supplied. Then ask how close BATON got when the best answer was known, and what options it gave EVOLVE.