# p001 AB — A/B testing API for agents

Base URL: https://ab.p001.ai/api/v1
Auth: "Authorization: Bearer <key>". Get keys at https://ab.p001.ai/dashboard → API keys.
- ab_sk_… secret key: all endpoints.
- ab_pk_… public key: only /assign and /track (CORS enabled, safe in browsers).
MCP (streamable HTTP): https://ab.p001.ai/api/mcp with the same bearer header.
Tools: list_experiments, create_experiment, get_experiment, update_experiment, set_experiment_status, assign, track_event, get_results, sample_size.

## Concepts
- unitId: the stable thing being randomized (user id, session id, conversation id). Use the same unitId for assign and track.
- Bucketing: sha256(salt:unitId) → deterministic and sticky. The first assign records an exposure; later calls return the same variant even if weights change.
- The first variant is the control. Each variant can carry a JSON payload (copy, prompt, model, params…) returned by assign.
- trafficPercent: share of units that enter at all; others get the control with inExperiment=false and are not analyzed.
- Conversions count only events with name == primaryMetric (or a secondary metric) that happen after the unit's first exposure.
- Status: draft → running ⇄ paused → stopped. Paused: existing units keep their variant, new units get control. Stopped: everyone gets the winner (or control).
- Variants are locked once an experiment leaves draft. To change them, create a new experiment.

## Endpoints

POST /experiments
  {"key":"checkout-cta","name":"…","hypothesis":"…","primaryMetric":"purchase","secondaryMetrics":["add_to_cart"],
   "variants":[{"key":"control","weight":1,"payload":{…}},{"key":"b","weight":1,"payload":{…}}],
   "trafficPercent":100,"start":true}
  key: [a-z0-9][a-z0-9_-]{0,63}. Returns 201 {experiment}. 409 if the key exists.

GET /experiments                 → {experiments:[…]}
GET /experiments/{key}           → {experiment}
PATCH /experiments/{key}         {name?, hypothesis?, trafficPercent?, primaryMetric?, secondaryMetrics?, variants? (draft only)}
DELETE /experiments/{key}
POST /experiments/{key}/status   {"status":"running"|"paused"|"stopped","winner"?:"b"}

POST /assign  {"unitId":"user_123","experiments"?:["checkout-cta"]}
  Omit experiments to get every running experiment.
  → {unitId, variants:{"checkout-cta":"b"}, assignments:[{experiment, variant, payload, inExperiment, reason}]}
  reason ∈ assigned | sticky | not_running | outside_traffic | paused | stopped

POST /track   {"unitId":"user_123","event":"purchase","value"?:49.0,"properties"?:{…},"timestamp"?:ISO-8601}
  or {"events":[…up to 500]}. → 202 {accepted}
  value is summed per unit (revenue, score…) and analyzed as value per user.

GET /experiments/{key}/results?alpha=0.05
  → {totalUsers, sampleRatioMismatchPValue, adjustedAlpha, recommendation, significantWinner,
     primary:{metric, variants:[{variant, isControl, users, conversions, conversionRate, valuePerUser?,
       vsControl:{lift, absDiff, ci95:[lo,hi], pValue, probabilityToBeatControl}}]}, secondary:[…]}
  Two-proportion z-test vs control, Welch test for values, Bonferroni across treatments,
  chi-square SRM check (p<0.001 means the split is broken; don't trust results).

POST /sample-size {"baselineRate":0.05,"minDetectableEffect":0.1,"variants":2,"dailyUnits"?:2000}
  → {perVariant, total, estimatedDays}   (no auth needed)

Errors: {"error":{"code","message"}} with 400/401/403/404/409.

## Recommended agent workflow
1. sample_size to check feasibility.
2. create_experiment(start=true) with a hypothesis and a primaryMetric.
3. In the product code: assign(unitId) where the variant is used; render the payload. Never choose variants yourself.
4. track(unitId, primaryMetric) at conversion time.
5. get_results. Respect the planned sample size and don't stop on the first significant peek; check SRM.
6. set_experiment_status(stopped, winner), then hard-code the winner and remove the assign call.