Comparison by task

Mean reward and mean USD cost for n = 3 trials. Click a row for the side-by-side traces. Gap and the Average row's deltas show the absolute value and the relative % change of Sonnet+MCP over Fable. Gap = Sonnet with Sourcegraph minus Fable on a plain checkout.

Task	Sonnet + Sourcegraph		Fable, without MCP		gap
Task	quality	cost	quality	cost	gap
`ccx-crossorg-217`	0.695	$0.46	0.104	$1.07	+0.591 +567%	compare →
`ccx-vuln-remed-135`	0.611	$0.23	0.231	$0.93	+0.380 +165%	compare →
`ccx-migration-289`	0.681	$0.85	0.574	$1.32	+0.107 +19%	compare →
`ccx-agentic-223`	0.625	$0.56	0.537	$0.99	+0.088 +16%	compare →
`ccx-migration-274`	1.000	$0.94	0.928	$1.09	+0.072 +8%	compare →
`ccx-vuln-remed-126`	0.759	$1.34	0.735	$0.84	+0.024 +3%	compare →
`ccx-config-trace-010`	1.000	$0.18	1.000	$0.46	0.000 0%	compare →
`ccx-incident-145`	0.172	$0.55	0.172	$1.19	0.000 0%	compare →
`ccx-crossorg-288`	0.551	$1.25	0.833	$1.45	-0.282 -34%	compare →
Average 9 tasks	0.677	$0.71	0.568	$1.04	+0.109 reward (+19%) -$0.33 cost (-32%)