Why reproducibility matters for biomedical literature AI

An evidence tool that gives a different answer every time is hard to trust and impossible to cite. But for a synthesis engine running over a living literature, perfect run-to-run identity is neither achievable nor, by itself, the right target. What matters is which layers are stable. A stable conclusion built on a fixed foundational base, with verifiable citations, is reproducible in the sense a scientist actually needs, even if the exact list of supporting papers shifts.

So we measured BioSkepsis at the level of layers, not a single pass-or-fail. We ran one question three times, two sessions back to back and a third several hours later, on the Pro tier, and compared the output along six dimensions. The query was held constant: what is the mechanistic evidence that SGLT2 inhibitors confer cardiovascular benefit independent of glucose lowering, with key supporting papers and PMIDs.

Results: what stayed stable and what drifted across runs

The table below summarises every dimension we measured. A general-purpose LLM run the same way typically gives a different prose answer each time with no fixed evidence base and no way to audit which sources are real; the contrast is the point.

The foundational-paper layer is the most stable biomedical signal

BioSkepsis computes its foundational-papers panel from how often papers are co-cited across the full retrieved corpus, not from which papers the language model happens to name in prose. Because co-citation structure is a stable property of the literature itself, this layer reproduces well at its core.

Six papers appeared in the foundational panel of every run: the two mechanistic anchors (PMID 27752710, PMID 29197997) and the four landmark outcome trials, DECLARE-TIMI 58 (PMID 30415602), DAPA-HF (PMID 31535829), EMPEROR-Reduced, and EMPA-REG OUTCOME (PMID 26378978). For a question about glucose-independent cardioprotection, that is the correct anchor set, and it held three times. What varied was the periphery: which secondary trials and mechanistic reviews rounded out the panel, tracking the slightly different corpus each run retrieved. The co-citation counts themselves shift between runs, so treat the panel as a stable membership of core papers, not an identical ranking.

Full-text availability is deterministic, not random

BioSkepsis reads the full text of open-access papers to ground its answers. A fair question is whether the same paper is read in one run but skipped in another. In this test it was not. Across papers that appeared in more than one run, none changed status. Open-access papers downloaded every time; paywalled papers failed every time with the same reason; open-access papers whose publishers block automated retrieval stayed blocked every time.

One caveat for honest reading of the numbers: the "failed" count mixes three different causes that should not be conflated. Genuine paywall is true unavailability. "Open access but the publisher blocked automatic retrieval" means the content exists but the crawler was refused. And "too many papers for your plan" is a tier quota, not a retrieval failure at all. A headline failure count overstates true unavailability if these are lumped together.

Honest pros and cons of this behaviour

This design has real strengths and real limits. Both belong in the open.

• Conclusions are stable: you get the same mechanistic answer to the same question, run to run.
• Citations are real and auditable: every checked PMID resolved correctly, with sentence-level grounding to the source passage.
• The verification layer is reproducible: across runs it consistently excluded contradictory and overreaching papers, including one paper that argues empagliflozin does not inhibit NHE1, correctly kept out of support.
• Full-text availability is deterministic, so a paper's evidence either contributes consistently or not at all.
• The foundational core reproduces, giving a fixed anchor set for a field.
• The specific per-claim citation list varies between runs; two runs can produce two different reference lists for the same conclusion.
• The ranked order of retrieved results is not reproducible, so list position should not be read as a fixed importance score.
• The full corpus is curated, not exhaustive; this complements, and does not replace, a comprehensive database search for a formal systematic review.
• Some open-access papers cannot be auto-retrieved because publishers block crawling, which narrows what full text can be read in a given run.

A guide to reproducible biomedical evidence retrieval, run to run

The variability above is manageable once you know which layer to trust. These practices apply to BioSkepsis specifically, and most apply to any AI literature tool.

Frequently asked questions