This paper describes the process involved in the development of a gold standard dataset in oral squamous carcinoma (OSCC) against which future filters for prognostic biomarkers could be tested. Given the lack of filters for biomarkers in OSCC, the authors set out to produce a set of key studies that a filter developed for use with other databases should retrieve. They designed a MEDLINE search strategy to retrieve prognostic studies using the prognostic filter developed by the Haynes group. An annotating system was used to review the references retrieved by the search. Two reviewers independently assessed the studies and a calibration process for the results is described. The agreement about relevance across reviewers was measured by Cohen’s kappa statistic, achieving a high level of agreement. This system for developing gold standards is likely to be generalizable to other subject domains.