Are medical postgraduate certification processes valid? A systematic review of the published evidence

Objective To collate the published works on validation of assessments used in postgraduate medical certification. Design Systematic review of original papers on reliability and validity of assessments used in medical postgraduate certification. Setting Medical and education research databases. Results Fifty-five papers were identified from 1985 to 2000. A wide range of approaches to validation were employed. Inter-rater reliability and internal consistency were the most reported foci for validation. There were just two papers on consequential validity, and only a few on construct validity. These two forms of validity are considered central in recent general education writing. The majority of papers were from general and family practice. There was a noticeable lack of papers from the UK Royal Colleges (except the Royal College of General Practitioners), despite 5 years of the new unified grade and the renewed emphasis on the role of the Royal Colleges in setting assessment criteria. Conclusions There is a relative scarcity of published papers on validation of assessment for postgraduate medical certification considering the influence these high stakes processes have on doctors career progression and employment opportunities. General and family practice institutions in a number of English speaking countries have set an example to others, by showing that rigour and transparency in assessment development and implementation can be reflected in publication.