The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using Semantically Unpredictable Sentences

This paper describes the experimental set-up used by the SAM (ESPRIT-BRA Project no. 2589: Multilingual Speech Input/Output: Assessment, Methodology and Standardisation) group for evaluating the intelligibility of text-to-speech systems at sentence level. The SUS test measures overall intelligibility of Semantically Unpredictable Sentences which can be automatically generated using five basic syntactic structures and a number of lexicons containing the most frequently occurring mini-syllabic words in each language. The sentence material has the advantage of not being fixed, as words can be extracted from the lexicons randomly to form a new set of sentences each time the test is run. Various text-to-speech systems in a number of languages have been evaluated using this test. Results have demonstrated that the SUS test is effective and that it allows for reliable comparison across synthesisers provided guidelines are followed carefully regarding the definition of the test material and actual running of the test. These recommendations are the result of experience gained during the SAM project and beyond. They are presented here so as to provide users with a standardized evaluation method which is flexible and easy to use and is applicable to a number of different languages.