Summary

Objectives: This study aims to demonstrate the usability of discourse analyses as a means of evaluating medical informatics systems by examining one particular computer-based data-to-text system for delivering neonatal health care information. Methods: Six textual summaries of clinical information, three produced by human clinicians and three by the data-to-text system, were subjected to fine-grain discourse analysis. Analysis was performed ‘blind’ on all six textual summaries. Analysis focused on the identification of lexical items and on the potential effects of these items on users of these clinical information summaries. Results: Results showed that there were clear differences between human- and system-generated clinical summaries, with human clinicians providing better narrative flow and textual detail. The data-to-text system successfully produced textual summaries although it fell short of human abilities. Conclusions: These results indicate potential future improvements to the system. Discourse analysis as used here may offer significant advantages in evaluating and developing similar medical informatics systems.