Care2Report: a neural-symbolic linguistic pipeline architecture for conversation summarization into medical reporting
by Sjaak Brinkkemper
In the healthcare domain the reporting of consultations for electronic medical records (EMR) has serious drawbacks: on-site summarization takes away precious time from the patient, decreases information exchange and patient-directed gaze, and leads to high administrative burden among healthcare providers (HCP).
We present the Care2Report research program at Utrecht University, where we are developing a proof-of-principle medical reporting system based on a three-phased pipeline architecture consisting of a collection of computational-linguistic software and machine learning components. (1) Multimodal recording: Preprocessing and transformation of audio and video input data from medical visits into text using speech and action recognition technology. (2) Semantic Interpretation: Formal representation of speech, measurements and treatments based on multimodal guideline-concordant inputs combined with ontological semantic technology. (3) Report generation: Generation of a visit summarization (medical report) based on medical domain practices, followed by report completion, checking by the HCP and uploading through a generic EMR-interface. The integrated multimodal recording allows for input of the complete interaction during a medical visit, including question-answering, clinimetrics tests (e.g. Barthel, physical examination), and free communication by patient/caretaker with HCP.
We present the design principles of the Care2Report system, its functional and technical architecture, the underlying formal representation, and the components of the dialogue summarization pipeline. We end with an invitational challenge for the development of a generic conversation summarization platform for reporting of structured meetings in any language.
-