• UABDivulga
26/11/2020

Jocomunico: a communication system that turns pictograms into natural language

jocomunico

Currently, severe speech impairments can be mitigated with augmentative and alternative communication (AAC) systems that replace or complement this function, transforming it, depending on the system, into different types of messages, many of them in telegraphic language. Still, the quality of the interaction is often frustrating for users. This research carried out by Joan Pahisa during his doctorate at the UAB Department of Information and Communications Engineering (DEIC) focused on creating a new system of AAC, Jocomunico. This pictogram-based system will help improve the quality of patient's interactions and with it, their quality of life.

Frame from the CAA app Jocomunico.

Jocomunico is an augmentative and alternative communication (AAC) system that is unique compared to existing systems. AAC is the set of tools and strategies that replace or support natural speech. In this case, Jocomunico is a pictogram-based communicator used by people with severe speech impairments that build sentences by means of boards of pictograms. 

The system is unique in that it transforms the telegraphic language resulting from pictograms into natural language, in Catalan and Spanish. For instance, a sentence like “escuela querer yo ir” (school want I go) would be translated into “Quiero ir a la escuela” (I want to go to school). 

Communication through AAC devices is slow and, when the resulting output are telegraphic-like sentences, it reduces the quality of human interaction and can cause frustration in the users. Trying to improve these aspects and, thus, their quality of life, was the main motivation behind this project.

In the course of the PhD*, which this solution is based upon, multidisciplinary research was done in the fields of AAC, machine translation, linguistics and software accessibility in order to develop and test the system in a real-life environment with end users. After the system was ready, these tests were performed during 3 months at a day care center for people with severe cerebral palsy. Aside from cerebral palsy, the 4 participants also had ranging degrees of linguistic competence and cognitive disabilities. Nevertheless, as all of them had used pictogram-based AAC for the past 30 years, their linguistic abilities were stable. This was very important in order to ensure that the observed changes would be due to the introduction of the new system.

During tests, participants were asked to produce prepared sentences and also to participate in one on one conversations adapted to their skills. By the end of the last session, participants were able to learn new linguistic skills, such as the use of basic verb tenses, as well as slightly increase their communication rate (number of words per minute).

Furthermore, and more importantly, the language expansion system also proved a source of motivation and, based on the feedback from participants, caregivers and speech language pathologists from the center, it improved the overall communication experience, as the output was in natural language. 

Finally, it is important to highlight that the system fit well with the linguistic competence of each person and required no learning curve during tests when none of its special features (different from most communicators), aside from language expansion, like gender, number, verb tense or sentence type modifiers were used, showing that it can be easily introduced to new users familiar with pictogram-based communication. 

Results shown in the research were later backed by the fact that, since its release, Jocomunico has been adopted by more than 1000 users throughout Spain.

*The thesis was directed by Jordi Herrera Joancomartí.

Joan Pahisa-Sol

Information and Communications Engineering Department.
Universitat Autònoma de Barcelona.

References

Pahisa-Solé J, Herrera-Joancomartí J. Testing an AAC system that transforms pictograms into natural language with persons with cerebral palsy. Assist Technol. 2019;31(3):117-125. doi:10.1080/10400435.2017.1393844. Article extracted from the doctoral thesis of Joan Pahisa (UAB), 2017.

 
View low-bandwidth version