Drawing Dvořák

Pat Healey, a computer scientist at Queen Mary, has looked at the role of sketching in multimodal interaction. As they pervade everyday life computers increasingly have to be multimodal – that just means they communicate, for example, with both sound and images. A satnav is multimodal. It gives you voice commands and also has a map showing the route. The combination is more effective than either would be alone.

Pat explored how well people can communicate music by drawing. What kind of drawings do people do to visualise Beethoven, for example, and how do they look different to ones of jazz music, say? The results help computer scientists understand how we naturally link sounds and images and so how best to combine them when designing devices that use multimodality more naturally.