Exploring Modality Switching for Performance-related factors in video games.
We all have played video games once in our lifetime whether it is on mobile phones or computers. There are different ways in which you can control the basic gameplay of a video game like the most traditional HCI modalities i.e keyboard and mouse in computers and touch in mobile phones. There are few games out there that use different controllers to play the game such as Nintendo’s Power Glove, Logitech’s True force steering wheel controller, etc. But to capture the true expressive capabilities of a user, they do not do a good job as compared to natural human input like speech and gestures, and through BCI(Brain-Computer Interfaces) which can directly be provided as input to computers.
What is Modality Switching?
Modality switching often involves shifting between communication channels that vary in their ability to transmit nonverbal and social information.
Some examples of modality switching games are LifeLine and Albedo which use a mixture of voice recognition and traditional HCI inputs like mouse and keyboard.
In an experiment carried out by Hayrettin Gürkök, Gido Hakvoort, and Mannes Poel for Modality Switching, performance and performance-related factors were prominently effective in modality switching. In their study participants played different games with modality switching. First, the users played unimodal games in which only ASR(Speech recognition) or BCI(Brain-Computer Interaction) is an available modality just to be aware of the performances of the modality process.
The game played in this experiment was Mind the sheep, which is a multimodal game where the player needs to herd a flock of sheep across a field by commanding a group of dogs. The game world contains three dogs, ten sheep, a pen, and some obstacles.
The aim is to pen all the sheep as quickly as possible.
Here’s how this unique game works:
To command a dog, the player positions the cursor at the point to which the dog is supposed to move. The player holds the mouse button pressed to provide the command to select the dog. Meanwhile, the game displays cues specific to the active modality (ASR or BCI). When ASR is the active modality, names appear under the dog images and the player pronounces the name of the dog they want to select. When BCI is the active modality, dog images are replaced by circles flickering at different frequencies and the player concentrates on the circle replacing the dog they want to select. The stimulation persists and, depending on the active modality, EEG or acoustic data is accumulated as long as the mouse button is held. When the user releases the mouse button, the signal is analyzed and a dog is selected based on this analysis. The selected dog immediately moves to the location where the cursor was located at the time of mouse button release. When configured, it is possible to switch between the modalities by pressing the Ctrl key on the keyboard.
Analysis:
It turned out to be that participants preferred ASR much more than BCI. Usually, participants changed modality when their commands were not understood by the game or they did not change at all knowing that their command would not be understood. Another reason to switch modality is out of sheer curiosity and the curiosity originated when their commands were interpreted incorrectly.
Conclusion:
The study showed that Non-switchers able to achieve better performance in the multimodal game but active switchers could improve their performance in the multimodal game only in comparison to the BCI game.
Participants tend to use ASR more than BCI because ASR was much quicker as compared to BCI when the participants got used to it and less tiresome since the participants had to focus on the dogs to make it move in BCI modality.
Citation:
Hayrettin Gürkök, Gido Hakvoort and Mannes Poel. “Modality switching and performance in a thought and speech controlled computer game”
