How it works
MatchMySound is an application that utilizes sound recognition technologies to compare two audio files – a pre-recorded reference track provided by the teacher or publisher and the one recorded by a learner. Our algorithm detects the differences between the audio files every 1/43rd of a second. It focuses on two main dimensions: sound and timing. The former includes the differences in sound qualities such as pitch, intonation, and articulation, while the latter provides the same for the timing correctness. The feedback is presented as both numerical scores and graphic displays that provide a finer level of detail. When creating an assessment, the teacher can specify goals for the scores. When students do not reach the goal, they are advised to try again. In addition to utilizing actual audio recordings, the system can also work with synthesized audio (MIDI).
While recording, the student only sees the progress bar which aligns to where they are within the piece or exercise. The primary function is to confirm that the computer is listening. Feedback is displayed only after the system determines that the student has finished playing. This feedback model is called the Knowledge of Results (evaluative feedback from an external source) in research. It also emulates the classical teaching practices in music education, where teachers usually let the student finish playing before giving their comments and suggestions. In MatchMySound’s case, not providing real-time feedback is a conscious and deliberate design choice to avoid problems, such as disrupting the eye movement of the learner.
One of the main design considerations was to avoid giving binary judgments of correct or incorrect, instead favoring a more continuous approach that stresses musicality. Music education is historically based around apprenticeships where students learn to imitate their teachers before developing their own style. This tool follows the same tradition by trying to match what the student is playing to what the teacher played or the author published, going beyond mere pitches and rhythms to enter the area of expressive qualities of a musical performance. As such, it just reports the musical differences between the two recordings.
Another design consideration for the system was that it worked for as many musical instruments as possible. Considering the range of musical instruments tested, we believe it will work with all pitched instruments, including the most popular polyphonic instruments such as guitar and piano.
The robust nature of the audio matching algorithm makes it possible to use the technology in many educational settings. The main functionality can be displayed on any web page in an iFrame and connected to other learning apps through LTI. Its potential applications include interactive music games. As a proof of concept, we have incorporated one of them, Note Ahoy!, into our teacher tool.
Research papers. Käo, K., Niitsoo, M. MatchMySound: Introducing Feedback to Online Music Education. In Y. Cao, T. Väljataga, J. Tang, H. Leung, & M. Laanpere (Eds.), New Horizons in Web Based Learning ICWL 2014 (pp. 217-225). New York: Springer-Verlag.
Käo, K., Niitsoo, M. Optimizing the interaction between a self-learning guitar student and a sound recognition based educational game. CFMAE: The Changing Face of Music and Art Education Interdisciplinary Journal for Music and Art Pedagogy