Piano Learning and Improvisation Through Digital Augmentation

The process of learning the piano for novices is usually tricky and time-consuming. Several approaches in augmented reality, such as piano-roll visualizations, have been explored but have not garnered enough success and adoption. These piano roll prototypes have introduced features and modules that assist novices in aspects such as sight-reading, timing and many others. The augmented piano has not yet been designed to provide a holistic approach to learning the instrument. We look at the overview of the existing landscape of augmented piano technologies.

In this PhD, we will explore how digital augmentation through adaptive visualisations and intelligent interactions can encourage piano novices to keep learning the piano.

We will design and build a piano roll training system integrated with adaptive visualisations using existing computational interaction and machine learning techniques. This will provide intervention support, thereby helping learners. We will evaluate and compare these visualisations in various user studies where they get to play piano pieces and develop their improvisation skills. We intend to uncover whether these adaptive visualisations will be helpful in the overall training of piano learners.

In this PhD we have formulated the following sub research questions to guide our method (UMAP DC 2021, MobileHCI DC 2021):

• RQ0: What other technological interventions have been introduced to support piano learning? ISS 2022 (forthcoming)
• RQ1: Can we build spatio temporal multi-target pointing models that understand piano users data during a learning improvisation task? ONGOING
• RQ2: From these models, what do we discover from how they learn piano improvisation? ONGOING
• RQ3: How do we guide the future design of the augmented piano towards a holistic piano learning experience? IMI 2022

To know more details in this dissertation, we invite you to read our existing publications (see below, PDF-available):

• Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning. UMAP DC 2021. PDF
• Encouraging Improvisation in Piano LearningUsing Adaptive Visualisations and Spatiotemporal Models. MobileHCI DC 2021. PDF.
• The Vision of a Human-Centered Piano. IMI 2022. PDF.
• A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?. (forthcoming ISS 22). PDF.