MAGE/pHTS in Collaborative Vocal Puppetry (CoVoP) - Mult-User Performative HMM-based Voice Synthesis on Distributed Platforms

Speech production is a complex phenomenon with many parameters. It is very difficult for one performer to control all aspects of a synthesizer that models this phenomenon. We designed and developed a distributed, multi-user system to tackle this difficulty, where users control different aspects of the synthesizer simultaneously and interactively; treating the complex production pro- cess as a social game. HMM-based synthesizers provide flexibility at a high level of naturalness, thus we chose HTS as our synthe- sizer. However, HTS needs severe architectural modifications to be used reactively, and a major achievement of this work was creating MAGE/pHTS, a library for performative HMM-based speech and singing synthesis. The resulting system provides interactive controls for phonetic content and context, as well as prosody using the previously existing HandSketch controller.