Recent diffusion-based generative models have achieved strong results in domain-specific audio generation tasks such as speech, singing, and instrumental music synthesis. However, these models are typically specialized and do not generalize well to mixed or intermediate audio types. In this work, we adapt a diffusion-based model originally designed for multi-instrument music synthesis to voice conversion, covering both speech and singing within a unified framework. Specifically, we extend musical note-based conditioning to include phonetic posteriorgrams (PPGs) and pitch contours, and reinterpret timbre conditioning as speaker or singer identity via feature-wise linear modulation. Experiments show that the adapted model matches or surpasses a dedicated voice conversion system in terms of naturalness and performer similarity, while maintaining accurate pitch control across speech and singing. At the same time, we observe limitations in phonetic fidelity and a degradation in vocal quality when incorporating instrumental training data. Furthermore, we demonstrate that off-the-shelf feature extractors provide effective conditioning signals, enabling large-scale self-supervised training without manual annotations. These results highlight the potential of cross-domain model transfer towards unified audio generation systems capable of handling speech, singing, and music.
The samples in this section are generated using our proposed attention-based T5-Voc model, trained on both speech and singing data.
The first column ("Source") presents real vocal excerpts that are converted across different singers.
Each subsequent column corresponds to a target singer, and the first row ("Target") provides a real reference excerpt for each.
Originally developed for instrumental music synthesis, this T5-based diffusion model has been effectively adapted for singing voice conversion.
The version conditioning mechanism—implemented using FiLM layers and initially designed for acoustic conditioning in instrumental music synthesis—can be repurposed to condition on singer identity.
Additionally, we use an off-the-shelf phonetic posteriorgram (PPG) extractor, trained only on speech, to condition the model on phonetic content.
Despite being trained exclusively on speech, the PPG extractor successfully captures the phonetic content of singing, which is preserved during singer conversion.
For a comparison with the FlowMAC model (based on MatchaTTS and ForwardTacotron), refer to the singing MUSHRA-like listening test section.
This section presents speech voice conversion samples generated by our proposed attention-based T5-Voc model, trained on both speech and singing data.
The first column ("Source") contains real speech excerpts, which are converted across target speakers shown in the subsequent columns.
The first row ("Target") provides a real reference excerpt for each target speaker.
As in the previous sections, the model leverages the FiLM-based version conditioning mechanism, originally designed for acoustic conditioning in instrumental music synthesis,
to effectively condition on speaker identity in the speech domain.
For a comparative evaluation with the FlowMAC model (based on MatchaTTS and ForwardTacotron),
please refer to the speech MUSHRA-like listening test section.
The samples in this section are from the speech MUSHRA-like listening test. Several models are evaluated: PAD-Voc: A voice conversion model based on ForwardTacotron, trained with a reconstruction loss. MAC-Voc: The FlowMAC model, trained with a flow-matching objective, and conditioned on the output of PAD-Voc. T5-Voc: Our proposed T5 attention-based diffusion model adapted from instrumental music synthesis to human voice conversion. T5-All: The same model as T5-Voc, but trained on a broader dataset including instrumental music and vocal-instrumental mixtures, in addition to pure vocal data.
The samples in this section are from the singing MUSHRA-like listening test. Several models are evaluated: PAD-Voc: A voice conversion model based on ForwardTacotron, trained with a reconstruction loss. MAC-Voc: The FlowMAC model, trained with a flow-matching objective, and conditioned on the output of PAD-Voc. T5-Voc: Our proposed T5 attention-based diffusion model adapted from instrumental music synthesis to human voice conversion. T5-All: The same model as T5-Voc, but trained on a broader dataset that includes instrumental music and vocal-instrumental mixtures, in addition to pure vocal data.
The samples in this section are taken from the singer similarity listening test. In this test, the same source excerpt is rendered using two different voices:
the target singer and a randomly sampled other singer. Listeners are asked to rate the similarity to the target singer.
We compare two models: T5-Voc and MAC-Voc, resulting in four converted samples per source excerpt. For each converted sample,
we also provide a reference excerpt (of different content) from the target singer, to guide the similarity judgment.
The samples in this section are generated using the attention-based T5-All model, trained on a combined dataset of vocal and instrumental music.
The instrumental content was derived from automatically transcribed piano rolls.
For vocal-only samples generated by this model, refer to the MUSHRA-like listening test sections for speech and singing.
While this paper focuses on the evaluation of vocal synthesis, we plan to explore the generation of vocal-instrumental mixtures more extensively in future work.
.jpg){kind=link}