When applying voice AI to actual fields, there are situations we frequently encounter.

In call centers, there are only two people: the agent and the customer. What should we do when requested to analyze a transcript of a meeting with 8 or more participants?

High-performance E2E (End-to-End) speaker diarization models only support up to 4 people. From the 5th speaker onwards, they are either muted or grouped into another speaker. Pyannote, a clustering-based model, supports up to 10 people, but it has limitations in actual service application due to speed and performance in commercial environments. Consequently, an "E2E model that supports 8 or more people while securing speed and performance" did not exist. 

To solve this problem, MAGO MAGO started research, and we are now releasing the results.

🔎 What is Speaker Diarization?

Speaker diarization is a technology that automatically distinguishes "who spoke when." If STT is a 'dictation' that transcribes voice into text, speaker diarization is the process of labeling that text with names. Only when these two are combined does it truly gain value as 'conversation data.'

Only when the two technologies are combined can they be used in actual business scenarios. In call center consultation analysis, the agent's utterances and the customer's utterances must be separated to evaluate their respective emotional states or response quality. In meeting transcripts as well, the analysis loses much of its meaning if "who made this statement" is not tracked.

🧐 Why is 4 people the limit?

Most high-performance speaker diarization models available today are designed for a maximum of 4 speakers (4-speaker). This is also true for NVIDIA NeMo's Streaming Sortformer model.

The reason is structural limitations. The model requires as many output layers as the number of speakers, and increasing this requires modifying the architecture along with large-scale retraining. Most of all, it is extremely difficult to obtain actual multi-speaker training data.

Looking at the actual service environments of MAGO MAGO, cases exceeding 4 people are not uncommon.

• Corporate meetings, educational content production environments: 8 or more people

• Call center channel integration: Complex speaker composition including agents, customers, ARS, etc.

• Meeting transcription service: Environments where multiple participants speak simultaneously

The 4-speaker limit immediately becomes an issue in these scenarios.

🔈 Expanding the Number of Speakers While Maintaining Existing Performance

MAGO MAGO has developed a model that supports up to 8 people based on NVIDIA NeMo's Streaming Sortformer.

The two most difficult challenges were acquiring training data and maintaining existing performance.

The training data issue was resolved with synthetic data. Based on AI-Hub's multi-speaker speech synthesis data (more than 3,400 Korean speakers, approximately 10,000 hours in volume), we customized NeMo's data simulator to generate synthetic conversation data with various speaker counts and utterance patterns. We mixed in a small amount of data containing noise from actual meeting environments to prevent overfitting, which optimizes only for clean environments.

The issue of maintaining existing performance was resolved with two techniques.

First, instead of simply performing random initialization on the newly added output layers, we used SVD-based Orthogonal Initialization. After performing SVD decomposition on the existing weight matrix, we initialize the new layers in a direction that is mathematically orthogonal to the representation space of the existing speakers. This minimizes the impact of the newly added speaker dimensions on the recognition of the existing 2 to 4 speakers.

Second, we applied different learning rates (Split Learning Rate) to the existing layers and the new layers. The existing layers were designed to preserve pre-trained representations with a low learning rate (1e-5), while the new layers were designed to quickly learn the new speaker range with a high learning rate (1e-4).

🖇️ Discussion with the NVIDIA NeMo Speech Team

After releasing this model on HuggingFace, we were contacted by the NVIDIA NeMo Speech team. The research team in charge of speaker diarization and speech recognition discovered MAGO MAGO's model and proposed technical discussions. Since then, both companies have discussed evaluation methodologies, training data composition, and multi-speaker streaming optimization strategies together.

In this process, we also gained important insights regarding evaluation datasets. Some AMI RTTM files commonly found on the internet have silent sections incorrectly labeled as speakers during the original XML conversion process, which can distort evaluation metrics. We also confirmed that for accurate evaluation, the correct RTTM files from official distribution paths must be used. We plan to re-run the evaluation based on this standard and will update the results on the model card.

💡 Model Release Information

Ultra Diar Streaming Sortformer 8spk v1 (mago-ai/ultra_diar_streaming_sortformer_8spk_v1)

• Base Model: nvidia/diar_streaming_sortformer_4spk-v2.1

• Extension: 4 → 8 speakers

• License: Apache 2.0

• Available immediately on HuggingFace

• 
  

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "mago-ai/ultra_diar_streaming_sortformer_8spk_v1"
)
diar_model.eval()

predicted_segments = diar_model.diarize(audio=["your_audio.wav"], batch_size=1)

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "mago-ai/ultra_diar_streaming_sortformer_8spk_v1"
)
diar_model.eval()

predicted_segments = diar_model.diarize(audio=["your_audio.wav"], batch_size=1)

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "mago-ai/ultra_diar_streaming_sortformer_8spk_v1"
)
diar_model.eval()

predicted_segments = diar_model.diarize(audio=["your_audio.wav"], batch_size=1)

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "mago-ai/ultra_diar_streaming_sortformer_8spk_v1"
)
diar_model.eval()

predicted_segments = diar_model.diarize(audio=["your_audio.wav"], batch_size=1)

The training code, NeMo patches, and synthetic data tooling are also open to the public on GitHub (mago-research/Ultra-Sortformer).

📍 Next Step: Korean-Specialized Version

The model released this time is multilingual-based. In order to boost performance in the Korean environment, which is MAGO MAGO's core domain, we are preparing a Korean-specialized fine-tuned version.

This version reflects the characteristics of Korean conversation (short utterances, frequent interruptions, mixed use of Korean and English) into the training data, and we will announce the release schedule in the future.

📍 Technology to Extract Information from Voice

Speaker diarization is not a technology that is complete on its own. It is only when we identify what was said with STT, distinguish who said it with speaker diarization, and add speech emotion recognition (SER) and intent analysis that voice data can actually be utilized for analysis.

MAGO MAGO operates Audion, which provides this entire pipeline as a single API. This open-source release is intended to verify and improve the underlying technology together with the community.

If you are considering introducing multi-speaker voice analysis, or if you would like to discuss technical aspects further, please feel free to contact MAGO.

📌 View Model on HuggingFace · GitHub Code · Inquire About Audion API Integration