Eligible models
All results obtained on challenge blind set (microsoft/PLC-Challenge (opens in new tab)), higher is better for all metrics.
| Rank | Team | PLCMOS | DNSMOS | CMOS | WAcc | Final Score |
|---|---|---|---|---|---|---|
| Clean (reference) | 4.51 | 3.89 | 0 | 0.97 | 0.98 | |
| 1 | Kuaishou | 4.28 | 3.8 | -0.55 | 0.88 | 0.85 |
| 2 | Amazon | 3.74 | 3.79 | -0.64 | 0.88 | 0.84 |
| 3 (tied) | Alibaba Inc. | 3.83 | 3.68 | -0.81 | 0.87 | 0.8 |
| 3 (tied) | Oldenburg University | 3.98 | 3.69 | -0.84 | 0.86 | 0.79 |
| 5 | Samsung | 3.28 | 3.51 | -1.1 | 0.86 | 0.75 |
| 6 | UCAS | 3.48 | 3.74 | -1.04 | 0.83 | 0.74 |
| Zero-filling baseline | 2.9 | 3.44 | -1.23 | 0.86 | 0.73 | |
| 7 | Shenzen University | 2.9 | 3.48 | -1.31 | 0.86 | 0.71 |
Ineligible models
| Team | PLCMOS | DNSMOS | CMOS | WAcc |
|---|---|---|---|---|
| GZHU-CUC | 4.406 | 3.958 | -0.279 | 0.854 |
| Augsburg University | 3.274 | 3.453 | -1.115 | 0.853 |
| VTCC | 3.371 | 3.293 | -1.284 | 0.852 |
Significance testing (eligible models)
Pairwise unpaired t-tests, no FWER correction.
Legend
- PLCMOS: Semi-Intrusive automated metric for packet loss concealment – PLC-Challenge/PLCMOS (opens in new tab)
- DNSMOS: Non-Intrusive automated metric for noise suppression – DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors
- CMOS: P.808 crowd source Comparison Category Rating – An Open source Implementation of ITU-T Recommendation P.808 with Validation
- WAcc: Word Accuracy (1 – Word Error Rate), calculated using Azure Cognitive Services
- Final Score: Mean of CMOS (range 0 to -3 normalized to 0 to 1) and WAcc