How it works

From mono waveform to source count, sound labels, quality-checked stems, and honest held-out limits.

The problem we solve

Given a single-channel (mono) recording with multiple overlapping sounds, estimate how many distinct sources are active, name what they are, and separate sparse speech into individual stems.

Why count before separating?

Most audio separators assume you already know how many speakers or instruments are talking at once (for example, fixed 2-speaker or 3-speaker models). In real rooms, the count changes from clip to clip. Counting first tells downstream tools whether separation is even feasible.

Why mono audio is hard

With two or more microphones you can use spatial direction-of-arrival. With a single channel, all sources sum into one pressure wave: overlapping harmonics, reverberation, and loudness differences mask quieter sounds as density grows.

Where this matters

Acoustic scene monitoring, meeting and classroom analytics, hearing-aid pre-processing, and dataset curation all need reliable source-density estimates before spending compute on separation or transcription.

The data

EchoCount is trained and evaluated on deduplicated mixtures from the SANADA Combined Acoustic & Speech corpus plus an isolated FSD50K benchmark subset.

0active deduplicated mixtures (15757 raw)
0within-folder duplicates excluded
2–31overlapping sources
0recognised sound classes
Split familyTrainValidationTestTotal
FSD acoustic mixtures5,9905905907,170
Speech mixtures (13 speakers, 3 held-out)6,9996956958,389
  • SHA-256 deduplication: 198 within-folder duplicate mixtures and 1 entire duplicate folder (Org_Fsd_Mix / Org_Mix_4_Sounds_Variable) are excluded before splitting so identical waveforms never leak across train, val, and test.
  • Speaker-disjoint evaluation: 3 of the 13 speakers in the speech corpus are held out completely to measure generalization to unseen voices.
  • Taxonomy: 5 coarse acoustic groups (Human voice, Musical instrument, Animal, Household/tools, Environment/other), 32 fine FSD classes, and 527 AudioSet classes (559 recognised classes total).

The full pipeline, step by step

Every uploaded clip passes through six deterministic stages on the server. No stage invents labels or stems when a model is missing or a quality gate fails.

The model reads a picture of sound

Audio becomes a 64-band log-mel spectrogram (64 × 501 = 32,064 cells): time runs left to right, pitch runs bottom to top, and brightness shows energy.

Illustration, not your audio
  1. Stage 01

    Decode & Window

    Convert to mono at 16 kHz. Clips longer than 8 s are centre-cropped to 128,000 samples (8.0 s) for counting; shorter clips are zero-padded. Original sample rate is preserved for stem export.

  2. Stage 02

    Log-Mel Spectrogram

    STFT with 1,024-sample Hann window (64 ms), 256-sample hop (16 ms), and 64 mel bands (40 Hz–8 kHz), producing a 64 × 501 time-frequency grid (32,064 cells).

  3. Stage 03

    Source Counting + MC-Dropout

    SourceCountCNN (2–31) and a sparse specialist (2–8) run 30 stochastic MC-Dropout forward passes (p = 0.4), calibrated via isotonic regression fitted on validation data.

  4. Stage 04

    3-Stage Open-Vocabulary Tagging

    GroupFineTaggerCNN scores 5 groups and 32 fine classes; AST scores 527 AudioSet classes; CLAP (HTSAT-Unfused) provides zero-shot audio-text matching and frame-level time ranges.

  5. Stage 05

    Conditional Separation

    Pure 2-speaker speech routes to pretrained SepFormer (WSJ0-2mix) + ECAPA-TDNN speaker verification. General mixtures only separate when a trained general checkpoint is active, plus a residual stem (mix − sum of stems).

  6. Stage 06

    Three Quality Gates

    Every separated stem is checked for near-copy correlation, energy conservation against the mixture, and self-consistency (re-tagging the isolated stem to confirm it still matches its label).

Inside SourceCountCNN

Conv 11 to 16 channels, BatchNorm, ReLU, max-pool
Conv 216 to 32 channels, BatchNorm, ReLU, max-pool
Conv 332 to 64 channels, BatchNorm, ReLU
Adaptive poolAny input size becomes a 4 × 16 map
Dense 256Flatten, linear, ReLU, MC-dropout 0.4 (30 passes)
Dual Head30 count classes + continuous regressor + isotonic calibration
Illustration of the analysis interval
0 s2 s4 s6 s8 s

One full 8-second clip produces one count estimate.

Quality gates & residual verification

1. Near-copy correlation

Flags stems whose waveform correlation exceeds 0.9 or speaker cosine exceeds 0.82 (measured 2-spk speech pairwise corr: 0.1179).

2. Energy conservation

Checks that stems + residual reconstruct mixture energy within ±5% relative L2 tolerance (pass rate: 100%).

3. Self-consistency re-tagging

Re-runs the tagger on each separated stem to verify the target sound still dominates (2-spk speech pass rate: 100%; synthetic FSD: 39.3%).

How we train

Training uses the train and validation splits only; the test split is locked until final evaluation.

  • Dual-head objective: Smooth L1 regression loss + Cross-Entropy classification loss over integer counts, balancing exact classification with ordinal closeness.
  • Optimization: AdamW with CosineAnnealingLR, SpecAugment time/frequency masking, and class-balanced sampling so rare source counts are not drowned out.
  • Early stopping & calibration: Best checkpoint is selected by validation macro MAE. Isotonic calibration and sparse-specialist blending are fitted strictly on validation predictions—never on test.

Active counter checkpoint: SourceCountCNN-v3-speech-fsd-specialist · 2,153,989 total parameters (8.24 MB) · best val epoch 11 · val MAE 1.6047 · val exact 45.0%.

How we evaluate (the metrics, explained)

All numbers on this site are loaded directly from evaluation/public_summary.json, exported from real held-out test runs.

MetricFormulaWhat it means
Exact Accuracymean(y_pred == y_true)Fraction of clips where the rounded integer count matches the exact ground-truth source count.
Within ±1 Accuracymean(|y_pred − y_true| <= 1)Fraction of clips within one source of truth—useful when dense mixtures overlap heavily.
MAE & BiasMAE = mean(|y_pred − y_true|), Bias = mean(y_pred − y_true)Average absolute error in sources, and whether the model systematically under-counts (negative) or over-counts (positive).
Bootstrap 95% CI1,000 resamples with replacementUncertainty interval around test metrics; bins with n < 30 are flagged unreliable.
Macro mAP & F1mean_c AP(c)Unweighted mean Average Precision across classes so frequent sounds cannot hide weak rare classes.
SI-SDRi (dB)SI-SDR(stem, target) − SI-SDR(mix, target)Scale-Invariant Signal-to-Distortion Ratio improvement in decibels over the unseparated mixture.

Per-count exact accuracy (reliable bins n ≥ 30)

Confusion matrix (key source counts K = 2, 3, 4, 6, 9)

Rows are true source counts; columns are predicted source counts. Notice how K=9 collapses below 9.

True \ Pred23469other_<=8>9Exact %n
K=2 193 53 2 0 0 0 0 77.8% 248
K=3 14 181 43 4 0 5 1 73.0% 248
K=4 2 24 80 11 0 30 1 54.0% 148
K=6 0 2 3 81 0 27 2 70.4% 115
K=9 0 0 0 72 0 39 4 0.0% 115

What we found (honest results)

EchoCount works well on sparse mixtures of familiar voices and everyday sounds, and degrades measurably on dense overlaps and unseen speakers.

BenchmarkSample sizePrimary scoreNotes
Counting · Overall Testn=1,28543.8% exact · 64.3% ±1MAE 1.6451 (95% CI 1.5198–1.7673)
Counting · Speech Testn=69553.1% exact · 78.8% ±1MAE 0.8014 · bias -0.3669
Counting · FSD Testn=59032.9% exact · 47.1% ±1MAE 2.639 · bias -0.761
Counting · Unseen Speakersn=352 (3 spk)13.1% exact · 29.5% ±1MAE 2.054 · speaker generalization limit
Tagging · 5 Groups (Test)n=5900.9926 macro mAPVal macro mAP 0.9948
Tagging · 32 Fine Classes (Test)n=5900.8052 macro mAPMacro F1 0.7612
Tagging · Open-Vocab (AST+CLAP)n=96 (16 cls)0.9068 macro APCombined open tagger AP 0.6979
Separation · Synthetic Query Bench32 mixes / 112 stems+2.51 dB SI-SDRiSelf-consistency 39.3% · Energy 100%

Dense mixture saturation (K ≥ 9)

Exact accuracy is strong at K=2 (77.8%), K=3 (73.0%), and K=6 (70.4%), then collapses to 0.0% at K=9 (bias −2.41 sources) as spectral overlap saturates the 64-band log-mel grid.

Unseen-speaker generalization gap

On the 3 held-out speakers (n=352), exact accuracy drops from 53.1% (overall speech) to 13.1% (MAE 2.05), showing the CNN partially memorizes training-set speaker timbres.

General non-speech separation

The synthetic query-separation benchmark averages +2.51 dB SI-SDRi across 16 classes, but only 39.3% of stems pass self-consistency re-tagging—so general separation is disabled in production unless a trained separator checkpoint is loaded.

Interesting facts & insights

32,064Time-frequency cells (64 × 501) compressed into one count
198Exact-hash duplicate mixtures removed before train/val/test splitting
13.1% vs 53.1%Unseen-speaker exact accuracy vs. overall speech test
-2.41Source bias at K=9 where spectral overlap saturates
1.739 sEnd-to-end 8 s analysis latency on Windows CPU (AMD64, PyTorch 2.15.0+cpu)
559Total sound classes across AudioSet (527) + FSD domain (32)

What to keep in mind

Counting is not separating

Estimating K sources does not automatically produce K clean audio files. Real separation is currently enabled for 2-speaker speech via SepFormer WSJ0-2mix.

Long files are centre-cropped

Counting analyses the central 8 seconds (16 kHz mono). Shorter files are zero-padded. The Analyze waveform marks the exact analysed region.

Uncertainty is not accuracy

MC-Dropout spread and softmax confidence show model decisiveness, not ground-truth correctness—especially on unseen voices or dense scenes.

Automatic server cleanup

Uploaded audio, generated stems, and ZIP archives are stored in isolated job folders and deleted automatically after the configured TTL and on server startup.

Reproducibility

Every number on this page can be reproduced from the repository CLI:

python scripts/download_models.py
python scripts/train.py all --full
python scripts/evaluate.py --export-public
python tests/golden_check.py --verify
python test_templates.py --full

References

Fonseca et al. (2022)

FSD50K: An Open Dataset of Human-Labeled Sound Events — IEEE/ACM TASLP · Source taxonomy & isolated everyday sound benchmark.

Gong et al. (2021)

AST: Audio Spectrogram Transformer — Interspeech · 527-class AudioSet backbone used in stage-2 tagging.

Elizalde et al. / Wu et al. (2023)

CLAP: Contrastive Language-Audio Pretraining — ICASSP · Zero-shot audio-text matching and open-vocabulary queries.

Subakan et al. (2021)

Attention Is All You Need In Speech Separation (SepFormer) — ICASSP · Pretrained WSJ0-2mix speech separator for 2-speaker mixtures.

Desplanques et al. (2020)

ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation — Interspeech · 192-D speaker embeddings for stem verification and clustering.

Gal & Ghahramani (2016)

Dropout as a Bayesian Approximation — ICML · Theoretical basis for 30-pass MC-Dropout count uncertainty.