1. Stream Opening and Observation
typedef struct mic_open_params {
unsigned int sample_rate_hz;
unsigned int channels;
unsigned char require_sync;
unsigned char remove_sync_marker;
unsigned char interpolate_sync_marker;
unsigned int read_block_frames;
} mic_open_params_t;
typedef struct mic_audio_block {
short *interleaved;
unsigned int frames;
unsigned int channels;
unsigned int sample_rate_hz;
unsigned long long sample_index;
unsigned long long monotonic_time_ns;
unsigned int flags;
} mic_audio_block_t;
typedef struct mic_stream_observation {
unsigned int reported_track_sample_rate_hz;
unsigned int effective_read_sample_rate_hz;
unsigned int reported_channel_count;
unsigned int effective_channel_count;
unsigned char inferred_ssl_mode;
unsigned char inferred_highspeed_mode;
} mic_stream_observation_t;
mic_status_t mic_open_recording(mic_device_t **devices, size_t device_count, const mic_open_params_t *params, mic_stream_t **out_stream);
mic_status_t mic_open_stream(mic_device_t *dev, const mic_open_params_t *params, mic_stream_t **out_stream);
mic_status_t mic_open_stream_group(mic_device_t **devices, size_t device_count, const mic_open_params_t *params, mic_stream_t **out_group_stream);
mic_status_t mic_start_stream(mic_stream_t *stream);
mic_status_t mic_stop_stream(mic_stream_t *stream);
mic_status_t mic_read_stream(mic_stream_t *stream, mic_audio_block_t *out_block, unsigned int timeout_ms);
mic_status_t mic_get_stream_observation(mic_stream_t *stream, mic_stream_observation_t *out_obs);
void mic_close_stream(mic_stream_t *stream);
1-1. Inheritance and Dispatch Policy
| Operation Class |
Base Dispatch |
Derived Implementation |
| Open / close | module_ops_t | mic, camera, sm3324 each implement their own backend open / close path |
| Stream creation | module_ops_t | mic uses audio PCM stream, camera uses video frame stream, sm3324 uses CDC sensor frame stream |
| Get capability | module_ops_t | Every derived module reports its feature bits through the same base query API |
| Sync participation | module_ops_t | mic and camera can later share a common synchronized timestamp framework |
| Rule |
Applies To |
Policy |
| 1ch highspeed | 192k / 384k | Each device emits its floating fifth MEMS plus in-band sync markers. Firmware SSL is unavailable; Host SSL is supported with at least four synchronized 5-MEMS devices. |
| 2ch / 6ch | 24k / 48k / 96k | Allowed only at the documented standard rates |
| Multi-device capture | Group open | Must fail unless sync is required and roles contain exactly one MASTER plus only FOLLOWER devices |
| GPIO conflict | Sync mode | If sync is enabled, GPIO APIs must either fail or disable GPIO mode first |
| Marker removal | Synced audio | Default host-visible PCM should have sync marker removed and interpolated |
| Host arrival offset | USB UAC layer | Packet arrival timing / ordering must not be used as sync reference; only in-band sync markers are authoritative |
| Multi-UAC servicing | Group stream | Drain every endpoint in one bounded nonblocking polling cycle; sequential full-block reads are forbidden because unserviced WASAPI buffers overflow |
| Clock drift | Between sync pulses | 1-2 sample mismatch may accumulate and may be corrected by small drop / pad alignment at sync boundaries |
| Marker acquisition / loss | Online sync | Current group capture uses a 248 ms firmware cadence: 20 ms x10, then 24 ms x2. Initial lock finds the 20 ms -> special marker -> 24 ms pattern independently for each master-follower pair. After lock, each pair updates only from its next common marker; a missing pair marker retains that pair offset, and three seconds without a pair update returns only that pair to initial-lock search. |
2. SSL, VAD, and STT Extension API
typedef struct mic_ssl_result {
int source_id;
unsigned long long time_ms;
float azimuth_deg;
float elevation_deg;
float center_hz;
float activity;
float level_db;
} mic_ssl_result_t;
typedef enum mic_ssl_backend {
MIC_SSL_BACKEND_SRP_EXPERIMENTAL = 0,
MIC_SSL_BACKEND_MUSIC_HARK_COMPAT = 1,
MIC_SSL_BACKEND_TOPS_EXPERIMENTAL = 2,
MIC_SSL_BACKEND_CSSM_EXPERIMENTAL = 3,
MIC_SSL_BACKEND_ADAPTIVE_EXPERIMENTAL = 4
} mic_ssl_backend_t;
typedef enum mic_ssl_music_algorithm {
MIC_SSL_MUSIC_ALGORITHM_SEVD = 0,
MIC_SSL_MUSIC_ALGORITHM_GEVD = 1,
MIC_SSL_MUSIC_ALGORITHM_GSVD = 2
} mic_ssl_music_algorithm_t;
typedef enum mic_ssl_window_policy {
MIC_SSL_WINDOW_POLICY_PAST = 0,
MIC_SSL_WINDOW_POLICY_MIDDLE = 1,
MIC_SSL_WINDOW_POLICY_FUTURE = 2
} mic_ssl_window_policy_t;
typedef enum mic_ssl_spectrum_weight {
MIC_SSL_SPECTRUM_WEIGHT_UNIFORM = 0,
MIC_SSL_SPECTRUM_WEIGHT_A_CHARACTERISTIC = 1
} mic_ssl_spectrum_weight_t;
typedef enum mic_ssl_peak_search {
MIC_SSL_PEAK_SEARCH_LOCAL_MAXIMUM = 0,
MIC_SSL_PEAK_SEARCH_HILL_CLIMBING = 1
} mic_ssl_peak_search_t;
typedef struct mic_ssl_params {
mic_ssl_backend_t backend;
unsigned int fft_size;
unsigned int max_sources;
unsigned int accuracy_angle_deg;
unsigned int azimuth_step_deg;
unsigned int elevation_step_deg;
mic_ssl_music_algorithm_t music_algorithm;
unsigned int correlation_window_frames;
mic_ssl_window_policy_t correlation_window_policy;
unsigned int ssl_period_frames;
mic_ssl_spectrum_weight_t spectrum_weight_type;
float a_characteristic_scaling;
unsigned char enable_eigenvalue_weight;
unsigned char enable_hierarchical_refine;
mic_ssl_peak_search_t peak_search_algorithm;
float min_freq_hz;
float max_freq_hz;
float speed_of_sound_mps;
float steering_distance_m;
float max_source_angular_speed_deg_s;
float noise_floor_dbfs_per_sqrt_hz;
float min_signal_snr_db;
} mic_ssl_params_t;
typedef struct mic_vad_params {
unsigned int mode;
unsigned int frame_ms;
float speech_ratio_threshold;
} mic_vad_params_t;
typedef struct mic_vad_result {
unsigned char is_speech;
float speech_probability;
unsigned long long time_ns;
} mic_vad_result_t;
typedef enum mic_stt_model_preset {
MIC_STT_MODEL_PRESET_CUSTOM_PATH = 0,
MIC_STT_MODEL_PRESET_LIGHT_MULTILINGUAL,
MIC_STT_MODEL_PRESET_HEAVY_MULTILINGUAL
} mic_stt_model_preset_t;
typedef struct mic_stt_params {
mic_stt_model_preset_t model_preset;
const char *model_path;
const char *model_dir;
const char *language;
unsigned int n_threads;
unsigned char translate;
unsigned char detect_language;
unsigned char use_gpu_if_available;
unsigned char enable_post_correction;
const char *post_correction_python_exe;
const char *post_correction_script_path;
} mic_stt_params_t;
typedef struct mic_stt_realtime_params {
unsigned int step_ms;
unsigned int stable_decode_count;
} mic_stt_realtime_params_t;
mic_status_t mic_read_device_ssl(
mic_device_t *dev,
mic_ssl_result_t *items,
size_t capacity,
size_t *out_count,
unsigned int timeout_ms);
mic_status_t mic_create_ssl(const mic_ssl_params_t *params, mic_ssl_t **out_ssl);
mic_status_t mic_ssl_attach_stream(mic_ssl_t *ssl, mic_stream_t *stream, const mic_device_geometry_t *geometry);
mic_status_t mic_ssl_process_bytes(mic_ssl_t *ssl, const dsp_source_t *src, mic_ssl_result_t *items, size_t capacity, size_t *out_count);
mic_status_t mic_ssl_process_live(mic_ssl_t *ssl, mic_device_t *mic, unsigned int run_ms, mic_ssl_result_t *items, size_t capacity, size_t *out_count);
mic_status_t mic_create_vad(const mic_vad_params_t *params, mic_vad_t **out_vad);
mic_status_t mic_vad_process_bytes(mic_vad_t *vad, const dsp_source_t *src, mic_vad_result_t *out_result);
mic_status_t mic_vad_process_live(mic_vad_t *vad, mic_device_t *mic, unsigned int run_ms, mic_vad_result_t *out_result);
mic_status_t mic_stt_resolve_model_preset(mic_stt_model_preset_t preset, const char *model_dir, char *out_path, size_t capacity);
mic_status_t mic_create_stt(const mic_stt_params_t *params, mic_stt_t **out_stt);
mic_status_t mic_stt_set_language(mic_stt_t *stt, const char *language);
mic_status_t mic_stt_set_post_correction_enabled(mic_stt_t *stt, unsigned char enable);
mic_status_t mic_stt_set_post_correction_python_exe(mic_stt_t *stt, const char *python_exe);
mic_status_t mic_stt_set_post_correction_script_path(mic_stt_t *stt, const char *script_path);
mic_status_t mic_stt_set_realtime_params(mic_stt_t *stt, const mic_stt_realtime_params_t *params);
mic_status_t mic_stt_feed_bytes(mic_stt_t *stt, const dsp_source_t *src);
mic_status_t mic_stt_run_live(mic_stt_t *stt, mic_device_t *mic, unsigned int run_ms);
mic_status_t mic_stt_process_realtime_bytes(mic_stt_t *stt, const dsp_source_t *src, char *confirmed_utf8, size_t confirmed_capacity, size_t *out_confirmed_len, char *unstable_utf8, size_t unstable_capacity, size_t *out_unstable_len);
mic_status_t mic_stt_process_realtime_live(mic_stt_t *stt, mic_device_t *mic, unsigned int run_ms, char *confirmed_utf8, size_t confirmed_capacity, size_t *out_confirmed_len, char *unstable_utf8, size_t unstable_capacity, size_t *out_unstable_len);
mic_status_t mic_stt_finalize_realtime(mic_stt_t *stt, char *confirmed_utf8, size_t confirmed_capacity, size_t *out_confirmed_len);
mic_status_t mic_stt_flush(mic_stt_t *stt, char *utf8_text, size_t capacity, size_t *out_len);
Required processing model: SSL, VAD, and STT must all be callable in two ways.
First, with recorded audio bytes through dsp_source_t.
Second, with a live MIC handle, in which case the function itself performs direct real-time processing against the device.
Backend selection note:
the SDK surface is now being shaped after HARK LocalizeMUSIC.
Public parameters therefore already need to express
MUSIC_ALGORITHM, WINDOW, WINDOW_TYPE,
SPECTRUM_WEIGHT_TYPE, ENABLE_EIGENVALUE_WEIGHT,
and PEAK_SEARCH_ALGORITHM.
The SDK now contains both the native SRP path and a separate
MIC_SSL_BACKEND_MUSIC_HARK_COMPAT path.
In the current revision, that HARK-compatible path is limited to
SEVD + host free-space steering + HARK-style peak search / weighting controls.
GEVD / GSVD and explicit noise-reference covariance are still pending.
Performance target policy: the HARK-compatible backend is the reference baseline, not the final ceiling.
If a stronger host pipeline is needed, additional robustness features may be added on top, but they must stay clearly separated
from the pure HARK-compatible path so benchmark attribution stays honest.
VAD uses libfvad because it is lightweight, proven, and easy to embed in a cross-platform static C SDK.
VAD and STT normalize input to 16 kHz mono inside the SDK before inference.
Known integer-rate MIC0528 inputs use the stateful decimation path with anti-alias filtering; callers do not need to resample first.
Before STT inference, STT applies a continuous envelope normalizer: a rising envelope is applied immediately,
its falling edge uses a 0.5 Hz release, the denominator is floored at 10% full scale, and output is held to a 50% peak target.
STT uses parakeet.cpp with an external GGUF model selected either by direct path or preset.
The recommended shipped presets are Nemotron 3.5 ASR Streaming 0.6B q4_k for multilingual realtime use
and Parakeet realtime EOU 120M q4_k for a lighter streaming endpointing model. Both run through the same
SDK C API and both accept explicit language selection or auto mode. On the July 24, 2026 validation branch, the default
realtime operating point is Nemotron streaming + CUDA if available + step_ms=1000.
Realtime STT exposes two strings: confirmed text that will not be rewritten and unstable text that may still change as more
audio arrives. Confirmed text is monotonic. The SDK keeps resampler and normalization state across realtime feeds so step
boundaries do not reintroduce sample-phase drift or level pumping.
If the CUDA backend is built and an NVIDIA GPU is present, the parakeet backend is allowed to use GPU execution.
STT post-correction is now part of the SDK contract. The C API exposes
enable_post_correction, post_correction_python_exe, and
post_correction_script_path inside mic_stt_params_t,
plus setter functions for runtime changes. The default backend is the Python tool
sdk/tools/stt_post_correct.py, which performs word-level correction for
ko/en/ja/zh. It uses symspellpy + wordfreq as the
general lexicon/scoring path, jieba for Chinese tokenization, and an optional
phonetic_mlm pass for Chinese confusion repair when that repository is present.
On Windows the SDK runs this backend through a temp-file based CreateProcess path so UTF-8 text,
quoting, and non-ASCII output stay deterministic. The example dashboards now enable this backend
by default and allow deployment-time override through the environment variables
MIC_STT_POST_CORRECT_PYTHON and MIC_STT_POST_CORRECT_SCRIPT.
If later testing shows that false positives or robustness are insufficient, Silero VAD is the preferred optional upgrade path.
Internal signal path note:
the default design assumption is that higher-level learned or frequency-aware processing stages operate after an internal
STFT transform. Even when the caller provides raw PCM bytes or a live mic handle, the SDK may internally
construct STFT frames before running classification, host-side SSL feature extraction, or auxiliary decision logic.
VAD and STT policy: processing is one channel per microphone, not one device total.
Multiple microphones may be processed in parallel, but each microphone contributes its own mono input and produces its own VAD or STT result.
If a microphone stream is multichannel, the API should either require an explicit per-microphone channel selection rule or reject the request.
Round-table realtime transcription policy:
angle-only SSL can be sufficient if the STT side emits utterance chunks and the host averages the SSL angle outputs over the same utterance time span.
In that design, the host reports "average angle during this utterance" together with the transcript text, without requiring per-speaker beamformed audio from SSL itself.
Host SSL input constraints:
SSL must operate on synchronized 6-channel SSL-mode recordings.
If more than one MIC0528 device participates, multi-device recording must reject the request unless exactly one device is explicitly configured as MASTER and all others as FOLLOWER. Firmware auto-election is forbidden because simultaneous push-pull masters contend on the shared wire.
For each 6-channel device block, the effective SSL channels are:
4-mic variant -> ch1-4 only,
5-mic variant -> ch1-5 only,
and ch6 is excluded because it is the subwoofer / low-frequency support channel. In practice, bytes-mode host SSL should first call mic_ssl_attach_stream() so the SSL object can derive this mapping from the opened recording stream.
Host SSL geometry rule:
the SDK derives absolute sensor positions as device_origin_m +
Rz(yaw) * Ry(pitch) * Rx(roll) * local_mems_offset. Rotation fields are degrees in a
right-handed coordinate system. The dashboard position grammar is id=x,y,z,roll,pitch,yaw;
the legacy three-value form id=x,y,z remains accepted as 0,0,0 rotation.
In 6ch mode, two 5-mic devices therefore yield 10 effective sensor positions; if one or both lack the floating microphone, the effective count drops accordingly.
In 1ch 192/384 kHz mode, each device contributes only channel 0 mapped to local mic_pos_m[4].
Host SSL therefore requires at least four synchronized MIC0528 devices with the floating fifth MEMS, and a four-device group produces four SSL input channels.
If the resulting effective sensor positions are coplanar, the output space is treated as a hemisphere.
If they are non-coplanar, the output space is treated as a full sphere.
Host SSL motivation and target:
the device-emitted SSL packets are useful for lightweight telemetry, but PC-side SSL exists because the on-device SSL is intentionally accuracy-limited by compute budget.
The host-side SSL path is therefore the accuracy-oriented path, and it must support multiple simultaneous sources.
A sound-camera application consumes mic_ssl_process_probability_map_bytes(). The returned grid is a conditional spatial posterior: probabilities sum to 1.0 given one active source. It is not an absolute source-presence probability, and the returned grid-cell values are not latitude/elevation cell-area weighted. SSL always consumes the original PCM amplitude and never applies the STT envelope/peak normalization. The noise gate follows amplitude spectral density: configured-band noise RMS grows with sqrt(bandwidth_hz). A zero noise_floor_dbfs_per_sqrt_hz selects -120 dBFS/sqrt(Hz), and a zero min_signal_snr_db selects 10 dB. A frame is active only when band_level_dbfs - 10*log10(bandwidth_hz) reaches their sum. Below it, both SSL APIs return no source, the probability-map API returns out_count=0 and out_center_hz=0, and the UI must clear stale contours and markers. Realtime hosts retain a 100 ms analysis window and advance it by 25 ms (75% overlap); source tracks may emit after the first accepted frame, and sound-camera maps update every 200 ms from the same 100 ms window. After the first frame, the alpha-beta direction tracker limits correction using elapsed time, max_source_angular_speed_deg_s, and one grid step so that a transient wrong peak cannot produce a large angular jump. The dashboard derives 50%, 80%, and 95% highest-posterior-density contour boundaries from an active map. Probability controls contour geometry and opacity only; contour color is derived from out_level_db, and the crosshair label uses the measured out_center_hz. The sound-camera crosshair is bound to the argmax cell of that same posterior frame. Do not position it from mic_ssl_process_bytes(), because that API returns a temporally tracked and smoothed direction which need not coincide with the instantaneous posterior maximum. Firmware/MIC SSL has no full posterior map, so contours are Host SDK SSL only.
Implementation correction: the map no longer min/max-normalizes an SRP/MUSIC score or applies a temperature softmax. It evaluates the one-plane-wave-plus-noise complex covariance profile likelihood per direction and selected STFT bin, sums the bin likelihood ratios, and normalizes only that resulting likelihood over the grid. posterior_temperature remains in the public signature solely for ABI compatibility and must be passed as zero.
Coplanar elevation convention: a coplanar 4-MEMS layout cannot distinguish opposite sides of its plane. The SDK returns the canonical lower hemisphere, -90 to 0 degrees. Visualization mirrors the same posterior and its exact argmax into the positive hemisphere; it does not run or blend a second localization estimate. The crosshair, glow, and HPD contours are all derived from the same normalized posterior array and frame.
Current research direction for host SSL:
repeated tuning of the internal MUSIC path is producing only incremental gains on AV16.3 style reverberant speech.
Therefore the next priority is not more threshold tweaking but direct comparison against stronger non-learning open-source baselines.
The main carry-in candidates are pyroomacoustics NormMUSIC / TOPS / WAVES / FRIDA.
Among them, TOPS and WAVES are the first practical port targets for speech-oriented robustness,
while FRIDA remains the higher-resolution candidate if computation and implementation complexity are acceptable.
The HARK-compatible path remains valuable as a validated reference path, but it should no longer be assumed to be the final accuracy path if open benchmark results keep trailing the target.
External baseline status:
the MIT-licensed LCAV/FRIDA repository has been brought into local evaluation because it bundles
MUSIC, WAVES, TOPS, and FRIDA under a permissive license.
On the corrected AV16.3 one-second combined16 clip, the first directly runnable imported result is currently
TOPS, which outperforms the imported MUSIC and WAVES paths on that clip.
The imported FRIDA implementation itself is still treated as a secondary follow-up because the repository version is primarily a
planar azimuth solver, while the present SDK benchmark loop is comparing full 3D directions.
Current TOPS parity checkpoint:
after splitting TOPS from the MUSIC normalization path, feeding it raw covariance, and evaluating on clipped one-second AV16.3 windows,
the host-side C TOPS path is now tracking the imported FRIDA TOPS baseline closely on the available local sequence.
For seq45-3p-1111 combined16 at 4 deg / 4 deg and 1024 FFT,
the current five-window clip mean is approximately 27.71 deg for the SDK C path versus
27.57 deg for imported FRIDA TOPS over the 15 s to 19 s windows.
The comparison path is now unified in ssl_wav_eval itself:
for backend=tops, passing analysis_window_frames=0 makes the tool evaluate the entire selected clip as one batch,
so the benchmark script no longer needs a separate ssl_grid_dump bypass for one-second parity checks.
This does not yet prove wider benchmark sufficiency, but it does show that the present C TOPS port is now close to the chosen permissive open-source reference on the locally available clip set.
CSSM research checkpoint:
imported FRIDA CSSM is a promising non-learning comparison target because on the same local
seq45-3p-1111 combined16 15 s to 16 s one-second clip it currently reaches about
8.54 deg, versus imported FRIDA TOPS at about 21.07 deg.
The current investigation has now closed one important gap:
a direct covariance dump comparison shows that the SDK raw covariance and the imported FRIDA covariance are structurally very close,
but differ by a large near-constant scale factor introduced by the differing STFT normalization paths.
A dedicated internal dump tool and comparison script were added so this can now be measured directly instead of inferred indirectly from final peaks.
After aligning the internal CSSM candidate grid with the imported FRIDA theta/phi convention, the host-side C CSSM branch now reproduces the same top candidate family on the local parity clip
(for example, 312 deg / 22 deg as the leading peak on the current 15 s to 16 s combined16 test window).
A later parity pass clarified one non-obvious implementation detail:
imported FRIDA CSSM aliases the initial self.src_idx array only in the single-source path.
In the multi-source path _peaks1D() replaces it with a new list, so each frequency keeps its own initial estimate.
The explicit CSSM backend follows that behavior. Adaptive mode deliberately retains a shared multi-frequency seed because it is more
stable on the current reverberant multi-source subset; this is a documented product heuristic rather than a claim of FRIDA parity.
With that detail restored, the current internal CSSM parity executable now matches the imported FRIDA peak family across the checked
15 s through 19 s one-second combined16 windows:
312/22, 332/42, 352/50, 344/74, and 8/66 degrees.
Extending that check to a denser twelve-window sweep from 15.00 s through 20.28 s gives
7 exact peak matches and 5 elevation-sign-only differences, with 0 residual azimuth-family mismatches.
Because the current AV16.3 combined16 array is planar, those remaining sign-only differences are consistent with hemisphere / mirror ambiguity rather than a remaining CSSM parity break.
The standalone evaluator and public mic_create_ssl() API both accept
MIC_SSL_BACKEND_CSSM_EXPERIMENTAL and
MIC_SSL_BACKEND_ADAPTIVE_EXPERIMENTAL.
A later benchmark-audit pass found two evaluation-side pitfalls that had been overstating some local CSSM numbers:
first, the benchmark helper had been collapsing moving-source full-clip output to a single mean-direction score, which is not the same thing as point-in-time localization accuracy;
second, the helper's topk argument also drives the backend's assumed source count, so running a single-source clip with topk=3 was unintentionally evaluating CSSM as a three-source problem.
The benchmark helper now prints both summary_mode and truth_max_active_sources so this mismatch is visible.
On the current local seq45-3p-1111 combined16 15 s to 16 s one-second clip, the corrected customer-path helper gives about 3.57 deg for SDK CSSM with assumed_num_sources=1,
versus about 8.54 deg for the imported FRIDA CSSM mean-direction reference on the same clipped audio.
The remaining work is therefore no longer coarse parity recovery; it is API-promotion hardening and wider-dataset validation.
TOPS customer-path protection note:
the FRIDA-style theta/phi grid alignment was useful for CSSM parity recovery, but applying that same grid change directly to the customer-facing TOPS path slightly worsened the current one-second combined16 benchmark behavior.
Therefore the FRIDA-grid treatment is now restricted to the internal CSSM branch only, while the public TOPS path keeps its prior grid behavior.
On the current local seq45-3p-1111 combined16 15 s to 16 s clip, the restored TOPS path is again producing about 20.66 deg single-clip error on the local benchmark helper instead of staying on the regressed branch.
Public benchmark acceptance policy:
the program target remains within 110% of the best comparable non-learning result, but the older
15.76 / 27.58 / 24.39 deg internal thresholds are no longer treated as SOTA evidence because their
source-selection, activity gating, source-association, and error metric were not locked to one published protocol.
AV16.3 comparisons must explicitly report azimuth error, array layout, activity-label provenance,
source count, window length, and evaluation cadence. LOCATA comparisons must use the official activity periods,
gating / assignment rules, detection and false-track measures, and azimuth / elevation errors.
Until those complete runners have produced reproducible reports, the honest status is
public-dataset non-learning SOTA proximity not yet demonstrated.
Evaluation-geometry policy (2026-07-20): AV16.3 and MIC0528 are separate benchmark families. Every AV16.3 result must use the
corpus's original two 8-channel, 0.1 m-radius UCA layout and its mouth annotations; a MIC0528 layout must never be substituted.
Every synthetic result must use only the MIC0528 data-sheet sensor layout, including the optional floating sensor when applicable.
The public web distribution currently exposes ten complete 16 kHz audio-plus-mouth-annotation sequences; the full corpus is described
as 43 recordings but its remaining material is distributed by the corpus owner separately. Crucially, the second field in an
.3dmouthgt row is is_manual (annotation provenance), not speech activity. Consequently,
mouth files alone cannot validly gate active speech on all ten public sequences. Only records carrying the official
sp_seg source-segment metadata may be used for an AV16.3 speech-activity score; the remaining records
are geometry / tracker regression checks unless an independently documented VAD protocol is used. The earlier all-ten
audio-only active-frame report is invalidated and is not a comparator. A non-learning reference becomes a pass/fail comparator
only after it is rerun or reported with this exact corpus subset and metric.
Adaptive backend and evaluator checkpoint:
finite-distance searches set through steering_distance_m are routed to near-field MUSIC,
regardless of whether the array is planar. TOPS and CSSM currently use far-field steering vectors, so routing a
finite-distance request to either backend violated the API's spherical-wave steering contract. For far-field
requests, adaptive still routes non-planar full-sphere arrays to MUSIC, planar single-source arrays to TOPS, and
planar multiple-source arrays to stabilized CSSM.
The full-clip evaluator must use the same HILL_CLIMBING peak extraction as the SDK: it first keeps the best elevation
for each azimuth and then finds azimuth peaks. Sorting every 3D grid cell directly incorrectly counted several elevation points on one
planar-array ridge as separate sources. The older sparse AV16.3 numbers are now withdrawn because they treated
WAV time as corpus time. The official corpus README states that WAV audio begins at timecode 00:00:10.00, while mouth annotations
retain session time. The evaluator converts both explicit clip starts and emitted SSL timestamps by this 10-second offset before
matching. A later audit also found that it had incorrectly read is_manual as a speech flag. Therefore
all AV16.3 numbers produced with that gate, including the old seq01 14.95 / 2.78 deg SRP-PHAT diagnostic, are
withdrawn. They must be regenerated with official sp_seg labels or excluded from speech-SSL
comparison. No AV16.3 result in this note is currently SOTA evidence.
MIC0528 synthetic benchmark matrix (corrected 2026-07-17):
the required configurations are 4 fixed sensors × 1 / 3 / 5 devices with every sensor on one
plane and hemisphere output, followed by 5 sensors including the floating sensor × 1 / 3 devices
with non-coplanar geometry and full-sphere output. Device origins for the multi-device fixtures form a regular
triangle or pentagon in the PCB plane. The per-device positions are the datasheet values:
fixed sensors at (0, -9.852, 0), (-14.352, 0, 0), (0, 9.852, 0), and (14.352, 0, 0) mm, plus the optional
floating sensor at (-4.180, -1.748, -20.000) mm.
The earlier 20-sensor result that used device origins above and below the plane was the wrong fixture and is
withdrawn.
Current 60-second synthetic results at 2-degree requested accuracy:
single-source mean errors are 1.30 / 0.90 / 0.75 deg for 4×1 / 4×3 / 4×5 hemisphere arrays and
0.80 / 0.79 deg for 5×1 / 5×3 sphere arrays. Their serial realtime factors are
0.015 / 0.165 / 0.537 / 0.036 / 0.428, so all single-source configurations run faster than 1×.
Three-source mean assignment errors are 11.98 / 10.80 / 9.42 / 4.78 / 4.77 deg in the same order,
with serial realtime factors 0.019 / 0.084 / 0.162 / 0.037 / 0.212.
Therefore 4×5, 5×1, and 5×3 pass the provisional 10-degree three-source target; 4×1 and 4×3 do not.
The complete machine-readable report is
D:\sigprocsdk_benchmarks\mic0528_array\report.json.
These are direct-path synthetic engineering checks, not public-dataset SOTA evidence.
Source-motion constraint:
the supported maximum source angular speed is 30 deg/s.
max_source_angular_speed_deg_s defaults to 30, tracker velocity is clamped using the
spherical tangent-plane rate, and the association gate includes
max speed * elapsed time + 2 * requested angle.
At a 500 ms SSL period and 2-degree accuracy this permits 19 degrees of movement between updates.
Synthetic benchmark trajectories are continuous and are rejected before rendering if their measured angular
speed exceeds this limit.
Benchmark-family clarification:
LOCATA is now the primary public acceptance benchmark: it has official per-source activity labels,
microphone/source trajectory ground truth, real reverberant recordings, and static/moving single- and multi-talker tasks.
The pass/fail protocol will lock one LOCATA array/task subset, its official assignment/gating rules, angular metric, window and
cadence, then compare only against non-learning results reported under that same protocol. AV16.3 remains a supplementary
real-speech regression dataset. Recent public SSL / SELD leaders often report on DCASE / STARSS-family tasks rather than AV16.3.
Those tasks are not a direct drop-in replacement for the LOCATA acceptance gate.
DCASE Task 3 style benchmarks are usually SELD tasks, which jointly score sound-event detection and localization,
often on stereo or ambisonic recordings, and frequently with learned models.
By contrast, the current SDK host-side SSL acceptance path is a non-learning multichannel DOA-only path for MIC0528-class arrays.
Therefore DCASE numbers are useful as a realism reference for ordinary office-like reverberation and background clutter,
but they should not be used as a literal pass / fail target for this non-learning C SDK unless the task definition,
channel model, source-count policy, and azimuth / 3D metric are first aligned.
LOCATA non-learning acceptance targets (audited 2026-07-20): the published best azimuth errors for the
Robot Head are 0.7 deg in static single-source Task 1 and 2.0 deg in
static multiple-source Task 2. Both are the DPD-plus-MUSIC family (submission 11), evaluated during the official
voice-active periods. The project's 110% gates are therefore 0.77 deg and 2.20 deg,
respectively, with probability of detection, false-alarm rate, association and OSPA reported alongside error.
These numbers are valid only for the same array, task and official evaluator; they are not claimed for MIC0528 geometry.
The top submission's method description is public, but no commercially reusable reference source release was found.
The current SDK dpd-music experimental backend is not yet that implementation:
the paper requires recording-level top-5% AMD-DPD selection, 512-sample/50%-overlap STFT, local time/frequency smoothing,
single-bin MUSIC and final k-means. It must not be used as evidence against these gates until the separate offline reproduction
path implements those steps and the official evaluator accepts its tracks.
HARK is public for research use but is not an SDK source candidate. The official LOCATA I/O and evaluation frameworks
are retained only as attributed benchmark references under their ODC-By terms, never linked into the distributed SDK.
Public-dataset audit status (2026-07-17):
the first full seq01-1p-0000 combined16 adaptive run exposed an evaluator argument bug that had
converted an omitted analysis window into full-clip mode; this is now fixed. The corrected fast MUSIC routing
produced a large azimuth error and is not competitive. Restoring planar single-source TOPS avoided that routing
mistake, but the current TOPS implementation did not finish even a 20-second interval within 120 seconds on the
development PC. Therefore AV16.3 SOTA proximity is not achieved, and the planar TOPS path still
needs major serial optimization before a complete run is meaningful. LOCATA validation is also pending because
the local dev.zip is truncated (about 0.97 GB versus the official 6.2 GB archive).
Current environmental assumption:
the intended deployment condition is a normal office or lab room with realistic speech reflections and mild clutter,
not a deliberately high-reverberation hall.
In other words, the SDK must remain usable under everyday indoor reverberation,
but it is not required to solve scenes that are effectively dominated by long late reverberation tails.
This assumption should guide both local synthetic tests and public-benchmark interpretation.
Current backend routing note:
after the latest benchmark audit, the generic music evaluation path now uses the SDK's
full-grid MUSIC route rather than the older HARK exact peak-list shortcut.
The reason is practical: under the present AV16.3 checks, the exact peak-list shortcut was producing a stable but clearly wrong
front/back-dominated result on some single-source clips, while the full-grid path at least tracks the same candidate family as the
public SDK runtime.
The older behavior is retained only as an explicit diagnostic route under the hark-exact evaluator name.
Distributed MIC0528 near-field multi-source correction (2026-07-20): when two or more physical sensor clusters of at least four
sensors are separated by more than 80 mm, finite-range multi-source MUSIC now builds a pseudospectrum for each compact cluster and
fuses the normalized device maps. This prevents long inter-device baselines from contributing high-frequency grating lobes directly
to one covariance subspace while retaining independent observations from every device. On the 60 s direct-path MIC0528 matrix,
5 x 3 full-sphere / 3 sources improved from 23.85 deg mean angular error to
4.77 deg; 4 x 5 hemisphere / 3 sources is 9.42 deg and passes the
provisional 10 deg mean target. 4 x 3 hemisphere / 3 sources is still 10.80 deg, so it
remains a known boundary case rather than a passed claim. The complete regenerated report is
D:\sigprocsdk_benchmarks\mic0528_array\report.json.