CSI-TextBench: A Dataset and Benchmark for Language-Grounded Ambient Sensing Perception

People spend much of their time indoors; WiFi is already deployed, always on, and sensitive to motion through Channel State Information (CSI). Prior WiFi sensing work spans activity recognition, localization, fall detection, and user identification, but systems are typically tied to fixed label spaces, which limits flexible querying, reuse, and composition. Language alignment has made vision and audio queryable via shared embedding spaces—for CSI, the obstacle is data: existing datasets provide categorical labels, not paired natural language.

Source datasets. CSI-Bench contributes five tasks (HAR, fall detection, motion source, proximity, user ID) from large-scale in-the-wild recordings across diverse commercial devices. XRF55 adds 55 fine-grained activities in controlled settings. SignFi contributes 276 sign-gesture classes at the finest semantic granularity. SignFi is released as preprocessing scripts and CSI–text registry files; raw CSI must be obtained from the original provider under its license.

Unified preprocessing. Sources differ in subcarriers, antennas, and sampling rates. Following the CSI-Bench pipeline, all inputs are standardized to an amplitude tensor X ∈ ℝ^C×K×T: amplitude only (discarding unstable phase), 5-second windows, per-sample normalization, and subcarrier alignment to fixed K. This reduces hardware-specific variation while preserving motion-induced dynamics.

Text construction. Because CSI has no natural textual counterpart, descriptions are generated from labels via task-aware templates with paraphrastic variation. A hierarchical scheme uses level-1 captions for the primary phenomenon (activity or gesture) and level-2 composite captions that combine attributes—for example, activity, proximity, and user identity—enabling multi-attribute scene alignment.

Signal-grounded semantics. Text is restricted to properties plausibly encoded in CSI—activity, gesture, proximity, motion source, and user-related behavior—while omitting deployment-specific attributes (e.g., exact device type or room identity) to limit spurious correlations.

Many-to-many structure. Each CSI sample is paired with roughly 4–9 descriptions; each text matches many samples. This dense many-to-many structure differs from typical vision–language data and stresses alignment objectives (e.g., in-batch “negatives” may share semantics with positives).

CSI–text alignment differs from image–text settings in three ways: dense many-to-many correspondence, no large-scale naturally paired text, and a sensing modality with distinct signal structure. All methods share a dual-encoder setup to isolate the alignment objective: a learned CSI encoder with projection maps CSI to z_csi ∈ ℝ²⁵⁶; a frozen pretrained text encoder (default: CLIP ViT-B/32) with projection yields z_text; embeddings are ℓ2-normalized before the loss.

The CSI encoder is a 1D-adapted ResNet-18 on input reshaped to [B, C·K, T]. Training uses AdamW, cosine schedule with warmup, batch size 128, and early stopping; 35 single-task models cover five objectives × seven tasks, plus multitask and supervised baselines as in the paper.

InfoNCE uses batch softmax contrastives (with false-negative pressure under shared semantics). SigLIP uses pairwise sigmoid losses suited to multiple positives. LanguageBind aligns CSI to a frozen text space with an asymmetric contrastive objective. Barlow Twins enforces cross-correlation consistency without explicit negatives. Masked prediction reconstructs text embeddings from partially masked CSI, probing structure beyond discrimination alone.

Layer 1 — Does alignment match supervision?

Zero-shot classification assigns the class whose mean text-prototype embedding best matches the CSI embedding. CLIP InfoNCE recovers within 0.3 points of the supervised baseline on HAR, motion source, and user ID, and within 2.4 points on 276-class SignFi—without any class labels. Cross-modal retrieval shows InfoNCE yields a globally well-structured space (e.g., strong R@5 / R@10 on XRF55 and SignFi), not only a sharp top-1.

Layer 2 — Generalization to unseen classes and phrasing

On SignFi, train on 200 classes and hold out 76: zero-shot on held-out classes is limited (language alone cannot disambiguate unseen fine-grained signs), but 1-shot linear probing reaches 54.0% and 10-shot reaches 74.6%—a deployment path closed-set classifiers do not offer. Open-vocabulary HAR tests novel prompts: layperson phrasings match training-template accuracy (~96%), while specialist wording drops unless anchored to phrasing closer to CLIP’s pretraining—showing alignment to text, with coverage bounded by the text encoder prior.

Layer 3 — One model for compositional scenes

A unified multitask CLIP InfoNCE model (seven tasks, proportional sampling, augmentations) retains ≥95% of single-task accuracy on six of seven tasks at inference via task-specific text prototypes only—no task heads. On CSI-Bench’s co-labeled subset, Level 2 composite retrieval over 140 joint configurations (5 activities × 4 proximities × 7 users) reaches 86.9% top-1, beating the ~70.4% joint accuracy from independent per-attribute QA—composite embeddings capture cross-attribute dependencies that factorized prediction misses.

Prior RF- and sensor-language efforts often focus on a single task or modality, or do not release reusable sample–text pairs at scale. CSI-TextBench complements this landscape as an open, multi-task WiFi CSI–text benchmark spanning controlled to in-the-wild settings, diverse hardware, and class counts from 2 to 276—with explicit evaluation of alignment objectives under many-to-many pairing.

CSI avoids identifiable video or audio, which can support privacy-sensitive assistive use cases (e.g., fall detection, smart-home automation) on existing infrastructure. Responsible deployment still requires consent, on-device or access-controlled processing, and clear policies. The benchmark builds on publicly available datasets collected under their respective approvals and licenses.