A Pretrained Model for Bayesian Filtering in Heterogeneous Dynamical Systems: what it means for business leaders

A single pretrained transformer now tracks hidden states across dynamical systems it was never tuned for, beating hand-built Kalman filters and per-system neural estimators on accuracy and speed, though only on synthetic data so far.

1. What the method is

FilterFM is a transformer that performs online Bayesian filtering, estimating a system's hidden state from a stream of noisy observations, with one pretrained checkpoint that serves dynamical systems differing in governing equations, physical scales, noise levels and even in their number of state and observation variables, from 2 to 100 dimensions. It is trained once on about 15,000 trajectories drawn from standard filtering examples, synthetic chaotic systems and high-dimensional composites, then applied without a transition model, without an observation map and without retraining, including on systems it never saw. Unlike earlier neural filters it decodes quantiles around each estimate, so it reports a median state and an uncertainty band rather than a point alone. The authors present it as the first pretrained filtering model and, to their knowledge, the first learned filtering model to provide uncertainty intervals.

2. Why the method was developed

State estimation underpins robotics, navigation, process monitoring and climate work, yet it has not had the foundation-model shift that forecasting has. Classical filters, from the Kalman family to particle and ensemble methods, need the transition function, the observation map and the noise statistics written down for each system, and they degrade when any of those is wrong. Learning-based filters such as KalmanNet remove some of that modelling burden but are trained from scratch for one system or one parametric family, so every new plant means a new model. Two recent attempts fell short: LLMFilter adapts a frozen language model with a system prompt but is trained per dataset and flattens time and variable channels into tokens, and FM-UKF drops a time-series foundation model into an unscented Kalman filter but still requires a known observation map. The obstacle is structural heterogeneity: without a shared interface across state and observation dimensions, no filter could be reused.

3. Who should care

This touches any team that keeps an estimate of something it cannot measure directly. Our control and robotics engineers, who today hand-tune an extended or ensemble Kalman filter per product, would get a drop-in estimator that needs trajectories rather than equations. Plant and process-monitoring groups with many similar but not identical assets could run one model across the fleet instead of one calibration each. Data-science teams that built bespoke neural estimators should note that a single shared model beat every per-system network here, which argues against continuing to train them one at a time. Product owners of latency-sensitive loops should note the sub-millisecond step time. Procurement and risk should note that training needs ground-truth latent states, which in practice means a simulator or an instrumented test rig, and that the model so far covers only synthetic Gaussian-noise systems.

4. How the method works

Observations are normalised per channel, grouped into fixed-length patches, and encoded at two levels. Patch-level blocks compress each completed patch into a causal history memory, which keeps attention cost manageable in high dimensions. Intra-patch blocks keep every individual time step, because a filter must correct on the current observation and not just a summary. At both levels, causally masked attention models each variable's own history, then a fixed set of prototype slots with positive and negative attention keys pools information across however many variables the system has, and a gated router writes it back, which is what lets one network accept 2 or 100 channels. Intra-patch tokens read the previous patch's memory through gated cross-attention, giving local evidence its context. A learned query per state channel then reads the one-step-shifted memory to form a prior, reads the current memory to compute a gated innovation correction, and a quantile head decodes the posterior into a median and an interval.

5. How it was evaluated

The authors assembled 14 in-domain benchmarks: low-dimensional nonlinear systems such as a pendulum, Selkov, Hopf and an oscillator, and high-dimensional chaotic ones such as Lorenz96, VL20, a 64-variable Burgers system and a 100-dimensional cubic sensor problem, each with 480 training and 20 held-out trajectories. FilterFM was pretrained once on roughly 15,000 trajectories including those. The per-system baselines, MEstimator, RobustStateNet, ProTran, KalmanNet and a recurrent Yau-Yau filter, trained on the same 480 trajectories per system. LLMFilter used a frozen LLaMA-7B backbone with per-system adapters, FM-UKF one transformer per state-dimension group, and the ensemble Kalman filter, three robust variants and a particle filter received each system's true observation model. Accuracy is raw-space RMSE, speed is milliseconds per step on one A800 GPU. Further tests scaled the true observation noise while filters kept their assumed noise, held out unseen chaotic systems for zero-shot filtering, and varied training-system count, trajectory count and model size.

6. How it performed

One checkpoint had the lowest error on 13 of the 14 systems and the lowest average RMSE, 0.232 against 0.306 for FM-UKF, 0.603 for LLMFilter and 0.686 for the ensemble Kalman filter, with the margin widest in high dimensions; the ensemble filter won only the pendulum, narrowly. It was fastest on every system at an average 0.145 milliseconds per step. Under observation-noise misspecification it had the lowest pooled error in 38 of 40 settings. The same checkpoint filtered unseen systems with three to six state variables without fine-tuning, and unseen-system error fell as training systems grew from 164 to 20,980, as trajectories grew to about 302,000, and as parameters grew. The limits are real: all data is synthetic with Gaussian noise, the zero-shot evidence covers small systems, supervision needs true states, and code and checkpoints are promised only upon publication. Our call: watch, and pilot on a simulator-backed product the day the checkpoint ships. (Source: arXiv 2610.40437, 2026)