DurstewitzLab

In a new #ICLR2026 publication we provide a novel algorithm for semi-analytically constructing the stable and unstable manifolds of fixed points and cycles of ReLU-based RNNs:
https://openreview.net/pdf?id=EAwLAwHvhk

Why is this important?

Because it provides insight into why and how trained RNNs produce their behavior, as important for scientific and medical applications and explainable AI more generally. In scientific ML, RNNs are a common tool for *dynamical systems reconstruction* (https://www.nature.com/articles/s41583-023-00740-7), where models are trained to approximate the dynamical system underlying observed time series. Trained RNNs are then to be analyzed further as formal surrogates of the systems trained on.

An RNN’s dynamical repertoire depends on the topological and geometrical properties of its state space. Stable and unstable manifolds of fixed and periodic points dissect a dynamical system’s state space into different *basins of attraction*, their intersections lead to chaotic dynamics with fractal geometry, and – more generally – they provide a type of skeleton for the system’s dynamics, forming structures like separatrix cycles or heteroclinic channels.

Our foundation model for dynamical systems reconstruction (DynaMix), the first model which can zero-shot forecast the long-term evolution of dynamical systems (see previous post below), has been accepted to #NeurIPS2025 with outstanding reviews (6,5,5,5) -- revised version with links to code base in Julia & Python now available at https://arxiv.org/pdf/2505.13192

https://mathstodon.xyz/@DurstewitzLab/114547380245483446

Unlike current AI systems, brains can quickly and flexibly adapt to changing environments.
This is the topic of our new perspective in Nature MI (https://rdcu.be/eSeif), where we relate dynamical and plasticity mechanisms in the brain to in-context and continual learning in AI. #NeuroAI

*Key take-homes:*

- In biological brains and behavior, often quick adaptation to novel rules or task contingencies occurs within just a few trials, often accompanied by sudden transitions in neural population activity (e.g. https://www.nature.com/articles/s41467-025-60943-7).

- Dynamical and plasticity mechanisms in the brain span a huge range of timescales, echoing the complex multiple time-scale dynamics inherent in our physical and biological world. Dynamics in the brain mirrors dynamics in the real world, a property current AI systems fundamentally lack.

- Neuro-dynamical mechanisms are set up to work close to bifurcation (critical) points, allowing fast reconfiguration of (ghost-)attractor landscapes for novel situations through neuromodulators or short-term plasticity.

- Recently identified plasticity mechanisms, like behavioral time-scale plasticity, can quickly ingrain one-shot experiences in synaptic structure, enabling powerful new training algorithms (e.g. https://www.nature.com/articles/s41467-024-55563-6).

- Aligning cognitive task designs in neuroscience and AI, subjecting animals and AI to the same types of test procedures, could facilitate transfer of results and insights.

- Dynamical systems reconstruction (DSR) models trained on physiological and behavioral data may provide means to *directly* translate algorithms as implemented in the brain into AI architectures.

Please see paper for citations and links to original work on all these points.

Can time series (TS) #FoundationModels (FM) like Chronos zero-shot generalize to unseen #DynamicalSystems (DS)? #AI

No, they cannot!

But *DynaMix* can, the first TS/DS foundation model based on principles of DS reconstruction, capturing the long-term evolution of out-of-domain DS: https://arxiv.org/pdf/2505.13192v1

Unlike TS foundation models, DynaMix exhibits #ZeroShotLearning of long-term stats of unseen DS, incl. attractor geometry & power spectrum, w/o *any* re-training, just from a context signal.
It does so with only 0.1% of the parameters of Chronos & 10x faster inference times than the closest competitor.

It often even outperforms TS FMs on forecasting diverse empirical time series, like weather, traffic, or medical data, typically used to train TS FMs.
This is surprising, cos DynaMix’ training corpus consists *solely* of simulated limit cycles & chaotic systems, no empirical data at all!

And no, it’s neither based on Transformers nor Mamba – it’s a new type of mixture-of-experts architecture based on the recently introduced AL-RNN (https://proceedings.neurips.cc/paper_files/paper/2024/file/40cf27290cc2bd98a428b567ba25075c-Paper-Conference.pdf), specifically trained for DS reconstruction.

Remarkably, DynaMix not only generalizes zero-shot to novel DS, but it can even generalize to new initial conditions and regions of state space not covered by the in-context information.

We dive a bit into the reasons why current time series FMs not trained for DS reconstruction fail, and conclude that a DS perspective on time series forecasting & models may help to advance the #TimeSeriesAnalysis field.

Interested in interpretable #AI foundation models for #DynamicalSystems reconstruction?

In a new paper we move into this direction, training common latent DSR models with system-specific features on data from multiple different dynamical regimes and DS: https://arxiv.org/pdf/2410.04814
(Fig. 7)

We show applications like transfer & few-shot learning, but most interestingly perhaps, subject/system-specific features were often linearly related to control parameters of the underlying dynamical system trained on …
(Fig. 4)

This gives rise to an interpretable latent feature space, in which datasets with similar dynamics cluster. Intriguingly, this clustering according to *dynamical systems features* led to much better separation of groups than could be achieved by more traditional time series features.
(Fig. 6)

Fantastic work by the incomparable Manuel Brenner and Elias Weber, together with Georgia Koppe!

How to reconstruct dynamical systems from many different data modalities observed simultaneously?

Here we introduce a novel generative modeling framework for this, based on control-theoretic ideas for efficiently guiding the training process:
https://arxiv.org/abs/2212.07892
(1/4)

Cool, 3 papers accepted at @ICMLConf@mastodon.social:

1) Out-of-Domain Generalization in Dynamical Systems Reconstruction (prev. vers.: https://arxiv.org/abs/2402.18377)

2) Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction
(details to follow)

3) Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics
(prev. vers.: https://arxiv.org/abs/2212.07892)

Fantastic teamwork, as usual, by Niclas Goering, Florian Hess, Manuel Brenner, Zahra Monfared, Jürgen Hemmer, Georgia Koppe

We have openings for several fully-funded positions (PhD & PostDoc) at the intersection of AI/ML, dynamical systems, and neuroscience within a BMFTR-funded Neuro-AI consortium, at Heidelberg University & Central Institute of Mental Health (Germany):
https://www.einzigartigwir.de/en/job-offers/scientific-employee-m-f-d-theoretical-neuroscience-1242/

More info below ...

Relevant publications:
https://www.nature.com/articles/s41583-023-00740-7
https://openreview.net/pdf?id=Vp2OAxMs2s
https://proceedings.mlr.press/v235/brenner24a.html
https://www.nature.com/articles/s41467-025-60943-7

We wrote a little #NeuroAI piece about in-context learning & neural dynamics vs. continual learning & plasticity, both mechanisms to flexibly adapt to changing environments:
https://arxiv.org/abs/2507.02103

We relate this to non-stationary rule learning & switching tasks with rapid performance jumps.

Feedback welcome!

How do animals learn new rules? By systematically testing different behavioral strategies, guided by selective attn. to rule-relevant cues: https://rdcu.be/etlRV

Akin to in-context learning in AI models, strategy selection depends on the animals' "training set" (prior experience). And humans follow similar strategies and express neural representations with similar properties as in rats.

Fantastic work by Florian Bähner, Hazem Toutounji, Tzvetan Popov and many others - I'm just the person advertising!

Symbolic dynamics builds a bridge from #DynamicalSystems to computation/ #AI!

In our #NeurIPS2024 (@NeurIPSConf) paper we present a new network architecture, Almost-Linear RNNs (Fig. 1), that finds most parsimonious piecewise-linear representations of DS from data:
https://arxiv.org/abs/2410.14240

These representations are topologically minimal (Fig. 5,7), profoundly easing interpretation and math. analysis of the underlying data-generating DS.

The AL-RNN furthermore naturally gives rise to a symbolic encoding that provably preserves important topological properties of the underlying dynamical system.
Symbolic dynamics directly links up with computational graphs, finite state machines, formal languages etc. (Fig. 2).

Spearheaded by Manuel Brenner and Christoph Hemmer, jointly with Zahra Monfared.

Magnitude-based pruning is a standard #ML #AI technique to produce sparse models, but in our @ICMLConf@mastodon.social paper https://www.arxiv.org/abs/2406.04934 we find it doesn’t work for #DynamicalSystems reconstruction.
Instead, via geometry-based pruning we find the *network topology* is far more important!

It turns out that even RNN weights small in relative size can have a dramatic impact on system dynamics as measured by attractor agreement. In fact, there is not much difference between small and large magnitude weights in contribution to DS reconstruction quality. (Fig. 1)

Following the lottery ticket hypothesis, we find that large RNNs still contain winning tickets that can be carved out by *geometry-based* pruning, but that these tickets are defined by *graph topology* with initial weight values hardly playing any role. (Fig. 4)

The ‘winning’ graph topology distilled from trained RNNs turns out to exhibit both hub-like and small world features. RNNs initialized with this topology perform significantly better than equally-sized RNNs with random, Watts-Strogatz or Barabási-Albert topology. (Fig. 6)

… and also train much faster. (Fig. 7)

This all makes sense: Natural and engineered DS often bear a specific sparse network topology that is crucial for shaping their dynamics!

Fantastic work lead by Christoph Hemmer with Manuel Brenner and Florian Hess!

‪Happy to discuss our work on parsimonious & math. tractable RNNs for dynamical systems reconstruction next week at
https://cns2025florence.sched.com/event/1z9Mt/neuroxai-explainable-ai-for-understanding-biological-brains

Can we learn from time series data a dynamical systems model that *generalizes* to unobserved dynamical regimes (basins of attraction), like a good scientific theory should?

Out-of-domain generalization in dynamical systems reconstruction: https://arxiv.org/abs/2402.18377

Our new paper, spearheaded by Niclas Göring, provides the first comprehensive mathematical treatment of OOD generalization in dynamical systems reconstruction. Great teamwork with Florian Hess, Manuel Brenner, and the indispensable Zahra Monfared.