Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control

Contact: simon.hirlaender(at)sbg.ac.at

Official repository for the FERMI Free-Electron Laser (FEL) paper, utilizing model-based and model-free reinforcement learning methods to solve complex particle accelerator operation problems. This work demonstrates the practical application of deep RL for intensity optimization, comparing the sample-efficient AE-DYNA (model-based) with the high-performing NAF2 (model-free) algorithms.

---

Algorithms

Algorithm	Type	Noise Robust	Sample Efficient
NAF	Model-free	✗	✓
NAF2	Model-free	✓	✓
ME-TRPO	Model-based	✗	✓
AE-DYNA	Model-based	✓	✓

---

Quick Start

Installation (TensorFlow 2)

conda create -n fermi_rl python=3.8
conda activate fermi_rl
pip install -r requirements.txt

Running Experiments

# NAF2 (Model-Free)
python src/run_naf2.py

# AE-DYNA (Model-Based)
python src/AE_Dyna_Tensorflow_2.py

Note

The legacy script src/AEDYNA.py requires TensorFlow 1.15 and stable-baselines (v2) in a separate environment.

---

Results

FERMI FEL Performance

NAF2 Training	NAF2 Convergence

AE-DYNA Training	AE-DYNA Verification

Inverted Pendulum Benchmarks

Noise Robustness	Sample Efficiency (NAF vs AE-DYNA)

---

Project Structure

.
├── src/                  # Python source code
│   ├── run_naf2.py       # NAF2 agent (TF2)
│   ├── AE_Dyna_Tensorflow_2.py # AE-DYNA agent (TF2)
│   └── AEDYNA.py         # AE-DYNA agent (TF1.15 legacy)
├── paper/                # LaTeX source and figures
│   ├── main.tex
│   └── Figures/
├── data/                 # Experimental data
└── requirements.txt

---

Citation

If you use this work, please cite:

@software{hirlaender_fermi_rl,
  author       = {Hirlaender, Simon and Bruchon, Niky},
  title        = {FERMI RL Paper Code},
  year         = 2020,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.4348989},
  url          = {https://doi.org/10.5281/zenodo.4348989}
}