Applied AI safety · adversarial ML · robust systems

Sihui (Sophie) Dai

Applied Researcher working on AI safety, automated adversarial evaluation, and robust machine learning.

I am an applied researcher working on AI safety, automated adversarial evaluation, and robust machine learning. My current work focuses on developing production safeguards for LLM applications and methods for systematically stress-testing models and defenses. During my Ph.D. at Princeton, I studied robustness to unforeseen, adaptive, and evolving adversaries. I am broadly interested in building AI systems that remain reliable as attacks, models, and deployment environments change.

CV Email GitHub

Research

Reliable models under changing attacks

Automated Adversarial Evaluation

I am interested in scalable red teaming, black-box optimization, adaptive attacks, evaluation infrastructure, and systematic testing of safeguards. I focus on repeatable evaluation methods that help find and measure failures before systems are deployed broadly.

Robustness to Adaptive and Evolving Adversaries

My Ph.D. work studied what happens when a defense is trained for one threat model but faces a different or newly discovered attack at test time. This line of work asks how to measure robustness gaps, how to adapt defended models without forgetting earlier robustness, and how benchmarks can expose brittleness across multiple attack types.

LLM Safety and Safeguards

My applied work involves production safeguards, data quality, evaluation, synthetic augmentation, and robustness testing for LLM applications. I focus on the research-to-production loop: defining failure modes and improving data and evaluations.

Publications

Selected Publications

  6. Larimar: Large Language Models with Episodic Memory Control

    Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarathkrishna Swaminathan, Sihui Dai, Aurelie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiri Navratil, Soham Dan, Pin-Yu Chen. ICML 2024.

    Memory-augmented LLM architecture for knowledge editing, forgetting, and context generalization.

    Paper Code

Complete Publication List

* indicates equal contribution where marked in the source CV.

Preprints

  1. Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai, Supriyo Chakraborty, Shixiong Zhang, Sambit Sahu, William Campbell. Preprint, 2026. arXiv
  2. MemReasoner: A Memory-augmented LLM Architecture for Multi-hop Reasoning. Sihui Dai*, Ching-Yun Ko*, Payel Das*, Georgios Kollias, Subhajit Chaudhury, Aurelie Lozano. Preprint, 2024. OpenReview
  3. Position: Towards Resilience against Adversarial Examples. Sihui Dai, Chong Xiang, Tong Wu, Prateek Mittal. Preprint, 2024.

Conference Papers

  1. ImReasoner: Improving Memory-based Language Models for Reasoning-in-a-Haystack Tasks. Ching-Yun Ko, Payel Das, Sihui Dai, Georgios Kollias, Subhajit Chaudhury, Aurelie C. Lozano, Pin-Yu Chen. ACL 2026. Paper
  2. Adapting to Evolving Adversaries with Regularized Continual Robust Training. Sihui Dai*, Christian Cianfarani*, Arjun Nitin Bhagoji, Vikash Sehwag, Prateek Mittal. ICML 2025. Paper Code
  3. Larimar: Large Language Models with Episodic Memory Control. Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarathkrishna Swaminathan, Sihui Dai, Aurelie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Jiri Navratil, Soham Dan, Pin-Yu Chen. ICML 2024. Paper Code
  4. PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defenses. Chong Xiang, Tong Wu, Sihui Dai, Jonathan Petit, Suman Jana, Prateek Mittal. USENIX Security 2024. Paper
  5. Characterizing the Optimal 0-1 Loss for Multi-class Classification with a Test-time Attacker. Sihui Dai*, Wenxin Ding*, Arjun Nitin Bhagoji, Daniel Cullina, Ben Y. Zhao, Haitao Zheng, Prateek Mittal. NeurIPS 2023 Spotlight. Paper
  6. MultiRobustBench: Benchmarking Robustness Against Multiple Attacks. Sihui Dai, Saeed Mahloujifar, Chong Xiang, Vikash Sehwag, Pin-Yu Chen, Prateek Mittal. ICML 2023. Paper Project
  7. Formulating Robustness Against Unforeseen Attacks. Sihui Dai, Saeed Mahloujifar, Prateek Mittal. NeurIPS 2022. Paper Code
  8. Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, Prateek Mittal. ICLR 2022. Paper
  9. Neural Networks with Recurrent Generative Feedback. Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan Nguyen, Doris Y. Tsao, Anima Anandkumar. NeurIPS 2020. Paper

Workshop Papers

  1. What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations? Sihui Dai, Mann Patel. ICML Hypothesis Testing Workshop, 2026. arXiv
  2. Parameterizing Activation Functions for Adversarial Robustness. Sihui Dai, Saeed Mahloujifar, Prateek Mittal. IEEE Deep Learning Security Workshop, 2022. Metadata
  3. Robustness from Perception. Saeed Mahloujifar, Chong Xiang, Vikash Sehwag, Sihui Dai, Prateek Mittal. ICLR Workshop on Security and Safety in Machine Learning Systems, 2021.
  4. Multi-task Bayesian Optimization via Gaussian Process Upper Confidence Bound. Sihui Dai, Jialin Song, Yisong Yue. ICML Workshop on Real World Experiment Design and Active Learning, 2020.

Contact

CV