FlowLLM

Official Resources

• Homepage: https://openreview.net/forum?id=0bFXbEMz8e
• Source Repository: https://github.com/facebookresearch/flowmm
• Documentation: https://github.com/facebookresearch/flowmm
• License: MIT License (as part of FlowMM repository)

Overview

FlowLLM is a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. Presented at NeurIPS 2024, FlowLLM addresses the challenge of material discovery by bridging discrete and continuous modeling: LLMs excel at generating discrete variables (atom types) while flow matching excels at refining continuous values (coordinates and lattice parameters).

FlowLLM first fine-tunes an LLM (LLaMA-based CrystalLLM) to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. This approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and the rate for stable, unique, and novel (S.U.N.) crystals by approximately 50%. Additionally, generated crystals are much closer to their relaxed state, reducing post-hoc computational cost.

Scientific domain: Generative models for materials, machine learning, crystal structure generation
Target user community: ML researchers and materials scientists developing generative models

Theoretical Methods

• Riemannian flow matching (RFM) for continuous refinement
• Large language model fine-tuning (CrystalLLM on LLaMA)
• Text-based crystal representation for LLM
• Graph-based representation for RFM refinement
• Fractional coordinates on flat torus for atom positions
• Conditional training with learned base distribution
• Two-step generation: LLM sampling → RFM refinement

Capabilities (CRITICAL)

• De novo generation of novel crystalline materials
• LLM as learned base distribution for flow matching
• 3x increase in stable material generation rate
• ~50% increase in stable, unique, novel (S.U.N.) crystal rate
• Generated crystals closer to relaxed state (reduced DFT cost)
• Handles atom types (discrete) and coordinates (continuous)
• Trained on MP-20 dataset (Materials Project subset)
• Crystal structure prediction capability
• Integration with CrystalLLM and FlowMM frameworks

Inputs & Outputs

Input formats:

• Unconditional or conditional prompts for LLM
• MP-20 dataset for training (Materials Project structures)
• CrystalLLM fine-tuned model

Output data types:

• Generated crystal structures (lattice, atom types, fractional coordinates)
• Stability, uniqueness, and novelty metrics
• Generated structures in CIF or similar formats
• Relaxation energy metrics

Interfaces & Ecosystem

• Programming language: Python (PyTorch-based)
• Base model: LLaMA (fine-tuned as CrystalLLM)
• Flow matching: Riemannian Flow Matching implementation
• Dataset: MP-20 (Materials Project subset)
• Part of FlowMM: Code in facebookresearch/flowmm repository
• Training: Requires CrystalLLM pre-training from facebookresearch/crystal-text-llm

Limitations & Known Constraints

• Requires LLM fine-tuning (computationally expensive)
• Two-stage training: CrystalLLM first, then RFM model
• Limited to MP-20 dataset composition for training
• Generated structures may still require DFT relaxation
• Research code, not production-ready

Performance Characteristics

• LLM inference: fast for initial structure generation
• RFM refinement: iterative but efficient
• Training: requires GPU and significant compute time
• Post-hoc relaxation cost reduced compared to other methods
• Outperforms diffusion-based and LLM-only approaches

Comparison with Other Codes

• vs FlowMM: FlowMM uses fixed base distribution; FlowLLM uses LLM as learned base
• vs Crystal Diffusion (CDVAE): CDVAE uses diffusion; FlowLLM uses LLM + flow matching
• vs CrystalLLM alone: LLM alone generates unrelaxed structures; FlowLLM refines with RFM
• vs USPEX: USPEX uses evolutionary search; FlowLLM uses generative ML

Best Practices

• Train CrystalLLM first using facebookresearch/crystal-text-llm
• Generate sufficient LLM samples for RFM training data
• Use conditional training with base_distribution_from_data=True
• Evaluate using S.U.N. metrics (stable, unique, novel)
• Compare relaxation energy with baseline models

Verification & Sources

Primary sources:

1. NeurIPS 2024 paper: https://openreview.net/forum?id=0bFXbEMz8e
2. GitHub: https://github.com/facebookresearch/flowmm
3. A. Sriram et al., NeurIPS 2024 - FlowLLM
4. arXiv: https://arxiv.org/html/2410.23405

Confidence: VERIFIED - Published at NeurIPS 2024 with code repository available