maggma

Official Resources

• Homepage: https://materialsproject.github.io/maggma/
• Documentation: https://materialsproject.github.io/maggma/
• Source Repository: https://github.com/materialsproject/maggma
• License: Modified BSD License

Overview

Maggma is a framework for building scientific data pipelines. It provides an abstraction layer (Stores) for various databases (MongoDB, S3, GridFS, Filesystem) and a "Builder" pattern for processing data from one store to another. It is the backbone of the Materials Project's new data infrastructure.

Scientific domain: Data engineering, materials informatics
Target user community: Database maintainers, data scientists

Capabilities (CRITICAL)

• Stores: Unified API for MongoDB, S3, JSON files, etc.
• Builders: Python classes that process items from a source store and write to a target store.
• Incremental Processing: Keeps track of processed items to allow incremental updates.
• Aggregation: Tools for grouping and reducing data.

Sources: Maggma documentation

Inputs & Outputs

• Input formats: Database records
• Output data types: Processed database records

Interfaces & Ecosystem

• MongoDB: Primary backend.
• Materials Project: Core infrastructure.
• Jobflow: Integration for workflow-driven building.

Workflow and Usage

1. Define Source Store (e.g., Raw VASP files on S3).
2. Define Target Store (e.g., MongoDB for properties).
3. Create Builder: class MyBuilder(Builder): ...
4. Run Builder.

Performance Characteristics

• Designed for processing millions of materials records.
• Parallel processing supported.

Application Areas

• Building materials databases (MP, various research groups)
• ETL (Extract, Transform, Load) pipelines

Community and Support

• Developed by Materials Project (LBNL)

Verification & Sources

Primary sources:

1. GitHub: https://github.com/materialsproject/maggma

Confidence: VERIFIED

Verification status: ✅ VERIFIED

• Website: ACTIVE
• Documentation: AVAILABLE
• Source: OPEN (GitHub)
• Development: ACTIVE
• Applications: Data pipelines, ETL