Schema Migrations¶

Overview¶

grai.build's migration system provides version-controlled schema evolution for your knowledge graph, inspired by Alembic for SQL databases. Track changes to your entities, relations, and properties over time, and safely apply or rollback schema modifications in production.

Key Features¶

📝 Automatic Change Detection - Compares your current schema against previous versions
🔄 Bidirectional Migrations - Every migration includes upgrade and downgrade paths
⚠️ Breaking Change Detection - Identifies potentially dangerous operations
🔒 Version Tracking - Maintains migration history in Neo4j
🧪 Dry-Run Mode - Preview changes before applying them
📊 Execution Statistics - See exactly what changed (nodes, relationships, properties)

Quick Start¶

1. Initialize Migration Tracking¶

Before using migrations, initialize tracking in your Neo4j database:

# Using connection from profiles.yml (recommended)
grai migrate init

# Or specify connection explicitly
grai migrate init --uri bolt://localhost:7687 --user neo4j --password secret

# Use specific profile and target
grai migrate init --profile production --target graph

This command creates the necessary infrastructure in Neo4j to track which migrations have been applied.

2. Create Your First Migration¶

After modifying your schema (adding properties, entities, or relations), create a migration:

# Compare against the previous git commit
grai migrate create "Add email_verified to customer entity" --compare-to HEAD~1

# Compare against a specific branch
grai migrate create "Add email_verified to customer entity" --compare-to main

This will:

Compare your current schema against the last migration
Detect all changes
Generate upgrade and downgrade Cypher scripts
Save the migration to .grai/migrations/

Output:

✓ Migration created successfully!

Revision: abc12345
Files created:
  • .grai/migrations/20250121_143052_add_email_verified.json
  • .grai/migrations/abc12345_upgrade.cypher
  • .grai/migrations/abc12345_downgrade.cypher

Changes detected:
  • 1 property added
  • 0 entities modified
  • 0 relations modified

3. Review the Migration¶

Inspect the generated Cypher to ensure it's correct:

# View upgrade script
cat .grai/migrations/abc12345_upgrade.cypher

# View downgrade script
cat .grai/migrations/abc12345_downgrade.cypher

4. Apply the Migration¶

Apply pending migrations to your database:

# Preview what will happen (dry-run)
grai migrate upgrade --dry-run

# Apply the migration (using profiles.yml)
grai migrate upgrade

# Or specify connection explicitly
grai migrate upgrade --uri bolt://localhost:7687 --user neo4j --password secret

Output:

✓ Applied migration abc12345
  Nodes created: 0
  Relationships created: 0
  Properties set: 1543

5. Rollback if Needed¶

If something goes wrong, rollback the last migration:

# Preview the rollback
grai migrate downgrade --dry-run

# Rollback the migration
grai migrate downgrade

Connection Configuration¶

Migration commands that connect to Neo4j (init, upgrade, downgrade) support multiple ways to specify connection settings:

1. Using Profiles (Recommended)¶

Create or use an existing profile in profiles.yml:

default:
  warehouse:
    type: bigquery
    # ... warehouse config
  neo4j:
    uri: bolt://localhost:7687
    user: neo4j
    password: my-password
    database: neo4j

production:
  neo4j:
    uri: bolt://prod-server:7687
    user: prod_user
    password: ${NEO4J_PASSWORD} # Environment variable

Then simply run:

# Uses "default" profile, "neo4j" target
grai migrate init
grai migrate upgrade

# Use specific profile
grai migrate init --profile production

# Use environment variable
export GRAI_PROFILE=production
grai migrate upgrade

2. CLI Override¶

Override profile settings or provide connection when no profile exists:

grai migrate init --uri bolt://localhost:7687 --user neo4j --password secret

3. Precedence¶

Connection settings are applied in this order (highest priority first):

CLI flags (--uri, --user, --password)
Profile from --profile flag
Profile from $GRAI_PROFILE environment variable
Default profile ("default")

Migration Workflow¶

Typical Development Flow¶

Modify your schema - Edit entity or relation YAML files
Commit your current schema - git add . && git commit -m "Current state"
Make schema changes - Add/modify properties, entities, or relations
Create migration - grai migrate create "description" --compare-to HEAD~1
Review changes - Check the generated Cypher files
Test locally - Apply to local Neo4j and verify
Commit migration files - Add migration files to git
Deploy to production - Run grai migrate upgrade on prod

Team Collaboration¶

When working with a team:

Pull latest migrations from version control
Check migration status: grai migrate list
Apply pending migrations: grai migrate upgrade
Create new migrations for your changes
Push migrations to version control

Commands Reference¶

Command Reference¶

`grai migrate init`¶

Initialize migration tracking in Neo4j database.

# Using connection from profiles.yml (recommended)
grai migrate init

# Or specify connection explicitly
grai migrate init --uri bolt://localhost:7687 --user neo4j --password secret

# Use specific profile/target
grai migrate init --profile production --target graph

Options:

--profile - Profile name from profiles.yml (default: "default" or $GRAI_PROFILE)
--target - Target name within profile (default: "neo4j")
--uri - Neo4j connection URI (overrides profile)
--user - Username for authentication (overrides profile)
--password - Password for authentication (overrides profile)
--database - Database name (default: neo4j)

`grai migrate create <description>`¶

Create a new migration based on schema changes.

# Compare against previous commit
grai migrate create "Add customer loyalty tier" --compare-to HEAD~1

# Compare against main branch
grai migrate create "Add customer loyalty tier" --compare-to main

# Preview without creating files
grai migrate create "Add customer loyalty tier" --compare-to HEAD~1 --dry-run

Options:

--compare-to, -c - Git ref to compare against (required: e.g., 'HEAD~1', 'main', 'abc1234')
--dry-run - Show what would be generated without creating files

`grai migrate diff`¶

Show pending schema changes without creating a migration.

grai migrate diff

Output:

Schema Changes Detected:

Entities:
  + customer.loyalty_tier (string)
  - customer.legacy_id (removed)

Relations:
  ~ PURCHASED: Added order_total property

⚠️ Breaking Changes:
  • Dropping property: customer.legacy_id

`grai migrate list`¶

List all migrations and their status.

grai migrate list

Output:

Migration History

Revision  Description                        Status    Applied At
abc12345  Add email_verified                 Applied   2025-01-21 14:30:52
def67890  Add customer loyalty tier          Applied   2025-01-21 15:45:23
ghi11111  Remove legacy_id from customer     Pending   -

`grai migrate upgrade`¶

Apply pending migrations to the database.

# Apply all pending migrations (using profiles.yml)
grai migrate upgrade

# Apply specific migration
grai migrate upgrade --revision abc12345

# Dry-run mode
grai migrate upgrade --dry-run

# Use explicit connection
grai migrate upgrade --uri bolt://localhost:7687 --user neo4j --password secret

Options:

--revision - Apply up to specific revision
--dry-run - Preview without executing
--profile - Profile name from profiles.yml (default: "default" or $GRAI_PROFILE)
--target - Target name within profile (default: "neo4j")
--uri - Neo4j connection URI (overrides profile)
--user - Neo4j username (overrides profile)
--password - Neo4j password (overrides profile)

`grai migrate downgrade`¶

Rollback the last applied migration.

# Rollback last migration (using profiles.yml)
grai migrate downgrade

# Rollback to specific revision
grai migrate downgrade --revision abc12345

# Dry-run mode
grai migrate downgrade --dry-run

# Use explicit connection
grai migrate downgrade --uri bolt://localhost:7687 --user neo4j --password secret

Options:

--revision - Rollback to specific revision
--dry-run - Preview without executing
--profile - Profile name from profiles.yml (default: "default" or $GRAI_PROFILE)
--target - Target name within profile (default: "neo4j")
--uri - Neo4j connection URI (overrides profile)
--user - Neo4j username (overrides profile)
--password - Neo4j password (overrides profile)

Migration File Structure¶

Directory Layout¶

.grai/
└── migrations/
    ├── 20250121_143052_add_email_verified.json
    ├── abc12345_upgrade.cypher
    ├── abc12345_downgrade.cypher
    ├── 20250121_154523_add_loyalty_tier.json
    ├── def67890_upgrade.cypher
    └── def67890_downgrade.cypher

Migration JSON Format¶

{
  "metadata": {
    "revision": "abc12345",
    "parent_revision": null,
    "description": "Add email_verified to customer",
    "created_at": "2025-01-21T14:30:52Z",
    "schema_version": "1.1.0",
    "project_name": "my_project",
    "change_summary": {
      "properties_added": 1,
      "properties_modified": 0,
      "properties_dropped": 0,
      "entities_added": 0,
      "entities_modified": 1,
      "entities_dropped": 0,
      "relations_added": 0,
      "relations_modified": 0,
      "relations_dropped": 0
    },
    "breaking": false
  },
  "changes": {
    "entity_changes": [
      {
        "change_type": "modify_entity",
        "entity_name": "customer",
        "property_changes": [
          {
            "change_type": "add_property",
            "property_name": "email_verified",
            "new_type": "boolean"
          }
        ]
      }
    ],
    "relation_changes": [],
    "breaking_changes": [],
    "warnings": []
  },
  "upgrade_cypher": "...",
  "downgrade_cypher": "..."
}

Change Types¶

Entity Changes¶

ADD_ENTITY - New entity type added
DROP_ENTITY - Entity type removed ⚠️ Breaking
MODIFY_ENTITY - Entity definition changed

Property Changes¶

ADD_PROPERTY - New property added
DROP_PROPERTY - Property removed ⚠️ Breaking
MODIFY_PROPERTY - Property type changed ⚠️ Breaking

Relation Changes¶

ADD_RELATION - New relation type added
DROP_RELATION - Relation type removed ⚠️ Breaking
MODIFY_RELATION - Relation definition changed

Key Changes¶

ADD_KEY - New key property added
DROP_KEY - Key property removed ⚠️ Breaking

Breaking Changes¶

Certain operations are flagged as breaking changes because they can cause data loss or break existing queries:

❌ Dropping an entity
❌ Dropping a property
❌ Changing a property type
❌ Dropping a key

When creating a migration with breaking changes, you'll see:

⚠️ WARNING: This migration contains BREAKING changes!

Breaking Changes:
  • DROP_PROPERTY: customer.legacy_id (string)
  • MODIFY_PROPERTY: customer.age (string → integer)

These changes may cause data loss or break existing queries.
Proceed with caution in production environments.

Continue? [y/N]:

Advanced Usage¶

Comparing Against Specific Versions¶

Compare current schema against a specific git commit:

# Compare against main branch
grai migrate diff --base main

# Compare against specific commit
grai migrate diff --base abc123

# Compare against previous migration
grai migrate diff --base HEAD~1

Custom Migration Scripts¶

You can manually edit generated Cypher scripts before applying:

Create migration: grai migrate "description"
Edit the generated .cypher files
Apply modified migration: grai migrate upgrade

Example: Add custom data transformation logic:

// abc12345_upgrade.cypher (manually edited)

// Generated: Add property
MATCH (n:customer)
SET n.email_verified = null;

// Custom: Set default values based on existing data
MATCH (n:customer)
WHERE n.email IS NOT NULL AND n.email <> ''
SET n.email_verified = true;

MATCH (n:customer)
WHERE n.email IS NULL OR n.email = ''
SET n.email_verified = false;

Migration Dependencies¶

Migrations are applied in order based on their creation timestamp. Each migration tracks its parent:

{
  "metadata": {
    "revision": "def67890",
    "parent_revision": "abc12345",  // Previous migration
    ...
  }
}

This creates a linear chain of migrations, ensuring they're applied in the correct order.

Squashing Migrations¶

For production deployment, you may want to squash multiple development migrations into one:

# Combine last 5 migrations into one
grai migrate squash --count 5 --message "Combined schema updates"

This creates a new migration that represents the cumulative effect of all specified migrations.

Production Best Practices¶

1. Test Migrations Locally First¶

Always test migrations on a local or staging database before production:

# Test upgrade
grai migrate upgrade --dry-run
grai migrate upgrade

# Verify data
# Run queries to check data integrity

# Test downgrade
grai migrate downgrade --dry-run
grai migrate downgrade

2. Backup Before Migrating¶

Create a Neo4j backup before applying migrations:

# Neo4j Enterprise backup
neo4j-admin backup --database=neo4j --backup-dir=/backups

# Or export data
CALL apoc.export.cypher.all("backup.cypher", {})

3. Use Dry-Run in CI/CD¶

Include migration validation in your CI pipeline:

# .github/workflows/test.yml
- name: Validate migrations
  run: |
    grai migrate upgrade --dry-run
    grai migrate downgrade --dry-run

4. Monitor Migration Performance¶

For large databases, monitor migration execution time:

time grai migrate upgrade

Consider running migrations during low-traffic periods for production.

5. Keep Migration History¶

Never delete migration files from version control. They form a complete history of your schema evolution.

6. Use Transactions¶

All migrations are executed within Neo4j transactions. If any statement fails, the entire migration is rolled back.

Troubleshooting¶

Migration Already Applied¶

Error:

Migration abc12345 is already applied

Solution: This is normal - grai.build prevents re-applying migrations. To re-run a migration:

Rollback: grai migrate downgrade
Reapply: grai migrate upgrade

No Changes Detected¶

Error:

No schema changes detected. Migration not created.

Solution:

Ensure you've modified your entity/relation YAML files
Check that grai.yml version has been updated
Verify files are saved

Breaking Changes in Production¶

Warning:

⚠️ This migration contains BREAKING changes!

Best Practice:

Review the specific breaking changes
Plan for data migration/transformation
Test thoroughly in staging
Consider creating a custom migration script
Schedule during maintenance window
Have rollback plan ready

Migration Conflict¶

Error:

Migration conflict detected: Multiple migrations with same parent

Solution: This happens when team members create migrations simultaneously:

Pull latest migrations from version control
Resolve conflicts manually
Recreate your migration
Push to version control

Database Connection Failed¶

Error:

Cannot connect to Neo4j at bolt://localhost:7687

Solution:

Verify Neo4j is running: neo4j status
Check connection credentials
Test connection: cypher-shell -a bolt://localhost:7687

Examples¶

Example 1: Adding a Property¶

Schema Change:

# entities/customer.yml
entity: customer
keys: [customer_id]
properties:
  - name: customer_id
    type: string
  - name: email_verified # NEW
    type: boolean

Create Migration:

grai migrate "Add email verification flag"

Generated Upgrade:

MATCH (n:customer)
SET n.email_verified = null;

Generated Downgrade:

MATCH (n:customer)
REMOVE n.email_verified;

Example 2: Adding an Entity¶

Schema Change:

# entities/product.yml
entity: product
keys: [product_id]
properties:
  - name: product_id
    type: string
  - name: name
    type: string
  - name: price
    type: float

Create Migration:

grai migrate "Add product entity"

Generated Upgrade:

CREATE CONSTRAINT product_product_id_unique IF NOT EXISTS
FOR (n:product) REQUIRE n.product_id IS UNIQUE;

CREATE INDEX product_product_id_index IF NOT EXISTS
FOR (n:product) ON (n.product_id);

Generated Downgrade:

DROP CONSTRAINT product_product_id_unique IF EXISTS;
DROP INDEX product_product_id_index IF EXISTS;

Example 3: Adding a Relation¶

Schema Change:

# relations/purchased.yml
relation: PURCHASED
from: customer
to: product
properties:
  - name: order_id
    type: string
  - name: purchase_date
    type: date

Create Migration:

grai migrate "Add purchase relation"

Generated Upgrade:

// Relation structure already defined in entity loading
// No structural changes needed for adding relations

Example 4: Modifying Property Type ⚠️¶

Schema Change:

# entities/customer.yml
entity: customer
properties:
  - name: age
    type: integer # Changed from string

Create Migration:

grai migrate "Change age from string to integer"

Generated Migration (requires manual editing):

-- BREAKING CHANGE: Manual data transformation required
-- This migration needs custom logic to convert existing string values to integers

MATCH (n:customer)
WHERE n.age IS NOT NULL
SET n.age = toInteger(n.age);

Integration with grai.build Workflow¶

Migrations integrate seamlessly with your existing workflow:

# 1. Initialize project
grai init my_project

# 2. Define entities and relations
# Edit YAML files...

# 3. Build and validate
grai build --validate

# 4. Load initial data
grai load --execute

# 5. Make schema changes
# Edit YAML files...

# 6. Create migration
grai migrate "Add new fields"

# 7. Apply migration
grai migrate upgrade

# 8. Continue developing
# Repeat steps 5-7 as needed

FAQs¶

When should I create a migration?¶

Create a migration whenever you modify your schema:

Adding/removing entities or relations
Adding/removing properties
Changing property types
Modifying keys

Can I edit migrations after creating them?¶

Yes! Migration files are just JSON and Cypher. You can edit them before applying, but:

Don't edit migrations that have already been applied
Don't edit migrations that have been shared with your team
Keep the revision ID unchanged

What if I need to apply migrations in a specific order?¶

Migrations are automatically applied in chronological order based on their creation timestamp. The parent_revision field ensures the correct sequence.

Can I rollback multiple migrations at once?¶

Currently, grai migrate downgrade rolls back one migration at a time. To rollback multiple:

grai migrate downgrade  # Rollback most recent
grai migrate downgrade  # Rollback second most recent
# etc.

Do migrations work with APOC?¶

Yes! Migrations are compatible with APOC and will use APOC functions if available for better performance.

Can I use migrations without Neo4j?¶

Migration generation works without Neo4j connection. You can create, view, and version migrations offline. However, applying migrations requires an active Neo4j connection.

Getting Started - Initial project setup
CLI Reference - Complete command documentation
Data Loading - Loading data from warehouses
Philosophy - Design principles

Support¶

For issues or questions:

Schema Migrations¶

Overview¶

Key Features¶

Quick Start¶

1. Initialize Migration Tracking¶

2. Create Your First Migration¶

3. Review the Migration¶

4. Apply the Migration¶

5. Rollback if Needed¶

Connection Configuration¶

1. Using Profiles (Recommended)¶

2. CLI Override¶

3. Precedence¶

Migration Workflow¶

Typical Development Flow¶

Team Collaboration¶

Commands Reference¶

Command Reference¶

grai migrate init¶

grai migrate create <description>¶

grai migrate diff¶

grai migrate list¶

grai migrate upgrade¶

grai migrate downgrade¶

Migration File Structure¶

Directory Layout¶

Migration JSON Format¶

Change Types¶

Entity Changes¶

Property Changes¶

Relation Changes¶

Key Changes¶

Breaking Changes¶

Advanced Usage¶

Comparing Against Specific Versions¶

Custom Migration Scripts¶

Migration Dependencies¶

Squashing Migrations¶

Production Best Practices¶

1. Test Migrations Locally First¶

2. Backup Before Migrating¶

3. Use Dry-Run in CI/CD¶

4. Monitor Migration Performance¶

5. Keep Migration History¶

6. Use Transactions¶

Troubleshooting¶

Migration Already Applied¶

No Changes Detected¶

Breaking Changes in Production¶

Migration Conflict¶

Database Connection Failed¶

Examples¶

Example 1: Adding a Property¶

Example 2: Adding an Entity¶

Example 3: Adding a Relation¶

Example 4: Modifying Property Type ⚠️¶

Integration with grai.build Workflow¶

FAQs¶

When should I create a migration?¶

Can I edit migrations after creating them?¶

What if I need to apply migrations in a specific order?¶

Can I rollback multiple migrations at once?¶

Do migrations work with APOC?¶

Can I use migrations without Neo4j?¶

Related Documentation¶

Support¶

`grai migrate init`¶

`grai migrate create <description>`¶

`grai migrate diff`¶

`grai migrate list`¶

`grai migrate upgrade`¶

`grai migrate downgrade`¶