CI/CD Pipeline Documentation¶

This document describes the CI/CD infrastructure for grai.build.

📋 Overview¶

grai.build uses GitHub Actions for continuous integration and deployment. Our pipeline includes:

✅ Continuous Integration - Automated testing and validation
🔒 Security Scanning - Dependency and code security checks
📦 Release Automation - Automated PyPI publishing
📚 Documentation Building - Example docs generation

🔄 Workflows¶

1. CI Workflow (`.github/workflows/ci.yml`)¶

Triggers: Push to main/develop, Pull Requests

Jobs:

Test¶

Runs on Python 3.11 and 3.12
Executes full test suite with pytest
Generates coverage report
Uploads coverage to Codecov

Lint¶

Checks code formatting with Black
Lints code with Ruff
Type checks with mypy (non-blocking)

Validate¶

Builds the package
Validates with twine
Tests CLI installation and availability

Integration¶

Spins up Neo4j 5.15 container
Tests full workflow:
grai init
grai validate
grai build
grai run (with Neo4j)
grai docs
Verifies generated documentation files

Status Badge:

![CI](https://github.com/asantora05/grai.build/workflows/CI/badge.svg)

2. Release Workflow (`.github/workflows/release.yml`)¶

Triggers:

Git tags matching v*.*.* (e.g., v0.3.0)
Manual workflow dispatch

Jobs:

Test¶

Runs full test suite before release

Build¶

Builds source and wheel distributions
Validates with twine
Uploads artifacts

Publish to Test PyPI¶

Publishes to test.pypi.org first
Allows verification before production
Uses TEST_PYPI_API_TOKEN secret

Publish to PyPI¶

Publishes to pypi.org
Only runs after Test PyPI succeeds
Uses PYPI_API_TOKEN secret

Create GitHub Release¶

Creates GitHub release with notes
Attaches distribution files
Auto-generates release notes

Creating a Release:

# Update version in pyproject.toml
vim pyproject.toml

# Commit version bump
git add pyproject.toml
git commit -m "chore: bump version to 0.3.0"
git push

# Create and push tag
git tag -a v0.3.0 -m "Release v0.3.0"
git push origin v0.3.0

3. Security Workflow (`.github/workflows/security.yml`)¶

Triggers:

Push to main
Pull Requests
Weekly schedule (Mondays at 00:00 UTC)

Jobs:

Dependency Scan¶

Runs safety check on dependencies
Runs pip-audit for known vulnerabilities
Continues on error (informational)

Code Scan¶

Runs Bandit security scanner
Checks for common security issues
Uploads scan report as artifact

CodeQL Analysis¶

GitHub's semantic code analysis
Scans for security vulnerabilities
Checks code quality issues

4. Documentation Workflow (`.github/workflows/docs.yml`)¶

Triggers:

Push to main (when docs/ or grai/ changes)
Manual workflow dispatch

Jobs:

Build Example Docs¶

Creates example project with grai init
Generates documentation with grai docs
Uploads as artifact

Note: GitHub Pages deployment is commented out - can be enabled when needed.

🔐 Required Secrets¶

To enable full CI/CD, configure these secrets in GitHub Settings → Secrets and variables → Actions:

PyPI Publishing¶

PYPI_API_TOKEN
Go to https://pypi.org/manage/account/token/
Create new API token
Scope: Project (grai-build)
Add to GitHub secrets
TEST_PYPI_API_TOKEN
Go to https://test.pypi.org/manage/account/token/
Create new API token
Scope: Project (grai-build)
Add to GitHub secrets

Codecov (Optional)¶

CODECOV_TOKEN
Go to https://codecov.io/
Add repository
Copy upload token
Add to GitHub secrets

📊 Status Badges¶

Add these to your README:

[![CI](https://github.com/asantora05/grai.build/workflows/CI/badge.svg)](https://github.com/asantora05/grai.build/actions/workflows/ci.yml)
[![Release](https://github.com/asantora05/grai.build/workflows/Release/badge.svg)](https://github.com/asantora05/grai.build/actions/workflows/release.yml)
[![Security](https://github.com/asantora05/grai.build/workflows/Security/badge.svg)](https://github.com/asantora05/grai.build/actions/workflows/security.yml)
[![codecov](https://codecov.io/gh/asantora05/grai.build/branch/main/graph/badge.svg)](https://codecov.io/gh/asantora05/grai.build)
[![PyPI version](https://badge.fury.io/py/grai-build.svg)](https://badge.fury.io/py/grai-build)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)

🤖 Dependabot¶

Dependabot is configured in .github/dependabot.yml to automatically:

Check Python dependencies weekly
Check GitHub Actions weekly
Create PRs for updates
Automatically label and commit with conventional format

Reviewing Dependabot PRs:

Check CI passes
Review changelog/release notes
Test locally if breaking changes
Merge when safe

🚀 Deployment Process¶

Releasing a New Version¶

Prepare Release

# Update version
vim pyproject.toml  # Change version = "0.3.0"

# Update CHANGELOG (if you maintain one)
vim CHANGELOG.md

# Commit
git add pyproject.toml
git commit -m "chore: bump version to 0.3.0"

Tag Release

git tag -a v0.3.0 -m "Release v0.3.0

Features:
- Added documentation generation
- Improved lineage tracking

Fixes:
- Fixed schema-only mode
"

Push Tag

git push origin main
git push origin v0.3.0

Monitor Workflow
Go to Actions tab
Watch Release workflow
Verify Test PyPI upload
Verify PyPI upload
Check GitHub Release created

Verify Release

# Test installation from PyPI
pip install --upgrade grai-build
grai --version

Hotfix Process¶

For urgent fixes:

# Create hotfix branch
git checkout -b hotfix/critical-bug main

# Make fix
vim grai/core/validator/validator.py
git add .
git commit -m "fix: resolve critical validation bug"

# Update version (patch bump)
vim pyproject.toml  # 0.3.0 → 0.3.1
git add pyproject.toml
git commit -m "chore: bump version to 0.3.1"

# Merge to main
git checkout main
git merge hotfix/critical-bug

# Tag and push
git tag -a v0.3.1 -m "Hotfix v0.3.1"
git push origin main --tags

🧪 Testing CI Locally¶

Using Act¶

Test GitHub Actions locally with act:

# Install act
brew install act  # macOS
# or
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

# Run CI workflow
act push

# Run specific job
act -j test

# Run with secrets
act --secret-file .secrets

Manual Testing¶

Before pushing:

# Run all checks locally
pytest --cov=grai --cov-report=term-missing
black --check grai/
ruff check grai/
mypy grai/

# Build package
python -m build
twine check dist/*

# Test installation
pip install dist/*.whl
grai --version

📈 Monitoring¶

Check Workflow Status¶

# Using GitHub CLI
gh run list
gh run view [run-id]
gh run watch

View Logs¶

# Download logs
gh run download [run-id]

# View specific job
gh run view [run-id] --log

🔧 Troubleshooting¶

CI Failures¶

Tests Fail:

Check test logs in Actions tab
Run tests locally: pytest -v
Check for environment differences

Linting Fails:

Run locally: black grai/ && ruff check grai/
Fix issues and commit

Integration Tests Fail:

Check Neo4j container logs
Verify connection strings
Test locally with Docker

Release Failures¶

PyPI Upload Fails:

Check API token is valid
Verify version doesn't already exist
Check package metadata with twine check

Tag Already Exists:

# Delete local tag
git tag -d v0.3.0

# Delete remote tag
git push origin :refs/tags/v0.3.0

# Recreate and push
git tag -a v0.3.0 -m "Release v0.3.0"
git push origin v0.3.0

📝 Best Practices¶

Always run tests locally before pushing
Keep workflows DRY - Use reusable workflows when possible
Use caching - Cache pip dependencies for faster builds
Fail fast - Run quick jobs (lint) before slow jobs (integration tests)
Monitor secrets - Rotate API tokens regularly
Review Dependabot PRs - Don't blindly merge
Tag semantically - Follow semantic versioning

CI/CD Pipeline Documentation¶

📋 Overview¶

🔄 Workflows¶

1. CI Workflow (`.github/workflows/ci.yml`)¶

Test¶

Lint¶

Validate¶

Integration¶

2. Release Workflow (`.github/workflows/release.yml`)¶

Test¶

Build¶

Publish to Test PyPI¶

Publish to PyPI¶

Create GitHub Release¶

3. Security Workflow (`.github/workflows/security.yml`)¶

Dependency Scan¶

Code Scan¶

CodeQL Analysis¶

4. Documentation Workflow (`.github/workflows/docs.yml`)¶

Build Example Docs¶

🔐 Required Secrets¶

PyPI Publishing¶

Codecov (Optional)¶

📊 Status Badges¶

🤖 Dependabot¶

🚀 Deployment Process¶

Releasing a New Version¶

Hotfix Process¶

🧪 Testing CI Locally¶

Using Act¶

Manual Testing¶

📈 Monitoring¶

Check Workflow Status¶

View Logs¶

🔧 Troubleshooting¶

CI Failures¶

Release Failures¶

📝 Best Practices¶

🔮 Future Improvements¶

📚 Resources¶

CI/CD Pipeline Documentation¶

📋 Overview¶

🔄 Workflows¶

1. CI Workflow (.github/workflows/ci.yml)¶

Test¶

Lint¶

Validate¶

Integration¶

2. Release Workflow (.github/workflows/release.yml)¶

Test¶

Build¶

Publish to Test PyPI¶

Publish to PyPI¶

Create GitHub Release¶

3. Security Workflow (.github/workflows/security.yml)¶

Dependency Scan¶

Code Scan¶

CodeQL Analysis¶

4. Documentation Workflow (.github/workflows/docs.yml)¶

Build Example Docs¶

🔐 Required Secrets¶

PyPI Publishing¶

Codecov (Optional)¶

📊 Status Badges¶

🤖 Dependabot¶

🚀 Deployment Process¶

Releasing a New Version¶

Hotfix Process¶

🧪 Testing CI Locally¶

Using Act¶

Manual Testing¶

📈 Monitoring¶

Check Workflow Status¶

View Logs¶

🔧 Troubleshooting¶

CI Failures¶

Release Failures¶

📝 Best Practices¶

🔮 Future Improvements¶

📚 Resources¶

1. CI Workflow (`.github/workflows/ci.yml`)¶

2. Release Workflow (`.github/workflows/release.yml`)¶

3. Security Workflow (`.github/workflows/security.yml`)¶

4. Documentation Workflow (`.github/workflows/docs.yml`)¶