Implement vendor system in OSSM Configurator, enhancing reproducibility of external asset files. Update README with detailed vendor system documentation, including project structure, manifest schema, and integration with component JSON files. Modify package.json and package-lock.json for dependency updates, and refactor export utilities to use ExcelJS instead of XLSX for Excel file generation.

2026-01-07 02:12:03 +00:00
parent 87b9ff88ef
commit 4bc0fd203f
1 changed files with 355 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -214,3 +214,358 @@ When adding new components or options:
 ## License

 This project is part of the Open Source Sex Machine (OSSM) project. Please refer to the OSSM project license for usage terms.
+
+---
+
+## Vendor System
+
+The OSSM Configurator includes a robust vendoring and monitoring system for external asset files (STL files, etc.) referenced in component JSON files. This system ensures reproducible builds by pinning external files to specific commit SHAs and automatically detecting when upstream changes occur.
+
+### Overview
+
+The vendor system:
+- **Vendors** external files from GitHub repositories into the `vendor/` directory
+- **Pins** files to specific commit SHAs for reproducible builds
+- **Monitors** upstream repositories for changes
+- **Automatically updates** vendored files when upstream changes are detected
+- **Preserves backward compatibility** by keeping original `url` fields in component JSON files
+
+### Project Structure (Vendor System)
+
+```
+OSSM-Configurator/
+├── manifest/
+│   └── vendor_manifest.json    # Canonical list of vendored files
+├── vendor/                     # Vendored file copies
+│   └── owner-repo/            # Organized by repository
+│       └── path/to/file.stl
+├── scripts/
+│   ├── generate_manifest_from_site.py  # Generate manifest from component JSONs
+│   ├── vendor_update.py               # Download and pin files
+│   └── check_updates.py               # Monitor upstream changes
+├── api/
+│   └── github_webhook/
+│       └── index.py            # Webhook handler for push events
+├── tests/
+│   ├── test_vendor_update.py   # Tests for vendor_update.py
+│   └── test_check_updates.py   # Tests for check_updates.py
+└── .github/
+    └── workflows/
+        └── check-vendor.yml    # Automated monitoring workflow
+```
+
+### Manifest Schema
+
+The `manifest/vendor_manifest.json` file contains metadata for each vendored file:
+
+```json
+{
+  "id": "unique-id-or-slug",
+  "source_repo": "owner/repo",
+  "source_path": "path/in/repo/file.stl",
+  "source_ref": "main",
+  "pinned_sha": "commit-sha-currently-pinned",
+  "pinned_raw_url": "https://raw.githubusercontent.com/owner/repo/<sha>/path/to/file.stl",
+  "local_path": "vendor/owner-repo/path/to/file.stl",
+  "checksum_sha256": "sha256-hex-digest",
+  "last_checked": "2024-01-05T12:00:00Z",
+  "upstream_latest_sha": "latest-commit-sha-observed",
+  "status": "up-to-date | out-of-date | unknown",
+  "license": "SPDX-identifier-or-URL",
+  "orig_site_json": "website/src/data/components/actuator.json",
+  "orig_item_id": "ossm-actuator-body-bottom"
+}
+```
+
+### Integration with Component JSON Files
+
+The vendor system adds a `vendor` object to each `printedParts` entry in component JSON files, while preserving the original `url` field for backward compatibility:
+
+**Before:**
+```json
+{
+  "id": "ossm-actuator-body-bottom",
+  "name": "Actuator Bottom",
+  "filePath": "OSSM - Actuator Body Bottom.stl",
+  "url": "https://github.com/KinkyMakers/OSSM-hardware/.../file.stl?raw=true"
+}
+```
+
+**After (with vendor metadata):**
+```json
+{
+  "id": "ossm-actuator-body-bottom",
+  "name": "Actuator Bottom",
+  "filePath": "OSSM - Actuator Body Bottom.stl",
+  "url": "https://github.com/KinkyMakers/OSSM-hardware/.../file.stl?raw=true",
+  "vendor": {
+    "manifest_id": "ossm-actuator-body-bottom",
+    "local_path": "vendor/KinkyMakers-OSSM-hardware/Printed Parts/Actuator/file.stl",
+    "pinned_sha": "abc123...",
+    "pinned_raw_url": "https://raw.githubusercontent.com/KinkyMakers/OSSM-hardware/abc123/...",
+    "checksum_sha256": "deadbeef...",
+    "last_checked": "2024-01-05T12:00:00Z",
+    "status": "up-to-date"
+  }
+}
+```
+
+The website continues to use `printedParts.url` by default. Site code can optionally prefer `vendor.*` fields when available for offline hosting or improved reliability.
+
+### Prerequisites
+
+- Python 3.11 or higher
+- GitHub API token (for authenticated requests and higher rate limits)
+
+### Setup
+
+1. **Install Python dependencies:**
+   ```bash
+   cd scripts
+   pip install -r requirements.txt
+   ```
+
+2. **Set up GitHub API token:**
+   ```bash
+   export GITHUB_API_TOKEN=your_token_here
+   # Or use GITHUB_TOKEN as fallback
+   ```
+
+   To create a GitHub token:
+   - Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
+   - Generate a new token with `public_repo` scope (or `repo` for private repos)
+   - Required permissions: Read access to repositories
+
+### Local Usage
+
+#### 1. Generate Manifest from Site Data
+
+Scan component JSON files and create/update the vendor manifest:
+
+```bash
+python scripts/generate_manifest_from_site.py \
+  --site-dir website/src/data/components \
+  --manifest manifest/vendor_manifest.json
+```
+
+This script:
+- Scans all JSON files in `website/src/data/components/`
+- Extracts `printedParts` entries with GitHub URLs
+- Creates manifest entries for each GitHub-hosted file
+- Supports nested structures (e.g., `systems.printedParts`, `bodyParts`, `knobs`)
+
+#### 2. Download and Pin Files
+
+Download files from GitHub and pin them to commit SHAs:
+
+```bash
+# Update all entries
+python scripts/vendor_update.py \
+  --manifest manifest/vendor_manifest.json
+
+# Update a specific entry
+python scripts/vendor_update.py \
+  --manifest manifest/vendor_manifest.json \
+  --entry ossm-actuator-body-bottom
+
+# Dry run (see what would be done)
+python scripts/vendor_update.py \
+  --manifest manifest/vendor_manifest.json \
+  --dry-run
+
+# Update and sync vendor metadata to site JSON files
+python scripts/vendor_update.py \
+  --manifest manifest/vendor_manifest.json \
+  --sync-site
+```
+
+This script:
+- Resolves commit SHAs for the specified ref (branch/tag)
+- Downloads files from pinned URLs
+- Computes SHA256 checksums
+- Updates manifest with metadata
+- Optionally syncs vendor metadata back to component JSON files
+
+#### 3. Check for Upstream Updates
+
+Monitor upstream repositories for changes:
+
+```bash
+python scripts/check_updates.py \
+  --manifest manifest/vendor_manifest.json \
+  --output report.json
+```
+
+This script:
+- Queries GitHub API for latest commit SHAs
+- Compares with pinned SHAs
+- Generates a report of up-to-date and out-of-date entries
+- Exits with non-zero code if any entries are out-of-date (useful for CI)
+
+### GitHub Action Workflow
+
+The `.github/workflows/check-vendor.yml` workflow automatically:
+
+1. **Runs daily** (2 AM UTC) or can be triggered manually
+2. **Generates manifest** from site data
+3. **Checks for updates** in upstream repositories
+4. **Creates a branch and PR** if updates are available
+5. **Updates vendored files** and syncs metadata to component JSON files
+
+#### Manual Trigger
+
+To manually trigger the workflow:
+1. Go to Actions tab in GitHub
+2. Select "Check Vendor Updates" workflow
+3. Click "Run workflow"
+
+#### Required Secrets
+
+Configure these secrets in GitHub repository settings:
+
+- `GITHUB_API_TOKEN`: GitHub personal access token with `public_repo` scope
+
+The workflow uses `GITHUB_TOKEN` for creating PRs (automatically provided by GitHub Actions).
+
+### Webhook Configuration
+
+The `api/github_webhook/index.py` provides a Flask-based webhook handler that can receive push events from upstream repositories.
+
+#### Setup
+
+1. **Deploy the webhook** (e.g., to a serverless platform, VPS, or local server)
+
+2. **Configure webhook secret:**
+   ```bash
+   export WEBHOOK_SECRET=your_webhook_secret_here
+   ```
+
+3. **Add webhook to upstream repository:**
+   - Go to repository Settings → Webhooks → Add webhook
+   - Payload URL: `https://your-domain.com/webhook`
+   - Content type: `application/json`
+   - Secret: (same as `WEBHOOK_SECRET`)
+   - Events: Select "Just the push event"
+   - Active: ✓
+
+#### Local Testing
+
+Run the webhook locally for testing:
+
+```bash
+export WEBHOOK_SECRET=test_secret
+export FLASK_DEBUG=true
+python api/github_webhook/index.py
+```
+
+The webhook will:
+- Verify GitHub signature using `X-Hub-Signature-256`
+- Process push events
+- Identify affected manifest entries
+- Trigger update checks for changed files
+
+### Running Tests
+
+Run the test suite:
+
+```bash
+# Install test dependencies
+pip install -r scripts/requirements.txt
+
+# Run all tests
+pytest tests/
+
+# Run specific test file
+pytest tests/test_vendor_update.py -v
+pytest tests/test_check_updates.py -v
+```
+
+Tests use `pytest` with `responses` for mocking HTTP requests to GitHub API.
+
+### URL Format Support
+
+The vendor system supports multiple GitHub URL formats:
+
+- `https://github.com/owner/repo/blob/main/path/to/file.stl?raw=true`
+- `https://github.com/owner/repo/raw/main/path/to/file.stl`
+- `https://raw.githubusercontent.com/owner/repo/main/path/to/file.stl`
+
+The system automatically:
+- Extracts owner, repo, path, and ref from URLs
+- Defaults to `main` branch if ref is missing
+- Handles URL-encoded paths
+
+### Reverting Vendor Changes
+
+To revert vendor metadata from component JSON files:
+
+1. **Remove vendor fields** from component JSON files manually, or
+2. **Restore from git history:**
+   ```bash
+   git checkout HEAD -- website/src/data/components/
+   ```
+
+The original `url` fields are never removed, so the site continues to work even if vendor metadata is removed.
+
+### Inspecting Vendor Status
+
+View the current vendor status:
+
+```bash
+# Check manifest
+cat manifest/vendor_manifest.json | jq '.[] | {id, status, pinned_sha, upstream_latest_sha}'
+
+# Check for updates
+python scripts/check_updates.py --manifest manifest/vendor_manifest.json
+```
+
+### Troubleshooting
+
+**Issue: Rate limit errors from GitHub API**
+- Solution: Set `GITHUB_API_TOKEN` environment variable with a personal access token
+
+**Issue: File not found at ref**
+- Solution: The script will try the default branch if the specified ref doesn't exist
+
+**Issue: Webhook signature verification fails**
+- Solution: Ensure `WEBHOOK_SECRET` matches the secret configured in GitHub webhook settings
+
+**Issue: Manifest entries not found**
+- Solution: Run `generate_manifest_from_site.py` to create/update manifest entries
+
+### Best Practices
+
+1. **Always pin to commit SHAs** (not branch names) for reproducible builds
+2. **Run `check_updates.py` regularly** to detect upstream changes
+3. **Review PRs** created by the GitHub Action before merging
+4. **Test site builds** after vendor updates to ensure compatibility
+5. **Keep manifest in version control** for tracking vendored files
+
+### Example Workflow
+
+Complete workflow for adding a new component with external files:
+
+1. **Add component to site JSON:**
+   ```json
+   {
+     "id": "new-part",
+     "url": "https://github.com/owner/repo/blob/main/file.stl?raw=true"
+   }
+   ```
+
+2. **Generate manifest:**
+   ```bash
+   python scripts/generate_manifest_from_site.py
+   ```
+
+3. **Vendor the file:**
+   ```bash
+   python scripts/vendor_update.py --entry new-part --sync-site
+   ```
+
+4. **Verify:**
+   ```bash
+   python scripts/check_updates.py
+   ```
+
+The GitHub Action will automatically monitor for future updates.