Skip to main content

Sync Module

This module provides high-level orchestration for schema and data synchronization workflows from Bear database to Cloudflare D1.

Module Overview​

  • Purpose: High-level orchestration for schema and data synchronization workflows from Bear database to Cloudflare D1
  • Architecture: Three-file design
    • schema.sh: Schema sync workflow (extraction, preparation, API request, validation)
    • data.sh: Data sync workflow (extraction, batching, multi-batch sync)
    • validation.sh: Post-sync validation (row counts, table existence, sample rows)

Available Functions​

Schema Sync Functions (src/lib/sync/schema.sh)​

  • sync_schema_to_d1(tables_array...): Encapsulates the complete schema sync workflow from Bear database to Cloudflare D1

Data Sync Functions (src/lib/sync/data.sh)​

  • sync_data_to_d1_workflow(tables_array...): Encapsulates the complete data sync workflow from Bear database to Cloudflare D1

Validation Functions (src/lib/sync/validation.sh)​

  • validate_sync_results(tables_array...): Encapsulates post-sync validation logic for verifying sync results

Sync Modes Documentation​

Full Sync​

  • Mode: --full-sync (default)
  • Behavior: Syncs both schema and data
  • Use case: Complete synchronization from scratch or after schema changes

Schema-Only​

  • Mode: --schema-only
  • Behavior: Syncs schema only, skips data sync
  • Use case: Schema updates without data changes

Data-Only​

  • Mode: --data-only
  • Behavior: Syncs data only, skips schema sync
  • Use case: Data updates without schema changes

Incremental​

  • Mode: --incremental
  • Behavior: Filters data by ZMODIFICATIONDATE > last_sync_timestamp
  • Use case: Regular syncs of only modified notes since last sync
  • Tracking: Last sync timestamp stored in ~/.bearflare/.last_sync

Single-Note​

  • Mode: --note-id <id>
  • Behavior: Filters data by ZUNIQUEIDENTIFIER = '<note_id>'
  • Use case: Real-time sync of a specific note after editing

Integration Guidelines​

Sourcing Order​

Sync modules should be sourced after all dependencies:

  1. Core utilities + timing helpers
  2. Logging + configuration
  3. Data processing (batching + transformations)
  4. Cloudflare D1 API integration
  5. Bear database operations
  6. Sync orchestration (last)

Global Variable Initialization​

Before calling sync functions, ensure all required global variables are initialized:

Before schema sync: TABLES_ARRAY, DROP_FIRST, VERBOSE, DRY_RUN, DEBUG_LOG, BEAR_DB_PATH, endpoint, CLOUDFLARE_API_TOKEN, TMP_FILES

Before data sync: ROWS_TO_SYNC, SOURCE_FILTERED_ROWS, SOURCE_TOTAL_ROWS, DATA_WHERE_CLAUSE, FILTER_LABEL, LIMIT_ROWS, OFFSET_ROWS, BATCH_SIZE, FORCE_OVERWRITE

Before validation: DATA_SYNCED_ROWS, DROP_FIRST, per-table maps (TABLE_SYNCED_ROWS, TABLE_D1_ROWS)