Skip to main content

Cloudflare D1 Module

This module provides Cloudflare D1 API integration for schema and data synchronization.

Module Overview​

  • Purpose: Cloudflare D1 API integration for schema and data synchronization
  • Architecture: Two-layer design
    • api.sh: Low-level primitives (response validation, scalar queries, error extraction)
    • sync.sh: High-level orchestration (batched data sync with retry logic)

Dependencies​

  • External commands: jq, curl, mktemp, bc (optional), awk
  • Internal modules:
    • src/lib/logging/logger.sh: log() function
    • src/lib/utils/core.sh: truncate_string(), format_number() functions

API Functions (src/lib/cloudflare/d1/api.sh)​

  • d1_all_statements_succeeded(response_body): Validates that a D1 API JSON response indicates overall success AND every statement succeeded
  • query_d1_scalar(sql_statement, jq_selector): Executes a scalar SQL query against the D1 API and extracts a single value using a jq selector
  • extract_cloudflare_error_summary(response_body): Best-effort extraction of a human-readable error message from a Cloudflare/D1 JSON response
  • extract_d1_metadata(response_body): Extracts D1 response metadata (rows_written, rows_read, duration) from a successful API response. Prints space-separated values to stdout: <rows_written> <rows_read> <duration_seconds>. Missing fields are printed as "N/A"

Sync Functions (src/lib/cloudflare/d1/sync.sh)​

  • sync_data_to_d1(batch_sql, batch_num, total_batches): Syncs batched SQL to D1 with retry logic, exponential backoff, and rate limit handling
  • query_d1_row_count(table_name): Queries D1 table row count for validation workflows

API Interaction Patterns​

Endpoint Construction​

The D1 API endpoint follows this pattern:

https://api.cloudflare.com/client/v4/accounts/{account_id}/d1/database/{database_id}/query

Authentication​

  • Method: Bearer token authentication
  • Header: Authorization: Bearer ${CLOUDFLARE_API_TOKEN}

Request Format​

  • Method: POST
  • Content-Type: application/json
  • Payload structure: {"sql": "SQL statement(s)"}

Retry Logic and Rate Limiting​

  • Maximum retry attempts: 3
  • Initial backoff: 1 second
  • Backoff algorithm: Exponential (doubles each retry: 1s → 2s → 4s)
  • Retry triggers: HTTP 429 (rate limit), HTTP 5xx (server errors), Network errors

Response Metadata​

D1 API responses include metadata in the result[0].meta object:

  • rows_written: Number of rows written/affected by the query
  • rows_read: Number of rows read by the query
  • duration: Query execution time in seconds (decimal)

The extract_d1_metadata() function extracts these fields for logging and monitoring purposes. The sync functions log structured INFO-level messages with these metrics:

  • DEBUG level: Full response bodies for detailed debugging
  • INFO level: Structured metadata messages with formatted row counts and execution time

Example INFO log output:

[2026-01-12 10:30:45] INFO: D1 batch executed: 1,250 rows written, 1,250 rows read, 45ms

Integration Guidelines​

Sourcing Order​

  1. Source api.sh first (provides d1_all_statements_succeeded())
  2. Source sync.sh second (depends on api.sh)

Global Variable Initialization​

Caller must initialize these variables before using module functions:

endpoint="https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/d1/database/${CLOUDFLARE_D1_DATABASE_ID}/query"
CLOUDFLARE_API_TOKEN="${CLOUDFLARE_API_TOKEN}"
TMP_FILES=()