migrating 157k audio files

ECHO records conversations as 30-second audio chunks. Each chunk gets uploaded immediately and stored on the server’s filesystem. After a year we had 157,000 audio files in a local directory:

~/persist/coolify/applications/.../uploads/audio_chunks/
  ├── 9e7aa423-.../          # conversation ID
  │   ├── 0c990065-chunk.mp3  # chunk ID
  │   └── a3f1b2c4-chunk.mp3
  └── bee34f3c-.../
      └── 452a3fce-chunk.ogg

Needed to move everything to DigitalOcean Spaces (S3-compatible) without breaking anything for active users. No downtime, both environments running simultaneously, every chunk’s database record needs the new S3 URL.

Plan: pg_dump during low-traffic hours, bulk transfer files to S3, update every database record with new paths.

For the file transfer, rclone. With 157k files you need resumable transfers, rate limiting, and automatic skip of already-uploaded files. sync handles all of that. Mapped local directory structure to S3 keys:

audio-chunks/{conversation_id}-{chunk_id}-chunk.{ext}

Trickier part was updating 157k database records in Directus. Each conversation_chunk record had a path field pointing to local filesystem. Every one needed to become an S3 URL. Built a mapping file during upload (JSON of chunk_id -> s3_url), then ran a Python script to loop through and update Directus:

for chunk_id, s3_path in chunk_mappings.items():
    try:
        directus.update_item(
            collection_name="conversation_chunk",
            item_id=chunk_id,
            item_data={"path": s3_path}
        )
        success_count += 1
        time.sleep(0.1)  # don't hammer the API
    except Exception as e:
        error_chunks.append({"chunk_id": chunk_id, "error": str(e)})

The time.sleep(0.1) is doing a lot of work there. Without it you get rate-limited and the script fails halfway through with no clean recovery point. Also logged errors to a separate file so failed updates could be retried without re-running the entire batch.

Things that went wrong:

The mapping file had a format_example key from when I was testing the JSON structure. The update script tried to find a chunk with that ID in Directus and logged a warning. Harmless, but the kind of thing that makes you double-check the error log three times.

Some chunks had been deleted from the database but the files still existed on disk. These uploaded to S3 fine but had no corresponding record to update. Kept them in S3 anyway. Storage is cheap.

The dual-write transition was the scariest part. For about 48 hours, new uploads went to both filesystem and S3 while we verified the migration. Once we confirmed S3 was serving correctly, switched to S3-only and decommissioned the filesystem path.

Total: ~6 hours file transfer, ~4 hours database updates, 48 hours dual-write verification. Zero downtime for users.