Attach Pipeline

[vasco-santos]

Attach Pipeline is responsible for pulling CAR files written in a S3 bucket and streamming them into R2.

Needs:

• CF Queue access

Architecture

The attach pipeline is composed by:

• a Gateway API running in Cloudlare Workers
• a Attach Queue to trigger requests to pull CARs
• a Car Puller handler running in Cloudflare Workers
• a R2 bucket

The Gateway endpoint receives requests with batches of CAR URLs (from S3) to be pulled into R2. The Car Puller handler, which pulls CAR files from S3 and write them into R2 (mirroring with same key format carCid/carCid.car).