Skip to main content
Data Exports writes your messaging data as Parquet files into an S3-compatible bucket that you own, on a schedule you choose. It is a dashboard feature under Admin > Data Exports and needs the admin.data_exports.manage permission. There is no API endpoint for it.

​
What you need

  • An S3-compatible bucket. AWS S3 works natively. Backblaze B2, Cloudflare R2, Wasabi, DigitalOcean Spaces and MinIO work with their endpoint URL.
  • An access key scoped to that bucket. The minimal policy is shown in the destination dialog and needs only s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on arn:aws:s3:::YOUR_BUCKET/*.
  • Optionally, a lifecycle rule on the bucket that aborts incomplete multipart uploads after a day. We check for it and warn if it is missing.
The secret key is encrypted with AWS KMS envelope encryption and bound to the destination record. It is never logged and never returned by the API. You can rotate it by editing the destination and revoke it at any time in your own account.

​
Setting up a destination

  1. Open Admin > Data Exports and choose New destination.
  2. Enter a name, the bucket name, region, access key ID and secret. Add a prefix if you want the files under a folder, and an endpoint URL for a non-AWS provider.
  3. Pick a schedule, the datasets to export, and, for parent organizations, which child organizations to include.
  4. Save. We write a small probe object under _validation/ in your bucket. If the write fails, the destination is not created and the error from your provider is shown.

​
Schedules and windows

Every window ends five minutes before the run starts so late carrier receipts are included. Each scheduled run continues from where the previous one ended, so a paused or failed run never leaves a gap; a destination that fell behind catches up seven days per run. The first run after a destination is created covers only the most recent hour or day. A Full Export walks your whole history in seven-day chunks, up to two years back, and can be started once every 30 days per destination. Failed chunks can be retried without restarting the batch.

​
File layout

Files land under your prefix in a Hive-style path that Athena, Glue, Snowflake, Databricks, BigQuery and DuckDB discover without configuration:
  • Parquet with SNAPPY compression. Files rotate every 100,000 rows (part-000, part-001, …).
  • The year, month, day and hour folders are when the export ran, in UTC, not when the rows were created. Filter on the row timestamps for reporting periods.
  • A window with no rows writes nothing. The job still appears in the history with No rows.
  • Rows inside a file are not ordered.
  • Timestamps are Parquet TIMESTAMP_MILLIS in UTC. Phone numbers are E.164 (+15555550123); short-code senders appear as the short code itself (471016).
  • The v1 prefix is the schema version. New columns are added at the end; existing columns are never renamed or removed within a version.

​
Datasets

​
message_events

One row per outbound and inbound message. A message is exported again in a later window whenever its delivery status changes after it was first exported (for example a delivery receipt that arrives the next day). Keep the row with the greatest updated_at per message_id.

​
project_summary

One row per project, written in the window that holds its completion time. Drafts, scheduled and in-progress projects are not exported until they complete.

​
opt_out_events

One row per opt-out, in the window that holds its creation.

​
click_events

One row per click on a tracked link, in the window that holds the click.

​
Reading the files

  • Athena or Glue: crawl the v1/ prefix or create a table with partition projection on schema_type, year, month, day and hour.
  • Snowflake: create an external stage on the bucket and load with COPY INTO or Snowpipe auto-ingest.
  • Databricks: point Auto Loader (cloudFiles) at the schema_type=... prefix.
  • DuckDB: SELECT * FROM read_parquet('s3://bucket/prefix/v1/schema_type=message_events/**/*.parquet').
For message_events, deduplicate on message_id keeping the row with the greatest updated_at.

​
Job history

The Export jobs tab lists every run with dataset, status, rows, size and duration. Jobs retry up to three times on transient failures; a job that keeps failing shows the provider’s error message. A failed run leaves no partial file behind.