S
supero.docs
Documentation/Connectors/File, HTTP API, Webhook & Public Catalog

File, HTTP API, Webhook & Public Catalog

Connect Supero to flat files on S3/SFTP/a URL, poll any REST API, receive verified inbound webhooks, or one-click add a curated public API — the connector plugins that move data over plain HTTP or object storage instead of a database driver.

Overview

File, HTTP API, and Webhook are the three connector plugins that need no database driver — they move data over object storage or plain HTTP rather than a SQL connection. The Public Catalog is a shortcut on top of the HTTP API plugin: a curated list of well-known public APIs you can add with a click. Once imported, the records behave exactly like any other schema-backed data, readable and writable through the standard CRUD and query endpoints.
File and Webhook are sync-only: a managed worker copies records into your Supero domain on a schedule, on manual demand, or the instant a verified event arrives. HTTP API is sync by default but can also be bound live_read, in which case nothing is copied and each request is proxied to the REST endpoint at query time. live_readwrite is not available for it, since there is no generic write-back for an arbitrary REST endpoint.
ConnectorMoves dataModesGood for
File (csv_file)Pulls from S3, SFTP, or a URL on a manual or cron triggersyncPeriodic exports, data-lake dumps, nightly reports
HTTP API (http_api)Polls a REST endpoint on a trigger, or reads it live per requestsync; live_read (no live_readwrite)Any REST API whose schema you do not control
Webhook (webhook)Receives inbound pushes — the external system POSTs to yousync (event-driven)Real-time integrations: payments, CI, commerce platforms
Public CatalogPolls a pre-configured public REST API on a triggersync by default; live_read available like any http_api connectorZero-setup prototyping and demo data
All three plugins in this doc are single-stream sources — one connector, one logical feed — so the single entry in target.tables[] normally uses the literal source value "_default". You will see that in every example below. "_default" is only accepted while target.tables holds exactly one entry; a multi-table target must name each stream explicitly.
bash
curl https://api.supero.dev/api/v1/connectors/catalog \
  -H "Authorization: Bearer $ACCESS_TOKEN"

ℹ️ See Connectors Overview for the shared model

Every connector — including the three covered here — follows the same create, test, discover, confirm lifecycle, and the same secret-masking and domain/project linkage rules. See Connectors Overview for that shared flow and the console walkthrough, and Database Connectors or Warehouse Connectors for SQL and warehouse sources instead. For a worked, end-to-end walkthrough of standing a source up, see Connect Your Database.

File Import Connector (csv_file)

The File connector (plugin: csv_file) reads one or more files matching a path or glob pattern and parses them into records. It supports three source backends and six file formats.
source_typeRequired configNotes
s3s3_bucket, plus aws_access_key_id and aws_secret_access_key (or an attached role); s3_region defaults to us-east-1file_path is a key or glob, e.g. exports/orders/*.csv
sftpsftp_host, sftp_username, and either sftp_password or sftp_private_key; sftp_port defaults to 22; optional sftp_known_hostsfile_path may include a directory, e.g. /uploads/*.xlsx
urlurl; optional auth via url_bearer_token, or url_api_key plus url_api_key_header, or a raw url_headers mapA direct HTTPS download — no bucket or host config needed
file_formatFormat-specific config
csv / tsvcsv_delimiter (default "," or tab for tsv), csv_has_header, csv_encoding (default utf-8), csv_skip_rows, csv_quote_char
jsonjson_records_path — a JSONPath to the array of records inside the file, for exports that nest results under a key
jsonlEach line is parsed as one record; no extra config needed
parquetColumn names and types are read from the file itself; no extra config needed
excelexcel_sheet — the sheet name to read; defaults to the first sheet
file_path accepts either a single key/path or a glob (data/exports/*.csv) to pull every matching file in one run. Each file is capped at 500MB, and fail_on_error (default false) controls whether one bad file aborts the whole run or is skipped so the rest still import. archive_after_processing plus archive_path move a file aside once it has been read, so the next run does not re-import it.

⚠️ Google Cloud Storage is not supported

Google Cloud Storage is not supported as a file source today. Use s3, sftp, or url. The file config type also lists gcs and azure_blob as source_type values, and the console still shows a Google Cloud Storage tile in the file-source picker — neither works: a connector created on one saves cleanly and then fails every run.

⚠️ Do not paste an s3:// URI into file_path

file_path is a key relative to the bucket, not a URI. Pasting s3://my-bucket/exports/orders.csv makes the whole string a literal key prefix that matches nothing, and the run completes having read zero rows with no obvious explanation. Put the bucket in s3_bucket and the key or glob in file_path.

ℹ️ Some file options are worker-side only

A few options are read by the import worker but are not attributes of the published file config type: the URL-source auth fields (url_headers, url_bearer_token, url_api_key with url_api_key_header), sftp_known_hosts, and fail_on_error. Send them if you need them, then GET the connector back and confirm the field survived before relying on it.
json
{
  "name": "nightly-orders-export",
  "source": {
    "plugin": "csv_file",
    "config": {
      "source_type": "s3",
      "s3_bucket": "acme-exports",
      "s3_region": "us-east-1",
      "aws_access_key_id": "AKIA...",
      "aws_secret_access_key": "REDACTED",
      "file_path": "orders/*.csv",
      "file_format": "csv",
      "csv_delimiter": ",",
      "csv_has_header": true
    }
  },
  "target": {
    "tables": [
      { "source": "_default", "schema": "order", "operation": "upsert", "batch_size": 200, "on_error": "skip", "enabled": true }
    ]
  },
  "trigger": { "type": "schedule", "cron": "0 3 * * *", "timezone": "UTC", "allow_manual": true }
}

HTTP API Connector (http_api)

The HTTP API connector (plugin: http_api) polls any REST endpoint. The only required field is url; method defaults to GET, and timeout_seconds is bounded to 1–300 seconds (default 30). Auth, pagination, response shape, rate limiting, and retries are all optional and default to the simplest behavior — no auth, a single page, the whole response body treated as the records array.

Authentication

auth.typeConfigSent as
none{ "type": "none" } — or omit auth entirely(no credential)
api_key{ "type": "api_key", "api_key": "...", "header_name": "X-API-Key", "api_key_location": "header" }A header named by header_name, or a query parameter named by api_key_param when api_key_location is "query"
bearer{ "type": "bearer", "token": "..." }Authorization: Bearer <token>
basic{ "type": "basic", "username": "...", "password": "..." }Authorization: Basic <base64>
oauth2{ "type": "oauth2", "token_url": "...", "client_id": "...", "client_secret": "...", "scope": "..." }A client-credentials token fetch, cached and refreshed automatically before it expires
oauth2_authorization_code{ "type": "oauth2_authorization_code", "auth_url": "...", "token_url": "...", "client_id": "...", "client_secret": "...", "scope": "...", "provider": "..." }A full three-legged authorization-code flow with a signed-state CSRF guard. You authorize once after creating the connector — see the OAuth section below

⚠️ The api_key fields are flat, not nested

The credential fields sit directly on the auth object: api_key, header_name, and api_key_location. There is no nested { "api_key": { "key": ..., "location": ... } } object, and key, key_name, and location are not field names — an auth block using them authenticates as if no key had been supplied.

ℹ️ Send url, not base_url and endpoint

The plugin registry’s JSON schema for http_api declares base_url and endpoint as required, which contradicts the authoritative config type, where url is the only mandatory field. Send url. base_url and endpoint are not read.

Three-legged OAuth (oauth2_authorization_code)

Client-credentials OAuth (auth.type "oauth2") needs no user interaction — the connector mints its own token. The authorization-code flow does: someone has to sign in at the provider and grant access once. Create the connector first, then authorize it.
ActionMethod and path
Start the flow (redirects to the provider)GET /api/v1/connectors/{connector_id}/oauth/authorize
The redirect target the provider sends the user back toGET /api/v1/connectors/oauth/callback
Refresh the stored tokenPOST /api/v1/connectors/{connector_id}/oauth/refresh
Check the current authorization stateGET /api/v1/connectors/{connector_id}/oauth/status
In the admin console this is the Connect Account button on the connector’s Overview tab, which appears only for an http_api connector using this auth type. The banner beside it reports one of OAuth2 Authorized, Token Expiring Soon, OAuth2 Token Expired, or OAuth2 Not Connected, and offers Refresh Token and Reconnect. These four endpoints ship and the console uses them, but their finer request and response behaviour is less settled than the CRUD surface — treat the console flow as the reference path.

Pagination

pagination.pagination_typeKey fieldsUse for
none(default)A single response containing all records
offsetlimit, limit_param, offset_param, offset_start, max_pages?limit=100&offset=200-style APIs
pagelimit, limit_param, page_param, page_start, max_pages?page=2&per_page=100-style APIs
cursorcursor_param, cursor_initial, plus response.next_cursor_pathOpaque cursor- or token-based APIs
link_headermax_pagesAPIs that return a standard HTTP Link: <url>; rel="next" header

Response mapping

response fieldPurpose
data_pathJSONPath to the records array in the response body (e.g. "data.users"); leave unset for a root-level array
total_pathJSONPath to a total-count field, if the API reports one
next_cursor_pathWhere to read the next cursor value, for cursor pagination
next_url_pathWhere to read a full next-page URL
has_more_pathA boolean field telling the connector whether more pages remain

Rate limiting & retries

SettingDefaultPurpose
rate_limit.requests_per_second10Client-side throttle applied before each outbound request
rate_limit.retry_after_headerRetry-AfterResponse header honored when the source returns a 429
retry.max_attempts3Attempts before a single request gives up
retry.initial_delay_seconds / max_delay_seconds1.0 / 60.0Exponential-backoff bounds between retries
retry.retry_on_status429, 500, 502, 503, 504Status codes that trigger a retry instead of an immediate failure
json
{
  "name": "billing-invoices-sync",
  "source": {
    "plugin": "http_api",
    "config": {
      "url": "https://billing.example.com/v1/invoices",
      "method": "GET",
      "auth": { "type": "bearer", "token": "YOUR_TOKEN" },
      "pagination": { "pagination_type": "offset", "limit": 100, "offset_param": "offset" },
      "response": { "data_path": "data", "total_path": "meta.total" },
      "rate_limit": { "requests_per_second": 5 }
    }
  },
  "target": {
    "tables": [
      { "source": "_default", "schema": "invoice", "operation": "upsert", "batch_size": 100, "on_error": "skip", "enabled": true }
    ]
  },
  "trigger": { "type": "schedule", "cron": "*/15 * * * *", "allow_manual": true }
}

Live Reads Over REST

An http_api connector does not have to copy anything. Set a table’s mode to live_read and Supero stops syncing it: each read against that schema is planned and proxied to your REST endpoint at request time, and the rows come back typed and mapped into your schema’s fields without ever being stored. This is the same live path the database and warehouse connectors use, with REST as the dialect instead of SQL.
json
{
  "target": {
    "tables": [
      {
        "source": "_default",
        "schema": "invoice",
        "mode": "live_read",
        "live": {
          "field_map": { "id": "id", "amount": "amount", "status": "status" },
          "primary_key": ["id"],
          "tenant_binding": { "mode": "fixed", "tenant": "northwind" }
        }
      }
    ]
  }
}
What appliesDetail
Modeslive_read only. live_readwrite is rejected at save time for http_api, because there is no generic write-back for an arbitrary REST endpoint.
WritesEvery create, update, and delete against a live_read table returns 403.
Tenant scopingThe same tenant_binding rules apply as for a SQL live table — fixed, or column with a tenant_value_map, and on_unmapped_tenant defaulting to deny.
Field guardingEvery field named in a filter, sort, or projection is checked against the discovered field list, and hidden_fields are blocked as query inputs, not just stripped from responses.
Outbound target vettingEach live REST read is re-checked against the same host guard as a connection test: a URL that resolves to a private, loopback, link-local, or cloud-metadata address is refused.
Response stampReads come back with access_mode "live-ro". Use that stamp, not the row count, to confirm the read reached your endpoint.

⚠️ A live REST table still needs a binding

Like any live table, an http_api table set to live_read is unreadable until its field mapping, primary key, and tenant binding are confirmed from discovery. It fails closed rather than serving unscoped rows. If you set the mode after creation, re-run discovery so the binding is built.
For a sync-mode http_api connector, incremental delivery is configured under an incremental block on the source config, with mode set to "incremental". Note the field name differs from the database plugin: an http_api source names its cursor cursor_field, where a database source names it cursor_column.

Webhook Connector (webhook)

The Webhook connector (plugin: webhook) is push, not pull: an external system POSTs events to a URL Supero gives you, and each verified delivery becomes a record. Set trigger.type to "webhook" and Supero exposes a receiving endpoint at:
text
POST https://api.supero.dev/api/v1/webhooks/{domain_id}/{connector_id}

🚨 Get the webhook URL from support before wiring a provider to it

The console does not display the inbound webhook URL anywhere, and the receive endpoint is not published on api.supero.dev today. You can create, configure, and save a webhook connector, and its signature verification is real — but a provider pointed at the address above will not reach it. Contact support for the URL that actually receives, and confirm a test delivery lands before you rely on it in production.
auth_typeCredential fieldBehavior
signature_hmachmac_secret (plus hmac_header, hmac_algorithm, hmac_prefix)The default. Recomputes an HMAC over the raw request body and compares it to the value in the signature header
secret_headersecret_header_value (plus secret_header_name, default X-Webhook-Secret)Rejects any request missing a matching shared-secret header
basic_authbasic_username + basic_passwordStandard HTTP Basic on the inbound request
none(no credential)Any request reaching the URL is accepted — only use this behind your own network access control
auth_type defaults to signature_hmac, not none — a webhook connector saved without touching the auth control expects an HMAC signature. hmac_algorithm accepts SHA-256 (the default), SHA-1, or SHA-512.
For auth_type "signature_hmac", the connector recognizes the exact header and signature conventions of the major platforms out of the box, so you rarely need to hand-roll verification:
Providerhmac_headerSignature format
StripeStripe-Signaturet=<timestamp>,v1=<hex hmac>
GitHubX-Hub-Signature-256sha256=<hex hmac>
ShopifyX-Shopify-Hmac-SHA256base64-encoded hmac
SlackX-Slack-Signaturev0=<hex hmac>
GenericX-Webhook-Signature, or any header you namePlain HMAC-SHA256 over the raw body
FieldPurpose
payload_typeHow the inbound body is parsed: "json" (default), "form", or "raw"
wrap_single_recordWrap a single-object payload as a one-record batch — true is the common case
include_metadataAttach delivery metadata (headers, received time) alongside the parsed payload
data_pathJSONPath into the payload for the actual record — e.g. "data.object" for Stripe events
event_type_headerHeader or payload field to read the event type from, used to filter with trigger.webhook_events
allowed_event_typesAn allow-list applied at the source config, alongside the trigger-level trigger.webhook_events list
allowed_ipsAn optional inbound IP allow-list. This is the right mitigation when you cannot rely on a signature.
response_mode"async" (acknowledge immediately, process in the background) or "sync" (process before responding)
json
{
  "name": "stripe-payment-events",
  "source": {
    "plugin": "webhook",
    "config": {
      "auth_type": "signature_hmac",
      "hmac_secret": "REDACTED",
      "hmac_header": "Stripe-Signature",
      "hmac_algorithm": "sha256",
      "payload_type": "json",
      "data_path": "data.object",
      "event_type_header": "type",
      "response_mode": "async"
    }
  },
  "target": {
    "tables": [
      { "source": "_default", "schema": "payment_event", "operation": "insert", "batch_size": 50, "on_error": "skip", "enabled": true }
    ]
  },
  "trigger": {
    "type": "webhook",
    "webhook_secret": "REDACTED",
    "webhook_events": ["payment_intent.succeeded", "charge.refunded"]
  }
}

🚨 Keep the signing secret out of the payload

hmac_secret and secret_header_value are write-only — stored server-side and masked on every read, the same as a database password. A webhook trigger always ends up with a secret: if you do not supply trigger.webhook_secret, validation generates one for you, so there is no unverified configuration reachable through create or update. trigger.webhook_events is an optional allow-list: a delivery whose event type is not on the list gets a 200 acknowledgement but does not trigger an execution, so senders never see an error for event types you have not wired up yet. A disabled connector also acknowledges with a 200 without running.

Public Catalog: One-Click Public APIs

The Public Catalog is a curated, static list of well-known public REST APIs you can add as a connector with little or no setup. Selecting an entry pre-fills a working http_api source config — url, response.data_path, and an auth skeleton — so schema discovery can infer a usable schema without you writing one by hand. Every catalog connector is a plain sync http_api connector under the hood; once created it behaves exactly like a hand-written HTTP API connector, including its own schedule and manual triggers.
bash
curl "https://api.supero.dev/api/v1/connectors/catalog?category=Government%20%26%20Open%20Data" \
  -H "Authorization: Bearer $ACCESS_TOKEN"
Auth modeWhat it means
noneNo key required — the connector works immediately
byo"Bring your own" — you supply a key in a setup field the catalog entry tells you about
Catalog entryCategoryAuthWhat you get
CoinGecko — Crypto MarketsCryptononeLive prices, market cap, and volume for top cryptocurrencies
CoinCap — AssetsCryptononeReal-time crypto asset prices, supply, and market data
Alpha Vantage — Stocks & FXFinance & MarketsbyoStock, forex, and crypto time series and quotes
Precious Metals — Gold & SilverFinance & MarketsbyoSpot prices for gold, silver, platinum, and other metals
Hacker NewsNewsnoneFront-page stories and full-text search
NewsAPI — HeadlinesNewsbyoBreaking headlines and articles from thousands of sources
Remotive — Remote JobsJobsnoneOpen remote-job listings across software, design, and marketing
Arbeitnow — Job BoardJobsnoneEuropean and remote job postings
USGS — EarthquakesGovernment & Open DatanoneRecent earthquakes worldwide
Public HolidaysGovernment & Open DatanoneOfficial public holidays for any country and year
OpenFDA — Drug EventsGovernment & Open DatanoneAdverse-event and drug label data from the FDA
Global Health StatsHealth & SciencenonePer-country health and disease statistics
OpenStreetMap — Place SearchMaps & PlacesnoneGeocode place names to coordinates and addresses
Open-Meteo — GeocodingMaps & PlacesnoneSearch cities and get coordinates, country, and population
Countries of the WorldReferencenoneCountry names, capitals, regions, population, currencies, and flags
Wikipedia SearchReferencenoneFull-text search across Wikipedia articles
TMDB — Popular MoviesEntertainmentbyoPopular and trending movies with ratings and metadata
Demo ProductsCommerce & DemononeA realistic sample product catalog for prototyping

Entity-Like vs. Event-Like Data

Each entry in target.tables[] carries its own operation, which controls how an incoming record is written to the destination schema. Picking the right one is the single most important decision when you configure a table for any of these three connectors.
operationBehaviorBest for
upsert (default)Matches an incoming record against an existing one by the schema’s derived name field and updates it in place, or creates it if none existsEntity-like sources with a stable business key that repeats between syncs — customers, products, accounts
insertAlways creates a new record, even if one with the same derived name already existsEvent-like, append-only sources with no meaningful "current state" per key — webhook deliveries, page views, sensor readings, log lines
updateUpdates an existing record and skips (does not create) any incoming record that does not already existReconciling denormalized state you know was seeded elsewhere
Use upsert for dimension-style data you want deduplicated across runs, and insert for fact-style data you want to accumulate. Getting this backwards is the most common cause of surprise duplicates (insert on entity data) or silently overwritten history (upsert on event data) — the Webhook example above uses insert for exactly this reason, since a payment event is a fact that happened, not a record with ongoing state to merge into.
Whichever operation writes them, imported records are ordinary Supero records once they land — addressable by uuid through the standard CRUD endpoints and filterable through the query endpoint. The practical difference is how you typically query them afterward: entity data by its business key, event data by a time-range filter on whatever timestamp field the source provides.
target.tables[] fieldDefaultPurpose
schema(required)The Supero schema/type the records are written to
source"_default"Which upstream stream this entry maps — always "_default" for the single-stream file/HTTP/webhook plugins
operation"upsert""upsert", "insert", or "update" — see above
batch_size100Records written per batch
on_error"skip""skip" a bad record and continue, or "fail" the whole run
enabledtrueSet false to pause syncing this table without deleting the mapping

Triggering, Scheduling & Discovery

trigger.type picks how a connector’s primary run is kicked off:
trigger.typeFires whenKey fields
manualYou call the run endpointallow_manual (default true)
scheduleA cron expression firescron (5-part cron string), timezone (default UTC), reconcile_cron (optional)
webhookAn external system posts to the connector’s webhook URLwebhook_secret, webhook_events (optional allow-list)
Regardless of trigger.type, allow_manual (default true) also lets you fire an on-demand run — useful for testing a scheduled or webhook connector without waiting for the next cron tick or the next real event. Set it false to lock a connector to its schedule or webhook only. A cron must have exactly five whitespace-separated fields; a six-field "with seconds" cron is rejected rather than silently downgraded to manual. reconcile_cron is a second, less frequent schedule that ignores the incremental watermark and re-pulls everything to repair drift; in the console it can only be set on a saved connector, under Settings, not in the create wizard.

⚠️ Confirm a scheduled run actually happened

A schedule you set may not run in every environment. Confirm a run actually happened in Run History, or in GET /{connector_id}/status, before depending on the cron; contact support if it has not. A connector whose cron cannot fire comes back from the API with next_run_at set to null and a validation warning explaining why.
bash
curl -X POST https://api.supero.dev/api/v1/connectors/{connector_id}/run \
  -H "Authorization: Bearer $ACCESS_TOKEN"

curl https://api.supero.dev/api/v1/connectors/{connector_id}/status \
  -H "Authorization: Bearer $ACCESS_TOKEN"
Test the source before you create anything — test-config runs the same plugin_valid / config_valid / source_reachable / auth_valid checks discovery does, without creating a connector or writing any data:
bash
curl -X POST https://api.supero.dev/api/v1/connectors/test-config \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "source": { "plugin": "http_api", "config": { "url": "https://api.example.com/data" } } }'
Before wiring a schema by hand, run discovery — it samples the source and proposes Supero schemas via AI inference, and nothing is created until you review and confirm the proposal. See the Connectors Overview page for the full discover-then-confirm walkthrough shared by every connector type.
ActionEndpoint
Trigger a manual runPOST /api/v1/connectors/{connector_id}/run
Cancel a running executionPOST /api/v1/connectors/{connector_id}/runs/{execution_id}/cancel — best-effort; a cancel can report that a running execution is not cancellable, in which case let it finish or time out
Get connector status and recent executionsGET /api/v1/connectors/{connector_id}/status
Re-test a saved connectorPOST /api/v1/connectors/{connector_id}/test
Start schema discoveryPOST /api/v1/connectors/{connector_id}/discover
Get discovery statusGET /api/v1/connectors/{connector_id}/discover/{execution_id}
Get discovery results (inferred schemas)GET /api/v1/connectors/{connector_id}/discover/{execution_id}/results
Update a connectorPUT /api/v1/connectors/{connector_id} — the only update verb; PATCH returns 405
List source pluginsGET /api/v1/connector-plugins — note that it currently returns only http_api, csv_file, and webhook
The full control-plane surface, including the OAuth endpoints and the warehouse Query console, is listed on Connectors Overview.
Once a table is confirmed, its records read and write through the same endpoints as any native Supero schema:
python
import supero

org = supero.connect(domain_name="acme-corp", api_key="ak_...")

invoices = org.crud.list("invoice", limit=20)
recent = org.crud.query("payment_event").filter(type="charge.refunded").order_by("-created_at").all()

Next Steps

  • •Connectors Overview — the supported-sources matrix, the console walkthrough, and the full control-plane API
  • •Database Connectors — PostgreSQL, Supabase, MySQL, SQL Server, Oracle, and MongoDB sources
  • •Warehouse Connectors — Snowflake, Databricks, ClickHouse, BigQuery, Redshift, and Microsoft Fabric, live-read only
  • •Live Data, Read-Only & Security — the security model behind live reads, including live reads over REST
  • •Webhooks & Event Streaming — inbound callbacks from managed integrations, and which of them verify a signature
  • •Schema Design — how the fields, references, and extends your discovered schemas use actually work