# Run scheduled jobs (/docs/build/schedules) A scheduled job is work your hosted app does on a timer, even when no member has the app open. For example: * **Refresh token balances every 15 minutes**, so a holders leaderboard stays current. * **Pull activity from an outside service every hour** into a custom tracker, so quests that depend on it can pay out. * **Check members' quest progress every morning** and update your own records. Members don't see the jobs themselves. They see the result: standings, quests and rewards that stay up to date. Jobs run in a Worker app that Domino hosts: on the staged build in test, and on the live app once a promotion puts that build live. Static apps and development previews don't run jobs. ## What you can do in Console [#what-you-can-do-in-console] The jobs are declared in your app's code. Open **Settings → Schedules**, with **Test** or **Live** selected, to: * **See every job**: when it last ran, its status and duration, what it reported, and when it runs next. * **Read a job's history** with **Run history**. * **Run a job now** with **Run now**, for example to check a fix. * **Pause, archive or resume** all of the project's jobs in that environment. * **Unblock a job** whose last run is **Uncertain** with **Recover uncertain run**. See [Inspect and control jobs](#inspect-and-control-jobs) and [Timeouts and recovery](#timeouts-and-recovery). ## Declare a schedule [#declare-a-schedule] Add `schedules` beside `app.secrets` and `app.storage` in your existing Worker configuration, then stage the app: ```json { "app": { "kind": "worker", "schedules": [ { "name": "refresh-data", "path": "/jobs/refresh-data", "every": "15m", "description": "Refresh custom tracker data from an external service" } ] } } ``` This is a fragment of `relay.json`. Keep your existing build scripts, output directories and Worker settings. Use a route your Worker handles, and set `runWorkerFirst: true` if public assets could intercept that route. | Field | Rule | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Unique within the app. A lowercase letter followed by lowercase letters, digits or hyphens, up to 64 characters. | | `path` | An absolute route such as `/jobs/refresh-data`. Use letters, digits, underscores, hyphens and slashes; no query, hostname, empty segment or traversal. | | `every` | An integer followed by `m`, `h` or `d`, from `5m` through `365d`. | | `cron` | Instead of `every`, a five-field UTC cron expression using numeric fields, `*`, ranges, lists and steps. Minute slots must be at least five minutes apart, including across an hour boundary. Examples: `*/15 * * * *` or `0 9 * * 1-5`. | | `description` | What the job does, from 1 to 500 characters. | * **Up to 10 jobs** per app. Declare exactly one of `every` and `cron` for each. * **When they run.** Intervals start when the build becomes active. Cron schedules use UTC. A new build restarts the clock from its own activation time. * **Test and live are separate.** Each keeps its own jobs, runs and next run times. ## Verify the call [#verify-the-call] Domino calls each job with a signed `POST`. Check the signature before doing any work, so nobody else can trigger your job: ```ts import { verifyScheduledRequest } from "@domino-sdk/relay/server"; declare function refreshData(runId: string): Promise; export default { async fetch(request: Request, env: { DOMINO_SCHEDULE_KEY: string }) { if (new URL(request.url).pathname !== "/jobs/refresh-data") { return new Response("Not found", { status: 404 }); } const run = await verifyScheduledRequest(request, env.DOMINO_SCHEDULE_KEY); if (!run || run.name !== "refresh-data") { return new Response("Unauthorized", { status: 401 }); } await refreshData(run.id); return new Response(null, { status: 204 }); }, }; ``` Implement `refreshData` with your app's own bounded background work. The timestamp check limits replays but doesn't remove duplicates, so use `run.id` as an idempotency key when writing to an outside service or store. > [!DEV] > Relay binds a separate `DOMINO_SCHEDULE_KEY` secret to scheduled Worker builds. The live app's key has a different value from staging's, so a request signed for one doesn't verify on the other. The helper returns the verified run's `id`, `build`, `name`, `scheduledAt`, `startedAt` and `manual`, or `null`. It checks the method, route, exact body, HMAC-SHA-256 signature and a 60-second timestamp window. It rejects query parameters, because Relay calls only the declared bare path. It reads a clone, so the original request body stays readable. The signing key is separate from `DOMINO_MANAGEMENT_KEY`; keep both in server code, and never send either to a browser or log it. ## Feed a custom tracker [#feed-a-custom-tracker] A common job collects outside data, such as on-chain balances or activity from another service, and sends it to a [custom tracker](/docs/build/trackers#feed-another-source-including-a-blockchain-indexer). Your app decides what to fetch and how to map it to the tracker's declared schema; Domino doesn't collect custom data for you. Regular refreshes keep tracker-based quests and leaderboards current even when members don't open the app. ```ts import { hostedManagementClient, verifyScheduledRequest, type HostedManagementEnv, } from "@domino-sdk/relay/server"; import { trackerIngestSchema } from "@domino-sdk/relay/schema"; declare function collectRecords(): Promise; export default { async fetch(request: Request, env: HostedManagementEnv & { DOMINO_SCHEDULE_KEY: string }) { if (new URL(request.url).pathname !== "/jobs/refresh-data") { return new Response("Not found", { status: 404 }); } const run = await verifyScheduledRequest(request, env.DOMINO_SCHEDULE_KEY); if (!run || run.name !== "refresh-data") { return new Response("Unauthorized", { status: 401 }); } // Fetch and map records for your custom source, including its verified accounts. const input = trackerIngestSchema.parse({ actionId: run.id, records: await collectRecords(), }); await hostedManagementClient(env).trackers.ingest("external-data", input); return new Response(null, { status: 204 }); }, }; ``` Implement `collectRecords` to fetch and map your outside source. Page through large datasets and keep each run bounded. > [!DEV] > The hosted app's management key already has custom tracker ingestion permission. A stable record key updates the same record on later runs; the run ID deduplicates ingestion for this run. See [custom tracker ingestion](/docs/build/trackers#feed-another-source-including-a-blockchain-indexer) for record shapes and account ownership. ## Report what a run did [#report-what-a-run-did] Have each job report what it did: an outcome, a summary, counts and per-item notes. Console then shows it under the job and in its run history. Without a report, Domino records only the HTTP status, so a `204` reads as **Succeeded** even when some of the work failed. ```ts import { scheduleResult, verifyScheduledRequest } from "@domino-sdk/relay/server"; declare const members: string[]; declare function refreshMember(member: string): Promise; export default { async fetch(request: Request, env: { DOMINO_SCHEDULE_KEY: string }) { const run = await verifyScheduledRequest(request, env.DOMINO_SCHEDULE_KEY); if (!run || run.name !== "refresh-data") { return new Response("Unauthorized", { status: 401 }); } const failed: { level: "error"; subject: string; message: string }[] = []; for (const member of members) { try { await refreshMember(member); } catch { failed.push({ level: "error", subject: member, message: "Refresh failed" }); } } return scheduleResult({ outcome: failed.length === 0 ? "succeeded" : failed.length < members.length ? "partial" : "failed", summary: `${members.length - failed.length} of ${members.length} members refreshed`, metrics: { processed: members.length, failed: failed.length }, diagnostics: failed, }); }, }; ``` | Field | Rule | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `outcome` | `succeeded`, `partial` or `failed`. `partial` and `failed` record the run as **Failed**, whatever the HTTP status. | | `summary` | Optional, up to 500 characters. It also ends the run's error, such as `App reported a partial run: 4 of 5 members refreshed`. | | `metrics` | Optional. Up to 50 named numbers, such as counts and gauges. Names start with a letter and use letters, digits, `_`, `.` or `-`. | | `diagnostics` | Optional. Up to 50 lines, each with a `level` (`error`, `warning` or `info`), a `message` of up to 500 characters, and an optional `subject` (such as a member or record ID) and `code`. | > [!DEV] > Relay reads a JSON answer (`Content-Type: application/json`) of up to 16,384 bytes with any HTTP status, so a `500` can carry a result too. It ignores other content types. A JSON answer that isn't a valid result, or is larger, never changes the run's status; the run's error starts with `Result ignored:` and says why. `scheduleResult` throws on an invalid or oversized result, so the mistake shows up in your code. > [!WARNING] Keep results free of secrets > Anyone who can read the project in that environment sees run results. Name members and records by ID; never include tokens, keys, request headers or personal data. ## Fit a run into 30 seconds [#fit-a-run-into-30-seconds] Each run must finish within 30 seconds, including every call it makes to Domino. * **Batch tracker ingests.** Ingests into one tracker run one at a time on the server, so parallel calls into the same tracker wait for each other and gain nothing. Send one `trackers.ingest` call per tracker per run, with up to 100 records, instead of one call per member. * **Time calls from what's left.** Pass a call's `timeout` from the time remaining, such as `trackers.ingest(reference, input, { timeout })`. The client's default is 30 seconds, the whole run. * **Retry with the same action ID.** A timed-out ingest may still have been stored. Sending the same `actionId` again returns the first answer instead of ingesting twice. An action ID is 1–100 letters, digits, `_` or `-`: the run's `id`, or `${run.id}-${step}`, fits; a `:` or `.` is refused before anything is sent. * **Stop early and say so.** When time runs low, stop, report a `partial` outcome with what's left, and let the next run continue. ## Delete old evidence nightly [#delete-old-evidence-nightly] Domino already deletes submitted evidence 90 days after the decision, unless you changed your project's period or turned it off; see [how long evidence is kept](/docs/concepts/quests#how-long-evidence-is-kept). A job is how you delete it sooner, or on a different schedule per quest, to keep a shorter promise to members. This one deletes the evidence of one quest's attempts decided more than 30 days ago, every night at 03:00 UTC. See [delete old evidence on a schedule](/docs/concepts/quests#delete-old-evidence-on-a-schedule) for what a pass deletes and keeps. Deleting evidence needs `review` permission, which the hosted app's own key doesn't have, and in live a pass also needs an organization admin's Console session or personal key, or a service key. An organization admin creates a [service key](/docs/start/existing-app#5-call-domino-from-your-backend) with only **Review**, in the environment the job runs in, and you save it as an [app secret](/docs/build/hosted-app#set-app-secrets) named `RETENTION_KEY`: ```json { "app": { "kind": "worker", "secrets": { "RETENTION_KEY": { "description": "Service key with review permission, for the evidence retention job" } }, "schedules": [ { "name": "evidence-retention", "path": "/jobs/evidence-retention", "cron": "0 3 * * *", "description": "Delete selfie evidence decided more than 30 days ago" } ] } } ``` ```ts import { hostedManagementClient, scheduleResult, verifyScheduledRequest, type HostedManagementEnv, } from "@domino-sdk/relay/server"; type Env = HostedManagementEnv & { DOMINO_SCHEDULE_KEY: string; RETENTION_KEY: string }; const day = 24 * 60 * 60 * 1000; export default { async fetch(request: Request, env: Env) { const run = await verifyScheduledRequest(request, env.DOMINO_SCHEDULE_KEY); if (!run || run.name !== "evidence-retention") { return new Response("Unauthorized", { status: 401 }); } // The app's own bindings, with the service key in place of the app's key. const management = hostedManagementClient({ ...env, DOMINO_MANAGEMENT_KEY: env.RETENTION_KEY }); const decidedBefore = run.scheduledAt - 30 * day; let deleted = 0; let more = true; // Each call deletes up to 100 attempts' evidence. Stop well inside the 30 seconds; the next night continues. for (let call = 0; more && Date.now() - run.startedAt < 20_000; call++) { const pass = await management.attempts.redactEvidenceBefore({ actionId: `${run.id}-${call}`, quest: "selfie", decidedBefore, reason: "Retention: selfies are kept for 30 days", }); deleted += pass.attempts.length; more = pass.more; } return scheduleResult({ outcome: more ? "partial" : "succeeded", summary: `${deleted} attempts' evidence deleted${more ? ", more left for the next run" : ""}`, metrics: { deleted }, }); }, }; ``` Each call's action ID comes from the run, so a retried call answers its first result instead of deleting again. The service key appears as the actor on each deleted attempt and in **Audit history**. ## Read member progress [#read-member-progress] A job that checks members' quests reads each member's progress: their state on each active quest, their recent attempts and their point balances. The read stays small however long the member has taken part. ```ts import { hostedManagementClient, type HostedManagementEnv } from "@domino-sdk/relay/server"; export async function completedQuests(env: HostedManagementEnv, member: string) { const progress = await hostedManagementClient(env).members.progress(member); return progress.quests.filter((quest) => quest.completed).map((quest) => quest.quest); } ``` From a terminal, `domino members progress MEMBER --json` prints the same read. It needs `read` permission, which the hosted app's management key has. > [!DEV] > `management.members.progress(member)` returns the member's progress on each active quest (`completed`, `completionCount`, the latest `attempt`), their 25 most recent `attempts` and their point `balances`. Every list is bounded by the project's configuration. > > Avoid `management.members.get(member)` in jobs. Member detail also returns every active quest release with its evaluator code, which runs to megabytes per member in a project with many quests. It's still where the member's rewards, completion records and each streak's qualifying days are. For the member's linked wallets, read `management.members.identityStates.list(member)`, which pages through their wallet and account identities. ## Inspect and control jobs [#inspect-and-control-jobs] Open **Settings → Schedules** in Console, with **Test** or **Live** selected: * **The list** shows the active build's jobs, each one's last run time, status and duration, its error and [result](#report-what-a-run-did), and the next run. * **Run history** under a job shows its latest 20 runs, newest first, each with its ID, build, status, error and result. Domino keeps the latest 20 runs of each job. * **Run now** queues one run, which goes through the same checks as a timed run. * **Paused** or **Archived**, chosen in the state menu, stops new runs for the whole project in that environment, and **Active** restores either. Console asks for a reason before it changes the state, and **Audit history** records it. Your choice stays through new builds and promotions, and pausing test's jobs leaves live's running. Removing a job from `relay.json` and staging stops it in test; promoting that build stops it in live. Deleting the project stops all its jobs, and purging it removes their records. The same controls are in the CLI: ```sh domino schedules list --json domino schedules list --environment live --json domino schedules runs refresh-data --json domino schedules run refresh-data domino schedules pause --reason "Rotating the API key" domino schedules resume domino schedules archive --reason "App retired" ``` > [!DEV] > Without `--environment live`, `domino schedules` uses test; it ignores `RELAY_ENVIRONMENT` and the environment in `relay.json`, and warns when either names live. Its `--json` result says which environment it used in `environment`. Listing and reading runs need `read` in the environment. Running jobs, changing state and recovering a job need `publish` there. A CLI sign-in session can't run or control live jobs, so use Console or an API key for those. The management client provides `management.schedules.list()`, `.runs(name)`, `.run(name)`, `.setState("active" | "paused" | "archived", { reason })`, where the reason is optional, and `.recover(name, { stopped: true })`. The routes are `GET /management/v1/schedules`, `GET /management/v1/schedules/:name/runs`, `PUT /management/v1/schedules/state`, and `POST /management/v1/schedules/:name/run` or `/:name/recover`. ## Timeouts and recovery [#timeouts-and-recovery] > [!WARNING] Finish all work before answering > A scheduled request has a **30-second timeout**. Don't put scheduled work in `waitUntil`, start work that outlives the response, or answer before ingestion finishes. Domino can't prove that remote work stopped after a lost connection. * **One try per run.** Domino tries each timed run once and never retries it automatically. Runs missed during downtime are skipped, and the next one runs as usual. * **Failures don't stop the schedule.** A failed response, or a `partial` or `failed` outcome, records a failure, and the next run still happens. So does a run whose build has no Worker to run it, since the job never started. * **No overlap.** The same job never runs twice at once. If Domino can't tell whether a run finished, it blocks that job, even across new builds. The run's error says what Domino saw: the HTTP status and cause from the dispatch, no answer within 30 seconds, or the error that interrupted the call. An exception your app throws before answering is shown there too, so keep secrets out of error messages. Once the timeout has passed, Console shows the run as **Uncertain** and offers recovery: ### 1. Check that the job stopped [#1-check-that-the-job-stopped] Look at your app's own records and any outside work the job started, and confirm the job has stopped. ### 2. Recover the job [#2-recover-the-job] Choose **Recover uncertain run**, turn on **I confirmed the previous job has stopped**, and choose **Allow future runs**. The CLI equivalent is: ```sh domino schedules recover refresh-data --yes ``` Recovery allows future runs; it never replays the uncertain one. Fix app errors in source and stage a new build, then use **Run now** to check the fix. A fix reaches live's jobs when that build is promoted. For exactly when a run counts as failed or uncertain, see [Schedule dispatch guarantees](/docs/reference/management#schedule-dispatch-guarantees).