Manage pipelines
The Pipelines page lists every pipeline in the project with its source, dataset, schedule and last result. While a sync is running the page refreshes by itself.
Actions on a pipeline
Section titled “Actions on a pipeline”Sync now sits at the end of each row. The other actions are in the ⋯ menu next to it.
| Action | What it does |
|---|---|
| Sync now | Starts a run immediately, in addition to the schedule. |
| Clear data and re-copy (⋯) | Replaces the tables this pipeline fills with a full new copy and starts incremental tables over from the beginning. Use it after moving a pipeline to a new dataset, to fix bad data, or after a schema change. It asks you to confirm. The old rows are dropped once the new copy is in; models reading these tables see the new data when it finishes, which can take a while for big sources, and some sources only keep a limited history. |
| Edit pipeline (⋯) | Change the name, schedule, BigQuery dataset, which tables are copied and how. |
| Edit source (⋯) | Change the source’s settings or credentials. |
| Pause / Resume (⋯) | Stops or restarts scheduled runs without deleting anything. |
| Delete (⋯) | Removes the pipeline (and optionally its source). Data already in BigQuery stays. |
| History | Click a pipeline’s row to see the last runs: when, how many rows, how much data, and whether it worked. Editors also see the reason for a failure. |
Editing a pipeline
Section titled “Editing a pipeline”Open Edit pipeline. The table list is read from the source again each time, so new tables appear unchecked and tables the source no longer has disappear. Before you save, the screen tells you what the changes mean:
- New tables are copied in full on the next run.
- Changing how a table is copied can mean it’s copied again from scratch on the next run.
- Removed tables stop updating. What was already copied stays in BigQuery.
- Changing the dataset sends new data to the new dataset from the next run. Tables already copied stay where they were. Tables copied by adding new rows carry on from where they left off, so the new dataset only gets rows from then on. To fill it with everything, use Clear data and re-copy (above) after saving.
Editing a source
Section titled “Editing a source”Open Edit source. Saved secrets show as saved and are left alone unless you type a new value. When you save, the connection is checked first, so a wrong password can’t break a working pipeline. If the check fails, nothing is saved.
If you point a source at a different account or database, the tables already copied no longer match it, so check the pipeline’s tables afterwards.
When a sync fails
Section titled “When a sync fails”Open History on the pipeline. Common reasons: credentials expired or were revoked (edit the source and enter new ones), the tool’s API limits were hit (the next scheduled run usually succeeds), or a table was removed at the source (edit the pipeline). Viewers see that a run failed; editors see why.