Lana Dann
06/16/2022, 7:51 PMFalse
. Code in threadKevin Kho
06/16/2022, 7:56 PMLana Dann
06/16/2022, 7:58 PM@task
def fivetran_sync_map_fn(connector_info: tuple[str, str, bool]) -> dict[str]:
"""
This represents a task in a flow that would trigger a Fivetran sync.
To use this task, you first need to set up your connector in the Fivetran dashboard:
<https://fivetran.com/dashboard>
You can retrieve the `connector_id` parameter by clicking into the connector under the "Setup" tab - it's titled "Fivetran Connector ID".
Setting this task will override the sync schedule set in the UI by default. If you don't want this, then set the param `manual=True`.
:param connector_info (tuple[str, str, bool]): tuple of (fivetran connector id, task name, manual)
:return (dict): connector_id (str) and succeeded_at (timestamp str)
"""
connector_id, _, manual = connector_info
fivetran = FivetranSyncTask(connector_id=connector_id)
status = fivetran.run(
api_key=API_KEY,
api_secret=API_SECRET,
poll_status_every_n_seconds=20,
manual=manual,
)
return status
even when i hard code it and just run
status = fivetran.run(
api_key=API_KEY,
api_secret=API_SECRET,
poll_status_every_n_seconds=20,
manual=False,
)
it’s kicking off as a manual run. but i’m not sure what tools i have to dig into this deeperKevin Kho
06/16/2022, 7:58 PMLana Dann
06/16/2022, 7:58 PMKevin Kho
06/16/2022, 7:59 PMLana Dann
06/16/2022, 8:08 PMif manual
block is being run but i don’t think i can add a debugger there right?Kevin Kho
06/16/2022, 8:11 PMif
path it tookLana Dann
06/16/2022, 8:42 PMArgs:
- api_key (str): `API key` per <https://fivetran.com/account/settings>; should be secret!
- api_secret (str): `API secret` per <https://fivetran.com/account/settings>; should be secret!
- connector_id (str, optional): if provided, will overwrite value provided at init.
- poll_status_every_n_seconds (int, optional): this task polls the Fivetran API for status,
if provided this value will override the default polling time of 15 seconds.
- manual (bool, optional): if provided, will overwrite Prefect's changes
to the Fivetran connector's schedule, keeping it on Fivetran auto scheduling
^ i interpret that as if manual=True
then prefect won’t override fivetran’s scheduling. from my tests it seems that setting manual=True
does override fivetran’s scheduling, and setting manual=False
won’t override their scheduling
https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/fivetran/fivetran.pyKevin Kho
06/16/2022, 8:50 PMLana Dann
06/16/2022, 8:52 PMKevin Kho
06/16/2022, 8:57 PM