https://prefect.io logo
#prefect-server
Title
# prefect-server
l

Lana Dann

06/16/2022, 7:51 PM
is there a way to get additional insight to what api calls are being made under the hood of the FivetranSyncTask? i’m running it like so and for some reason it’s running all of my syncs as manual syncs even though i have manual set to
False
. Code in thread
k

Kevin Kho

06/16/2022, 7:56 PM
What version of Prefect are you on? Could you move the code part to the thread to make the main channel cleaner when you get the chance?
l

Lana Dann

06/16/2022, 7:58 PM
Copy code
@task
def fivetran_sync_map_fn(connector_info: tuple[str, str, bool]) -> dict[str]:
    """
    This represents a task in a flow that would trigger a Fivetran sync.
    To use this task, you first need to set up your connector in the Fivetran dashboard:
        <https://fivetran.com/dashboard>
    You can retrieve the `connector_id` parameter by clicking into the connector under the "Setup" tab - it's titled "Fivetran Connector ID".
    Setting this task will override the sync schedule set in the UI by default. If you don't want this, then set the param `manual=True`.

    :param connector_info (tuple[str, str, bool]): tuple of (fivetran connector id, task name, manual)

    :return (dict): connector_id (str) and succeeded_at (timestamp str)
    """
    connector_id, _, manual = connector_info
    fivetran = FivetranSyncTask(connector_id=connector_id)
    status = fivetran.run(
        api_key=API_KEY,
        api_secret=API_SECRET,
        poll_status_every_n_seconds=20,
        manual=manual,
    )
    return status
even when i hard code it and just run
Copy code
status = fivetran.run(
        api_key=API_KEY,
        api_secret=API_SECRET,
        poll_status_every_n_seconds=20,
        manual=False,
    )
it’s kicking off as a manual run. but i’m not sure what tools i have to dig into this deeper
k

Kevin Kho

06/16/2022, 7:58 PM
I think the manual stuff was added in 0.15.7
l

Lana Dann

06/16/2022, 7:58 PM
i’m on prefect 1.0
i actually tested this before and it worked and it worked so it’s v confusing/surprising lol
k

Kevin Kho

06/16/2022, 7:59 PM
I don’t have a better suggestion for the calls under the hood other than reading the task code 😅
The manual usage seems pretty simple . You could add debug logs to your local prefect to get more logs?
l

Lana Dann

06/16/2022, 8:08 PM
what debug logs should i add though? i basically want to see if that
if manual
block is being run but i don’t think i can add a debugger there right?
actually maybe i can locally
k

Kevin Kho

06/16/2022, 8:11 PM
Yes you can locally by editing the prefect package and just see which
if
path it took
l

Lana Dann

06/16/2022, 8:42 PM
okay, after debugging i actually think that maybe fivetran changed something but now your documentation may be incorrect:
Copy code
Args:
            - api_key (str): `API key` per <https://fivetran.com/account/settings>; should be secret!
            - api_secret (str): `API secret` per <https://fivetran.com/account/settings>; should be secret!
            - connector_id (str, optional): if provided, will overwrite value provided at init.
            - poll_status_every_n_seconds (int, optional): this task polls the Fivetran API for status,
                if provided this value will override the default polling time of 15 seconds.
            - manual (bool, optional): if provided, will overwrite Prefect's changes
                to the Fivetran connector's schedule, keeping it on Fivetran auto scheduling
^ i interpret that as if
manual=True
then prefect won’t override fivetran’s scheduling. from my tests it seems that setting
manual=True
does override fivetran’s scheduling, and setting
manual=False
won’t override their scheduling https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/fivetran/fivetran.py
k

Kevin Kho

06/16/2022, 8:50 PM
Gotcha ok will take a look at it. Thanks for raising! Does that mean manual=True is what you intend?
l

Lana Dann

06/16/2022, 8:52 PM
yes, but i was stuck doing the opposite of what i wanted to do but at least we have a fix that doesn’t require a patch on your side. i think you just have to change the documentation. for what it’s worth i tested this earlier and it worked as you documented it. but i’m not sure if something changed on their side because i don’t see it in the release notes
🙏 1
k

Kevin Kho

06/16/2022, 8:57 PM
Thanks for the info and looking into it
Just got to looking at this, and agree that the docstring is weirdly worded. Saw no changes to both Prefect and Fivetran either 🤷‍♂️. Will open an issue just for a docstring change.
5 Views