Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I am trying to run a stitch job using the code exactly how it shows here. <https://alessandrolollo.github.io/prefect-stitch/>. I'm getting an error that says "TypeError: missing a required argument: 'credentials'".

Hi <@U03VBV4PA6T> - the docs for that collection appear to be out of date

from <https://github.com/AlessandroLollo/prefect-stitch/blob/main/prefect_stitch/tasks.py|this>, it looks like you'll need to create a `StitchCredentials` block and pass that into `start_replication_job` ,

where you could either register the block with your workspace and then create and instance of it in the UI (recommended)

*in the CLI*
`prefect block register -m prefect_stitch.credentials` (you only have to do this once)

*in your flow*
```from prefect_stitch.credentials import StitchCredentials

@flow
def your_flow():
   stitch_creds = StitchCredentials.load("my-new-creds")

   start_replication_job(credentials=stitch_creds) ```
or you could do everything programmatically (and keep your access token in a <https://docs.prefect.io/api-ref/prefect/blocks/system/?h=secret#prefect.blocks.system.Secret|Secret> block)

*in your flow*
```from prefect_stitch.credentials import StitchCredentials

@flow
def your_flow():
   stitch_creds = StitchCredentials(access_token=Secret.load("my-stitch-access-token"))

   start_replication_job(credentials=stitch_creds) ```

<@U02GMEZU18B> I was able to successfully trigger the stitch job. But the flow run completes as soon as the stitch job is triggered. This means there is no indication of when it actually loads data. And my hope was not run the next flow until the load is complete. Is there anyway to do this?

Ahh I see, the <https://github.com/AlessandroLollo/prefect-stitch/blob/main/prefect_stitch/tasks.py|task> is written by the maintainer (which is not Prefect in this case) to just hit the endpoint and then return.

We usually like to implement a `verb_and_wait_for_completion` which involves polling the status of the job that's kicked off, but I'm noticing <https://www.stitchdata.com/docs/developers/stitch-connect/api#replication-job--object|here> that the implementor may not have done so because there's no endpoint to check a replication job status that I can find, only start and stop.

This looks like a limitation of stitch, but if there's an endpoint to check job status that I'm missing, I can open an issue to add support for that

Got it thanks. After digging in, there isn’t an endpoint to check the job status. The job is broken up into extraction and load. And it is possible to check the status of both of those through the API. It’s a bit tricky though because there isn’t a job ID linking each part. And a load only occurs if an extraction results in incremental changes. There also can be multiple loads that need to finish if there are multiple tables with incremental changes. However, the API doesn’t indicate if there are loads that need to done and how many/which ones after the extraction is complete. 

Anyway I just put together a hacky solution using the extraction and load status through the API that isn’t perfect but hopefully it will work! 

if your solution ends up working for you, feel free to open an issue on prefect-stitch and then we can work on getting that functionality into the collection one way or another!