https://prefect.io logo
Title
a

Ashley Felber

12/13/2022, 7:21 PM
I am trying to run a stitch job using the code exactly how it shows here. https://alessandrolollo.github.io/prefect-stitch/. I'm getting an error that says "TypeError: missing a required argument: 'credentials'".
n

Nate

12/13/2022, 7:45 PM
Hi @Ashley Felber - the docs for that collection appear to be out of date from this, it looks like you'll need to create a
StitchCredentials
block and pass that into
start_replication_job
, where you could either register the block with your workspace and then create and instance of it in the UI (recommended) in the CLI
prefect block register -m prefect_stitch.credentials
(you only have to do this once) in your flow
from prefect_stitch.credentials import StitchCredentials

@flow
def your_flow():
   stitch_creds = StitchCredentials.load("my-new-creds")

   start_replication_job(credentials=stitch_creds)
or you could do everything programmatically (and keep your access token in a Secret block) in your flow
from prefect_stitch.credentials import StitchCredentials

@flow
def your_flow():
   stitch_creds = StitchCredentials(access_token=Secret.load("my-stitch-access-token"))

   start_replication_job(credentials=stitch_creds)
a

Ashley Felber

12/13/2022, 8:22 PM
Great I will try that
@Nate I was able to successfully trigger the stitch job. But the flow run completes as soon as the stitch job is triggered. This means there is no indication of when it actually loads data. And my hope was not run the next flow until the load is complete. Is there anyway to do this?
n

Nate

12/14/2022, 3:51 PM
Ahh I see, the task is written by the maintainer (which is not Prefect in this case) to just hit the endpoint and then return. We usually like to implement a
verb_and_wait_for_completion
which involves polling the status of the job that's kicked off, but I'm noticing here that the implementor may not have done so because there's no endpoint to check a replication job status that I can find, only start and stop. This looks like a limitation of stitch, but if there's an endpoint to check job status that I'm missing, I can open an issue to add support for that
a

Ashley Felber

12/14/2022, 11:43 PM
Got it thanks. After digging in, there isn’t an endpoint to check the job status. The job is broken up into extraction and load. And it is possible to check the status of both of those through the API. It’s a bit tricky though because there isn’t a job ID linking each part. And a load only occurs if an extraction results in incremental changes. There also can be multiple loads that need to finish if there are multiple tables with incremental changes. However, the API doesn’t indicate if there are loads that need to done and how many/which ones after the extraction is complete. Anyway I just put together a hacky solution using the extraction and load status through the API that isn’t perfect but hopefully it will work!
👍 1
n

Nate

12/15/2022, 12:00 AM
if your solution ends up working for you, feel free to open an issue on prefect-stitch and then we can work on getting that functionality into the collection one way or another!