https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jessica Smith

06/10/2022, 12:52 PM
Curious if there is a best practice on something I'm struggling with. I have a task that creates a sql alchemy engine from a parameter, and this is passed on to the next set of tasks. However, you can't pickle a sql alchemy engine, which means I have checkpointing turned off for the Get Engine task. This means that if one of the queries fails and retries it will not have the input it needs. Do I need to just create the engine inside the task? Or is there another way people have used?
1
Is there a way I can have the GetEngine task marked to be re-ran before the query task retries?
a

Anna Geller

06/10/2022, 1:23 PM
I understand what you mean - it's considered best practice to define the sqlalchemy engine within a task, but you can also pass the connection string as a parameter value or as a Secret to downstream tasks
k

Kevin Kho

06/10/2022, 1:29 PM
You can’t retry an upstream task based on the result/state of a downstream task in 1.0
j

Jessica Smith

06/10/2022, 1:31 PM
Ah, I could replace the GetEngine task with a GetConnectionString task
🙌 1
that would do the same thing and be pickleable
💯 1
thanks for the idea, i'll go with that!
s

Stéphan Taljaard

07/15/2022, 6:44 AM
Hey @Jessica Smith. I also had a similar use-case in the past. I wanted to do Prefect-
.map()
task runs on a list of len >100, each using the SQLAlchemy eninge. Though that can be flaky, if engine breaks, many mapped task run's can't auto-recover. Curious what are your thoughts on such a use-case?
🙏 1
🙌 1
j

Jessica Smith

08/09/2022, 2:59 PM
Hey @Stéphan Taljaard, I think you sent this message right as I was switching jobs to a job that doesn't use Prefect. I'm back on the Slack now, but I doubt you still are looking for an answer for this. If you are, let me know, happy to give it some thought
🙏 1
4 Views