Hi. I have a prefect flow, that has a task (among others) that execute several commands in a EC2 Instance. The thing is that task takes about 3 or 4 days to finish, but when it has been running for 12 hours, it crashes because of timeout. Is this an expected behavior ? Does prefect tasks supports executions over that time? Is there a way to keep running it?

I don’t think this is expected on the Prefect side and you have have long executions. Are you using boto3 and does boto3 have a default timeout? Is it exactly 12 hours?

It takes too long, because of the script we are running. That time is expected, since we have too many data to process. I will check boto3 timeout.

Is it a single task that takes 3 to 4 days?

Yes, it is

On the Prefect side, it can die from memory running out. What is your error message?

@Santiago Gonzalez you can try setting the flow’s heartbeat mode to thread on your run configuration for that job. I tested this today with a long running Spark job and it helped with that. But, as Kevin mentioned not sure if this will work for a job that runs for days, but it’s worth trying:

run_config=UniversalRun(env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})

I ’ve just realized that the issue was in the code, since we were setting

executionTimeout

in boto3

SSM

send command

to 12 hours. Thanks anyway @Anna Geller.