Hi. I have a prefect flow, that has a task (among ...
# prefect-server
Hi. I have a prefect flow, that has a task (among others) that execute several commands in a EC2 Instance. The thing is that task takes about 3 or 4 days to finish, but when it has been running for 12 hours, it crashes because of timeout. Is this an expected behavior ? Does prefect tasks supports executions over that time? Is there a way to keep running it?
I don’t think this is expected on the Prefect side and you have have long executions. Are you using boto3 and does boto3 have a default timeout? Is it exactly 12 hours?
What is the use case that takes a while? Maybe if you give more details, we can find a way to make this work?
It takes too long, because of the script we are running. That time is expected, since we have too many data to process. I will check boto3 timeout.
Is it a single task that takes 3 to 4 days?
Yes, it is
On the Prefect side, it can die from memory running out. What is your error message?
@Santiago Gonzalez you can try setting the flow’s heartbeat mode to thread on your run configuration for that job. I tested this today with a long running Spark job and it helped with that. But, as Kevin mentioned not sure if this will work for a job that runs for days, but it’s worth trying:
Copy code
run_config=UniversalRun(env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"})
I ’ve just realized that the issue was in the code, since we were setting
in boto3
send command
to 12 hours. Thanks anyway @Anna Geller.
🙌 1