When using the GitLab storage, in what scenarios is the flow code pulled? Is there caching? My assumption is that flow code is pulled from gitlab only upon initial start and restart from a failed state. Can the flow code be pulled while some tasks are running?
k
Kevin Kho
05/26/2022, 4:13 PM
Exactly as you describe. Not while tasks are running
That can take effect next flow run. It won’t take affect for the current flow run because the first step to start a flow is loading in and reading the script. So basically it’s the same as changing Python script code after a Python script is running. It already started the process so it won’t change
m
Matthew Seligson
06/03/2022, 7:42 PM
If you restart a flow run, it will call get_flow() on the storage object right? So it will load potentially new script code
k
Kevin Kho
06/03/2022, 7:44 PM
That’s true because we don’t cache any code on our side. It will raise an error if the DAG structure itself has changed, but if it’s an edit within a Task code, Prefect doesn’t look into that
m
Matthew Seligson
06/03/2022, 7:46 PM
Right. If I have a running flow with task A running some parallel task B in a failed state, then I change B’s task code and then restart the flow from B, will B load the new code?
Matthew Seligson
06/03/2022, 8:04 PM
Every time you restart a flow, even if it is currently running, won’t it start up a new pod (if using a kubernetes runner)?
k
Kevin Kho
06/03/2022, 8:05 PM
I believe that is right. It will load the new code. Can test in a bit if it helps you. I believe it will be true for all agents
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.