Hui Zheng01/05/2021, 9:08 PM
Jim Crist-Harif01/05/2021, 9:11 PM
Hui Zheng01/05/2021, 9:13 PM
which contains only the task and flow definition.
@task def task_1() @task def task_2() with Flow("my_flow") as flow: ...
from my_flow import flow with raise_on_exception(): fetch_results = my_flow.run() ...
from my_flow import flow flow.schedule = Schedule(...) flow.storage = Docker(...) if push_to_cloud: flow.register(...) else: # build for local test flow.storage.build(push=False)
Jim Crist-Harif01/05/2021, 9:22 PM
storage class, and specifying a fixed docker image as part of the flow's
. This makes it easier to separate the deployment environment from the flow code itself (since the flow code is stored external to the image). • I recommend defining and registering a flow in the same file, something like:
if the flow is imported (say in your tests or as part of other code) it won't be registered, but running the flow file as a script will still register it. Since your flow's module is then
with Flow(...) as flow: ... if __name__ == "__main__": flow.register(...)
when registered, this means that cloudpickle will successfully serialize your flow when using pickle-based storage. If you're storing as a python file (
) then this doesn't matter.
, which fully serializes all function definitions written in the
module. Functions written in other modules (say imported from a different file) are only serialized by reference (e.g.
) meaning that the source for those functions must be already available in the deserialization environment. In prefect's case, this usually comes up when dealing with user-written tasks. If you write all your tasks in the same file you call
in, then run that as a script all your task definitions will be stored in the
class. If you write them in other files, those files will need to be installed in your execution environment (e.g. a docker image) for things to work properly. However, if you set
class, the flow will be loaded from a
file again on execution, which avoids this issue (but for most storage classes puts the burden of getting the
file to that location on you). See https://docs.prefect.io/core/idioms/file-based.html for more info.
Hui Zheng01/05/2021, 9:30 PM
file, which is a different file from what we run
couldn’t get some class object type global variables such as
local_tz = pytz.timezone(tz)
Jim Crist-Harif01/05/2021, 9:59 PM