Hey everyone, I have a probably noob question. I have created a fairly easy flow with 3 steps, that does some calculations. The execution usually takes 20-30 mins. When I'm trying to register the flow it seems that prefect evaluates it and start calculating. I haven't left it run, but probably after execution ends the flow will be registered, but that doesn't make any sense. Shouldn't the register just pack it and send to the server, or am I missing something here?

Hi @Dimosthenis Schizas - could you share the basics of your flow code? (you can omit task internals)

Hey nicholas, thanks for your fast response. Here:

with Flow(name="sub_states_transformation") as flow:
    raw_df = get_subscription_data()
    transformed_df = transform_to_subscription_state(raw_df)
    load_to_snowflake(transformed_df)

flow.storage = GitHub(
    repo="data", path="flows/subscription_states.py",
    access_token_secret=PrefectSecret("GITHUB_ACCESS_TOKEN")
)

Could you share the rest of the script, including how you're instantiating your tasks?

Please ignore me next time! I have forgotten to add the task decorator. I feel so dumb

Haha we will never ignore! Easy to overlook something like that, glad you got it figured out 🙂

But I have another question that is still valid. I have a problem to organize my project. My idea was to have some generic methods in a folder called

generic

, then have the tasks in a

tasks

folder and the flows in a separate folder named

flows

. All folders are in the same level, so when I tried to register the flows in

flows

directory I got importerror

Ah - since you're using GitHub storage, your agent will only pull the script reference for your flow; it doesn't pull additional dependencies as well and can only reference those installed in your execution environment. You'll need to choose a different storage method like Docker to do this or put your tasks in the primary flow script

Thanks, one more reason to move to docker

While the upfront mental cost for Docker might be a little higher, having discreet controlled containers for your flows is incredibly valuable, and makes deployment a lot easier so I definitely recommend 👍