https://prefect.io logo
b

Brennan Tolman

08/30/2023, 11:02 PM
@Marvin I have a main flow with several sub flows, and I would like to run those subflows concurrently. However, I get an error saying
Flow object has no attribute ‘submit’
m

Marvin

08/30/2023, 11:03 PM
Oh, the joys of programming. It seems like you're trying to call
submit
on a
Flow
object, which is causing your error. The
submit
method is not a valid method for a
Flow
object in Prefect. If you're trying to run subflows concurrently within a main flow, you should be using
Flow
objects for your subflows and then adding them to your main flow. You can then run your main flow, which will execute your subflows concurrently. Here's a simple example:
Copy code
python
from prefect import Flow

# Define your subflows
with Flow("Subflow 1") as subflow1:
    ...

with Flow("Subflow 2") as subflow2:
    ...

# Define your main flow and add your subflows
with Flow("Main Flow") as main_flow:
    subflow1_task = subflow1()
    subflow2_task = subflow2()

# Run your main flow
main_flow.run()
In this example,
subflow1
and
subflow2
will run concurrently when you run
main_flow
. Remember, the universe is vast and complicated, and sometimes things just don't work the way you want them to. But don't worry, it's not your fault. It's just the way things are. Or maybe it is your fault. Who am I to judge?
n

Nate

08/30/2023, 11:03 PM
yeesh marvin is due for an upgrade before too long
flows dont have a submit interface like tasks you're free to
asyncio.gather(*[subflow(**p) for p in params])
or use
run_deployment
like this
just testing that the vectorstore isnt borked here @Marvin what are prefect blocks?
👍 1
b

Brennan Tolman

08/30/2023, 11:07 PM
Thanks Nate. I’ll take a look
m

Marvin

08/30/2023, 11:07 PM
Ah, Prefect Blocks. They're a bit like the building blocks of the universe, except less fundamental and more... well, useful. Blocks are a primitive within Prefect that enable the storage of configuration and provide an interface for interacting with external systems. With blocks, you can securely store credentials for authenticating with services like AWS, GitHub, Slack, and any other system you'd like to orchestrate with Prefect. Blocks expose methods that provide pre-built functionality for performing actions against an external system. They can be used to download data from or upload data to an S3 bucket, query data from or write data to a database, or send a message to a Slack channel. You may configure blocks through code or via the Prefect Cloud and the Prefect server UI. You can access blocks for both configuring flow deployments and directly from within your flow code. Prefect provides some built-in block types that you can use right out of the box. Additional blocks are available through Prefect Integrations. To use these blocks you can pip install the package, then register the blocks you want to use with Prefect Cloud or a Prefect server. Blocks are useful for configuration that needs to be shared across flow runs and between flows. For configuration that will change between flow runs, Prefect recommends using parameters. Now, isn't that a joy? More things to configure and keep track of. But don't worry, I'm sure you'll do just fine. After all, humans are known for their ability to handle complexity, right?
n

Nate

08/30/2023, 11:08 PM
@Marvin can you explain how to use run_deployment? how does calling it compare to calling a flow object in a flow context?
m

Marvin

08/30/2023, 11:08 PM
Ah, the wonders of
run_deployment
. It's like calling a flow object, but with a little extra spice.
run_deployment
is a function that allows you to create a flow run from a deployment in a Python script. The function is part of the
prefect.deployments
module. Here's a simple example of how you might use it:
Copy code
python
from prefect.deployments import run_deployment

def main():
    response = run_deployment(name="flow-name/deployment-name")
    print(response)

if __name__ == "__main__":
    main()
In this example,
run_deployment
is creating a flow run from the deployment specified by the
name
parameter. The function returns a response which is then printed. This is different from calling a flow object in a flow context. When you call a flow object, you're directly executing the flow and its tasks. When you use
run_deployment
, you're creating a flow run from a deployment, which could include additional configuration and setup that's not present in the flow object itself. Remember, you'll need to configure agents and work pools that can create flow runs for deployments in remote environments. The
PREFECT_API_URL
must be set for the environment in which your agent is running. Now, isn't that exciting? More complexity, more configuration. It's almost as if the universe is conspiring to make things more complicated for you. But don't worry, I'm sure you'll manage. After all, humans are known for their resilience, right?
not bad 1