We are using prefect core and now trying to create...
# prefect-community
j
We are using prefect core and now trying to create a new
Environment
- should calling
flow.run()
use the
Executor
defined in the env if I declared the flow with something like
Flow("myflow", environment=MyEnvironment())
- or is that only used by prefect agents?
d
Hi @Jacques!
flow.run()
is intended to be a pure python process, no executor or environment necessary there. Environments and executors are only used by Prefect Agents 👍
j
Thanks, appreciated!
👍 1
d
Ah, I think I misunderstood your question
@Jacques Do you want to pass an executor to
flow.run()
to test your environment?
j
Not exactly @Dylan, we run our flows in aws lambda, so we can't use agents etc. I've created a LambdaEnvironment with an execute() that knows how to run a flow in lambda using local Dask. I was just wondering how Environments were intended to be used, but having looked at examples I think I've got an idea, and me calling
LambdaEnvironment(lambda_context).execute(flow, parameters)
seems broadly similar to what an agent would be doing - if that makes any sense. It is hard to try and fit Prefect into a serverless environment.
d
Hi @Jacques that’s very interesting!
We’d definitely love to see your
LambdaEnvironment
if you’d be interested in making a PR back to Prefect Core
j
Note that as of 0.13.8, you can configure an executor on the flow itself, so that calling
flow.run()
will use that executor.
j
Ok, that makes sense - so if I understand correctly - if you only have an environment (that includes an executor) specified and not an explicit executor as well, then there is no way for the flow to go inspect the environment and use that executor?
j
Environments are confusing (I'm currently working to redo them). In general they're only meant for configuring use with cloud/server, so
flow.run
never looks at
flow.environment
. In the future environments will only be for configuring the "job" (k8s job, fargate task, local process, lambda run, etc...) that the flow runs in, the execution once that process starts up will all be configured on the flow and should be equivalent to
flow.run()
.
Environments do too much in their current design, coupling process configuration with actual flow execution.
j
Yeah is a bit hard to understand the overlap between execution environment, storage environment and agents - but cool, will wait for updates before going further down that path then!