https://prefect.io logo
j

Jacques

10/02/2020, 2:27 PM
We are using prefect core and now trying to create a new
Environment
- should calling
flow.run()
use the
Executor
defined in the env if I declared the flow with something like
Flow("myflow", environment=MyEnvironment())
- or is that only used by prefect agents?
d

Dylan

10/02/2020, 2:30 PM
Hi @Jacques!
flow.run()
is intended to be a pure python process, no executor or environment necessary there. Environments and executors are only used by Prefect Agents 👍
j

Jacques

10/02/2020, 2:30 PM
Thanks, appreciated!
👍 1
d

Dylan

10/02/2020, 3:29 PM
Ah, I think I misunderstood your question
@Jacques Do you want to pass an executor to
flow.run()
to test your environment?
j

Jacques

10/06/2020, 1:56 PM
Not exactly @Dylan, we run our flows in aws lambda, so we can't use agents etc. I've created a LambdaEnvironment with an execute() that knows how to run a flow in lambda using local Dask. I was just wondering how Environments were intended to be used, but having looked at examples I think I've got an idea, and me calling
LambdaEnvironment(lambda_context).execute(flow, parameters)
seems broadly similar to what an agent would be doing - if that makes any sense. It is hard to try and fit Prefect into a serverless environment.
d

Dylan

10/06/2020, 2:27 PM
Hi @Jacques that’s very interesting!
We’d definitely love to see your
LambdaEnvironment
if you’d be interested in making a PR back to Prefect Core
j

Jim Crist-Harif

10/06/2020, 2:30 PM
Note that as of 0.13.8, you can configure an executor on the flow itself, so that calling
flow.run()
will use that executor.
j

Jacques

10/06/2020, 2:33 PM
Ok, that makes sense - so if I understand correctly - if you only have an environment (that includes an executor) specified and not an explicit executor as well, then there is no way for the flow to go inspect the environment and use that executor?
j

Jim Crist-Harif

10/06/2020, 2:36 PM
Environments are confusing (I'm currently working to redo them). In general they're only meant for configuring use with cloud/server, so
flow.run
never looks at
flow.environment
. In the future environments will only be for configuring the "job" (k8s job, fargate task, local process, lambda run, etc...) that the flow runs in, the execution once that process starts up will all be configured on the flow and should be equivalent to
flow.run()
.
Environments do too much in their current design, coupling process configuration with actual flow execution.
j

Jacques

10/06/2020, 2:44 PM
Yeah is a bit hard to understand the overlap between execution environment, storage environment and agents - but cool, will wait for updates before going further down that path then!