with a local agent and local executor is there an easy way t Prefect Community #ask-community

with a local agent and local executor, is there an...

Daniel Ahn

03/05/2021, 6:16 PM

with a local agent and local executor, is there an easy way to keep track of the location of an external file? i'm trying to do spark-submit with the

ShellTask

, but it needs the location of the file i'm submitting with. Right now, I've hardcoded the location in my file, and switching to that location with

helper_script

option on

ShellTask

👀 1

Daniel Ahn

03/05/2021, 6:21 PM

Copy code

from prefect import task, Flow, Parameter
from prefect.tasks.shell import ShellTask

workingPath = "<location where pi.py lives>"
runPiSparkJob = ShellTask(helper_script=f"cd {workingPath}")

with Flow("area of circle") as flow:
    runPiSparkJob(command='spark-submit pi.py 10')

Daniel Ahn

03/05/2021, 6:22 PM

when I run everything locally without server/agent, I simply placed the pi.py next to my flow python script. And it was convenient because that file is already in my working directory

Daniel Ahn

03/05/2021, 6:24 PM

if i can place an external scripts relative to the flow script, and have reference to it during agent's runtime, it would be nice

nicholas

03/05/2021, 8:43 PM

Hi @Daniel Ahn - there are a few ways you could do this, one simple way would be to provide your local agent with the

import_paths

flag, which is a list of strings refs to import locations you want flows run through that agent to be able to access

Daniel Ahn

03/08/2021, 12:39 PM

@nicholas thank you for your suggestion. interesting.. not sure if it's going to scale when multiple flows with different dependencies are schedule with a local agent.

5 Views

Open in Slack

Previous Next