https://prefect.io
Join Slack
Hi, I am experiencing some weird behavior using prefect server. I can run a flow on prefect core, it...
# ask-community
o
Hi, I am experiencing some weird behavior using prefect server. I can run a flow on prefect core, it executes completely but when I spin up the UI and run the exact same flow on a local agent (same machine and environment), I get an ModuleNotFoundError. I have modules I import as part of my flow
k
I honestly can’t think of anything other then an environment mismatch. What env is your agent running in cuz I know it’s one of pyenv or pipenv that doesn’t work (I use conda myself)
o
I started the agent in the same conda environment
Is there a different way to make sure the local agent is running in the same environment. It looks that way from the logs. It's running in an venv call "prefect"
k
If you use 
prefect agent local start
in conda, then it should be from the same environment
o
Hmmm, That's what I currently do.
Sorted this out. I had to explicitly append the path to the module I wanted to import within every task I imported it rather than once in the entire script. I guess this is related to how the task are passed to the executors
Copy code
sys.path.append(DATA_DIR) #moved inside task
logger = prefect.context.get("logger")

@task()
def get_i():
    sys.path.append(DATA_DIR)
    fetch = importlib.import_module("logistics-pipeline")
    //
	//
    <http://logger.info|logger.info>("Get  Task Completed")
a
Hey Ovo, I was experiencing the same issue, and Anna Geller's comment in a thread from 2 months ago helped me resolve it: https://prefect-community.slack.com/archives/CL09KU1K7/p1635253346416400?thread_ts=1635243185.382700&amp;cid=CL09KU1K7
❤️ 1
r
@Anna Geller @Kevin Kho is there a way to send the custom package code to S3 as part of the flow registration upload? I’d like to be able to run both locally and on ECR/EC2 (via UniversalRun). Currently I’m using S3 storage. Is the only option to use docker images?
k
No there is no way to do that for now. Recently though 
cloudpickle 2.0
has support for deepcopying of modules so maybe it will become possible. For now though, you need to either have the custom modules installed on the EC2 (if they don’t change often), or package it in a Docker container. I am not seeing how you would get this working for both ECS and EC2 simultaneously because I think both will require configuration exposed by their respective RunConfigs
😉 1
11 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by