https://prefect.io logo
Title
m

Max Watermolen

02/02/2022, 6:41 PM
Howdy, Oh man I am struggling to figure out why all my tasks are failing with
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/magnox/'")
Env: • Local Agent + Local Storage (Running in docker [python:3.10] ... yeah yeah lol) • External Postgres 12 • Prefect Server in Kubernetes (Official HELM) • Django Integrated Code: The Source is located in
/code/
We have added
/code
to the path and Python path on execution. Notes: • Sometimes a single container will work fine (using the same exact image and everything)(Cant seem to figure out why) • I have tried with and without Dask • I tried to symlink the /code -> /home/magnox to no avail
k

Kevin Kho

02/02/2022, 6:44 PM
Hey @Max Watermolen, this thread will help you with that. Can you Move some of the code snippets in the thread to keep the main channel cleaner?
m

Max Watermolen

02/02/2022, 6:48 PM
Thanks for the reply, and for sure Please help, Im lost lol Code Flow: 1. Launch.sh (Sets Env) a. Entrypoint.py i. Registration.py 1. Tasks.py tasks.py:
with Flow('AutomoxSyncOrginizations', executor=LocalDaskExecutor(), schedule=automox_sync_orgs_schedule) as automox_sync_orgs:
    x= dummy()
Registration.py:
def register(schedule=True):
    automox_sync_orgs.run_config = run_config
    automox_sync_orgs.register(
        project_name=os.getenv("RELEASE").lower(),
        labels=["REACTOR", "AUTOMOX"],
        set_schedule_active=schedule,
        add_default_labels=False,
        idempotency_key=automox_sync_orgs.serialized_hash()
    )
Agent in entry.py:
############################
# TASK IMPORTS BELOW
############################
import automox.registration

############################
# REGISTER FLOWS BELOW
############################
print(f"[+] About to Register Flow - {os.getenv('RELEASE').lower()}")
automox.registration.register(schedule=True)
print("[+] DONE Registering Flow")
############################
# LAUNCH AGENT BELOW
##########################

LocalAgent(labels=["REACTOR","AUTOMOX"], hostname_label=False, name=os.getenv("HOSTNAME"), import_paths=["/code"]).start()
Launch.sh:
#!/bin/bash

cd /code
export PYTHONPATH="/code"
export PATH="${PATH}:/code"

echo $PATH
echo $PYTHONPATH
python /code/entrypoint.py
So if im understanding I need to add
storage=Local(path="/code/automox/tasks.py", stored_as_script=True)
to the flow?
So it refrences the file on the container?
k

Kevin Kho

02/02/2022, 6:55 PM
Let me look a bit at the setup
👍 1
❤️ 1
If you want to store it as a script, use that yes. If you want Prefect to serialize it and save it in the
.prefect
folder automatically, you can leave it blank. This is where the Local agent specifically will grab it from so this path needs to exist relative to the local agent whether or not that agent is running in a container. You get me?
m

Max Watermolen

02/02/2022, 6:59 PM
Yes, So are there any downsides to storing as a script? although is there a way to force it to pickle it even if its already registered?
k

Kevin Kho

02/02/2022, 7:00 PM
No downside. In my opinion it’s preferred. I don’t think there is a point to pickle if you used script based
🚀 1
m

Max Watermolen

02/02/2022, 7:00 PM
Excellent, Ill test that and report back! Thank you so much!
k

Kevin Kho

02/02/2022, 7:02 PM
You can pair LocalAgent with LocalRun and
LocalRun
can take in a working dir like this so if you import other modules, it will have access to them
m

Max Watermolen

02/02/2022, 7:05 PM
Oh thats what I forgot to post
run_config = LocalRun(
    working_dir="/code",
    env={
        "DJANGO_SETTINGS_MODULE": "magnox.settings.base"
    }
)
That worked! Thank you! Is there a way to not have the hostname labels attached to the flows, theses popped up after the change
In the register I did set
add_default_labels=False,
k

Kevin Kho

02/02/2022, 7:39 PM
Wait, are you good now? Cuz that sounds right
m

Max Watermolen

02/02/2022, 7:40 PM
Well I somehow need to clear that hostname label
k

Kevin Kho

02/02/2022, 7:41 PM
flow.storage = Local(…, add_default_labels=False)
and then re-registering should work. Maybe you put it on the Flow object?
m

Max Watermolen

02/02/2022, 7:43 PM
I think the .register() on the flow takes it, and it is set. Somehow changing the storage to local seems to be overriding it
k

Kevin Kho

02/02/2022, 7:47 PM
That’s why you add the
add_default_labels
to the storage. This is to enforce local agents across multiple machine exclusively pick up the Local storage Flows.
m

Max Watermolen

02/02/2022, 7:47 PM
Ahh ok, I see It wasnt listed in the params
k

Kevin Kho

02/02/2022, 7:48 PM
Ah cuz it is in the base Storage class
m

Max Watermolen

02/02/2022, 7:49 PM
Yeah, I passed it so it should take as a kwarg
That did it! Thank you again for the help @Kevin Kho
👍 1