Max Watermolen
02/02/2022, 6:41 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/magnox/'")
Env:
• Local Agent + Local Storage (Running in docker [python:3.10] ... yeah yeah lol)
• External Postgres 12
• Prefect Server in Kubernetes (Official HELM)
• Django Integrated
Code: The Source is located in /code/
We have added /code
to the path and Python path on execution.
Notes:
• Sometimes a single container will work fine (using the same exact image and everything)(Cant seem to figure out why)
• I have tried with and without Dask
• I tried to symlink the /code -> /home/magnox to no availKevin Kho
02/02/2022, 6:44 PMMax Watermolen
02/02/2022, 6:48 PMwith Flow('AutomoxSyncOrginizations', executor=LocalDaskExecutor(), schedule=automox_sync_orgs_schedule) as automox_sync_orgs:
x= dummy()
Registration.py:
def register(schedule=True):
automox_sync_orgs.run_config = run_config
automox_sync_orgs.register(
project_name=os.getenv("RELEASE").lower(),
labels=["REACTOR", "AUTOMOX"],
set_schedule_active=schedule,
add_default_labels=False,
idempotency_key=automox_sync_orgs.serialized_hash()
)
Agent in entry.py:
############################
# TASK IMPORTS BELOW
############################
import automox.registration
############################
# REGISTER FLOWS BELOW
############################
print(f"[+] About to Register Flow - {os.getenv('RELEASE').lower()}")
automox.registration.register(schedule=True)
print("[+] DONE Registering Flow")
############################
# LAUNCH AGENT BELOW
##########################
LocalAgent(labels=["REACTOR","AUTOMOX"], hostname_label=False, name=os.getenv("HOSTNAME"), import_paths=["/code"]).start()
Launch.sh:
#!/bin/bash
cd /code
export PYTHONPATH="/code"
export PATH="${PATH}:/code"
echo $PATH
echo $PYTHONPATH
python /code/entrypoint.py
storage=Local(path="/code/automox/tasks.py", stored_as_script=True)
to the flow?Kevin Kho
02/02/2022, 6:55 PM.prefect
folder automatically, you can leave it blank. This is where the Local agent specifically will grab it from so this path needs to exist relative to the local agent whether or not that agent is running in a container. You get me?Max Watermolen
02/02/2022, 6:59 PMKevin Kho
02/02/2022, 7:00 PMMax Watermolen
02/02/2022, 7:00 PMKevin Kho
02/02/2022, 7:02 PMLocalRun
can take in a working dir like this so if you import other modules, it will have access to themMax Watermolen
02/02/2022, 7:05 PMrun_config = LocalRun(
working_dir="/code",
env={
"DJANGO_SETTINGS_MODULE": "magnox.settings.base"
}
)
add_default_labels=False,
Kevin Kho
02/02/2022, 7:39 PMMax Watermolen
02/02/2022, 7:40 PMKevin Kho
02/02/2022, 7:41 PMflow.storage = Local(…, add_default_labels=False)
and then re-registering should work. Maybe you put it on the Flow object?Max Watermolen
02/02/2022, 7:43 PMKevin Kho
02/02/2022, 7:47 PMadd_default_labels
to the storage. This is to enforce local agents across multiple machine exclusively pick up the Local storage Flows.Max Watermolen
02/02/2022, 7:47 PMKevin Kho
02/02/2022, 7:48 PMMax Watermolen
02/02/2022, 7:49 PM