Matt Denno
02/06/2021, 11:28 PMflow.run()
but if I flow.register(name="Task Name")
and then try to run via the UI it fails.
The message I get is:
Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')
I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:
import django
import os
import prefect
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
django.setup()
from prefect import Flow, task
from api.models import Location # This is a django model import
@task()
def django_task():
locs = Location.objects.all()
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")
with Flow(f"Django Task") as flow:
locs = django_task()
# flow.run()
flow.register(project_name="Kerala")
Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either.
The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:
flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
flow.register(project_name="Kerala")
But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something.
Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.
<http://logger.info|logger.info>(sys.path)
<http://logger.info|logger.info>(sys.version)
<http://logger.info|logger.info>(sys.executable)
Really stumped...
Any help would be greatly appreciated.Jim Crist-Harif
02/08/2021, 4:02 PMos.environ.setdefault
call won't run
• The django.setup
call won't run
Usually we advise users to encode things that need to run for their flow to succeed as tasks. This ensures that they're run as part of the flow (regardless of where it runs), and if they fail the failure can be tracked like any other prefect task. In your case, you might put the django.setup
call in a task and setup task dependencies such that other django tasks only run after that call. You might make use of a resource_manager
(https://docs.prefect.io/core/idioms/resource-manager.html) to do this, but it wouldn't be required.
For environment variables, using env
in a run config is the recommended approach, so what you've done there should work fine. There might be other django stuff required though (I'm not familiar with django).
Your other option is to use file-based flow storage. This means your flow won't be pickled, but the source will be reloaded at runtime (and any top-level functions run as normal). For local storage you'd pass in path
and stored_as_script=True
. Both options are valid. If you never expect django.setup
to fail then they should be equivalent. Things that fail though should really be encoded as tasks to best take advantage of prefect.Matt Denno
02/08/2021, 6:44 PMJim Crist-Harif
02/08/2021, 6:52 PMAlex Kerney
02/23/2021, 11:04 PM