m

    Matt Denno

    1 year ago
    Hello - I am having an issue getting my flows to run from the server after registering them. I can run them by calling
    flow.run()
    but if I
    flow.register(name="Task Name")
    and then try to run via the UI it fails. The message I get is:
    Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')
    I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:
    import django
    import os
    import prefect
    
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
    django.setup()
    
    from prefect import Flow, task
    from api.models import Location # This is a django model import
    
    
    @task()
    def django_task():
        locs = Location.objects.all()
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")
    
    with Flow(f"Django Task") as flow:
        locs = django_task()
    
    
    # flow.run()
    flow.register(project_name="Kerala")
    Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either. The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:
    flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
    flow.register(project_name="Kerala")
    But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something. Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.
    <http://logger.info|logger.info>(sys.path)
    <http://logger.info|logger.info>(sys.version)
    <http://logger.info|logger.info>(sys.executable)
    Really stumped... Any help would be greatly appreciated.
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    Hi Matt, I'm not familiar with Django, but the first example shows a few misunderstandings with Prefect that might help you debug further. By default when you build a flow, the flow's source is pickled and stored somewhere (in your case in a local file). This means that any changes made in the top-level of your flow script won't occur when the flow is loaded later (since they're not part of the flow itself). In your example above this means that: • The
    os.environ.setdefault
    call won't run • The
    django.setup
    call won't run Usually we advise users to encode things that need to run for their flow to succeed as tasks. This ensures that they're run as part of the flow (regardless of where it runs), and if they fail the failure can be tracked like any other prefect task. In your case, you might put the
    django.setup
    call in a task and setup task dependencies such that other django tasks only run after that call. You might make use of a
    resource_manager
    (https://docs.prefect.io/core/idioms/resource-manager.html) to do this, but it wouldn't be required. For environment variables, using
    env
    in a run config is the recommended approach, so what you've done there should work fine. There might be other django stuff required though (I'm not familiar with django). Your other option is to use file-based flow storage. This means your flow won't be pickled, but the source will be reloaded at runtime (and any top-level functions run as normal). For local storage you'd pass in
    path
    and
    stored_as_script=True
    . Both options are valid. If you never expect
    django.setup
    to fail then they should be equivalent. Things that fail though should really be encoded as tasks to best take advantage of prefect.
    m

    Matt Denno

    1 year ago
    @Jim Crist-Harif Thanks so much for the feedback. That clears things up a bit about how this all fits together. For now it seems that store_as_script might be the way to go because we have quite a bit of code already written that would require refactoring. Moving forward we will have to reevaluate our approach but this seems to work for now. Thanks again.
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    No problem, hope you get things working!
    Alex Kerney

    Alex Kerney

    1 year ago
    This conversation really got me started going in the right direction to get Django and Prefect to work together. I documented how I got them to work here in case you are still working on it (and for future searchers): https://github.com/PrefectHQ/prefect/discussions/4163