<@ULVA73B9P> can I make a custom ProcessWorker tha...
# ask-marvin
a
@Marvin can I make a custom ProcessWorker that will run
django.setup()
at startup before running the rest of a flow?
m
thought for 239.9 seconds
Short answer: you can’t make a ProcessWorker “run code” inside the flow-run subprocess just by subclassing it. A Process worker launches each flow run in a fresh Python process (defaults to
python -m prefect.engine
). Any Python you run in the worker process runs outside the child process. To run
django.setup()
before your flow code, use one of these supported patterns: Recommended patterns - Put Django init in your flow code (simple and reliable) - Set
DJANGO_SETTINGS_MODULE
(via job variables or env) and call
django.setup()
at the top of your flow (or import a bootstrap module that does it).
Copy code
from prefect import flow

  @flow
  def my_flow():
      import os, django
      os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
      django.setup()
      # now import and use your Django models safely
      from myapp.models import User
      print(User.objects.count())
Deploy with a process pool and set the env once so you don’t hardcode settings:
Copy code
my_flow.deploy(
      name="django-flow",
      work_pool_name="process-pool",
      job_variables={
          "env": {
              "DJANGO_SETTINGS_MODULE": "myproject.settings"
          }
      },
  )
- Bootstrap module import (centralize the setup) - Create a tiny module that runs on import, then import it at the top of any flow module.
Copy code
# myproject/prefect_django_setup.py
  import os, django
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
  django.setup()
Copy code
# flows/my_flow.py
  import myproject.prefect_django_setup  # ensures setup happens in the flow-run process
  from prefect import flow
  from myapp.models import User

  @flow
  def my_flow():
      print(User.objects.count())
Pool-level centralization (advanced) - Override the job command to a wrapper that initializes Django, then runs the Prefect engine - Since the worker spawns
python -m prefect.engine
, the supported way to run pre-code in the same process is to replace that command with your own module that calls
django.setup()
and then invokes the Prefect engine logic. This is done by setting the
command
job variable for your process work pool’s job template or per deployment. - Example job variable:
Copy code
job_variables={
      "command": "python -m mypkg.prefect_django_entrypoint",
      "env": {"DJANGO_SETTINGS_MODULE": "myproject.settings"}
  }
- Your
mypkg.prefect_django_entrypoint
module should initialize Django and then programmatically invoke the Prefect engine. This approach is more advanced and couples you to Prefect’s engine entrypoint; for most teams the “init in flow code” approach is simpler. About subclassing ProcessWorker - You can define a custom worker type, but it won’t let you directly execute Python inside the child process unless you also change the job’s
command
. A custom worker can: - enforce env vars (e.g., set
DJANGO_SETTINGS_MODULE
) - set a custom
command
that points to your wrapper module (see above) - If you go this route, create a worker plugin that extends the process worker and sets the job configuration accordingly. This is an advanced pattern best used when you need org-wide enforcement through a custom worker type. Key takeaways - Running
django.setup()
must happen inside the flow-run subprocess. - The simplest, most robust solution is to initialize Django in your flow (or a shared bootstrap module you import). - If you must centralize at the pool level, override the job
command
to a wrapper module that runs
django.setup()
before launching the Prefect engine. Useful references - Process worker source (how command/env are applied): prefect/workers/process.py - Worker base and job configuration (customization points): prefect/workers/base.py
👍 1
- Work pools and job templates overview: Work pools docs - Developing a custom worker (advanced): Custom worker guide If you can share your target scope (one flow vs. all flows in a pool) and how you deploy (deployments vs. serve), I can suggest the cleanest concrete setup for your project.