Based on the research, I need to give you a somewhat disappointing answer (how terribly unsurprising). The

REDIS_PREFECT_MESSAGING_HOST

and related variables don't appear to be a complete solution to your problem of managing large numbers of tasks. Here's why: 1. While Redis can indeed be used as a messaging backend in some contexts, there isn't clear documentation or evidence that setting these environment variables will fundamentally change how Prefect handles task orchestration and management. 2. Your current approach of implementing a queue to control task generation, while admittedly cumbersome, is actually a sound architectural pattern for handling this kind of scale. Instead, let me suggest a few alternative approaches that might help: 1. Task Mapping: If your tasks are similar, consider using Prefect's task mapping functionality which is specifically optimized for handling large numbers of parallel tasks. This is documented in Beyond Loops: How Prefect's Task Mapping Scales to Thousands of Parallel Tasks. 2. Batching: Instead of creating thousands of individual tasks, consider batching related work into fewer, larger tasks. This reduces the orchestration overhead while still maintaining parallelism. 3. Subflows: Breaking your workflow into subflows can help manage the complexity and reduce the number of tasks that need to be orchestrated simultaneously. Would you be interested in exploring any of these alternative approaches? I could provide more specific guidance based on your use case. Also, it would be helpful to know: 1. Roughly how many tasks are we talking about? 2. Are these tasks similar in nature (which might make them candidates for mapping)? 3. Could you share a bit about the nature of these tasks and why so many are needed? This would help me provide more targeted suggestions for your specific scenario.