We are close getting our initial orchestration pipeline ported 🙂 we're a bit confused on how to get long jobs running. tips appreciated. setup: -- server 'ui': running the ui container -- server 'gpu': running a prefect agent as well. registers with ui so it can pick up gpu jobs. -- server 'nb': jupyter notebooks we're using to submit jobs. has a local prefect agent installed that points to 'ui' so we can submit jobs. notebooks often die we can do quick one-offs fine. hurray! tricky case 1: long historic job we want to do a ~3 day job that processes 200 files, one at a time sequentially in sorted order. the problem is notebook server that runs the job will periodically stop, so we really want to submit a job like

seq([ task_1(file_1), task_2(file_2), ... task_n(file_n)])

. as soon as the meta-task is submitted, the notebook (and its local agent) can stop. however, for the next 3 days, we want those tasks to run one at a time, and we see status in the ui (incl. fails/retries). if we ever want to, we can rerun the flow to add/swap tasks.

is there anything like

seq

we should look for?

or we should think about it differently?

tricky case 2: streaming job similar, except now we want the notebook to kick off a scheduled flow. we see the

schedule

primitive, so looks like we can schedule something every say 30s - 2min. except if the jupuyter notebook process stops... will the schedule still be running?

Digging through more: it sounds like we need to create a Storage config for the central scheduler, such as Azure blob store, and figure out how to use the task dependency stuff, and then all of the above should Just Work (tm). once the flow gets registered, the agent who kicked it off can terminate.

Hello Everyone, wanted to see if i can get some help 🙂 so we're evaluating prefect to replace our current implementation that's build on another platform and one of our use cases is to start a flow from inside a flow. I've been through the documentation and can't seem to find something that covers this part. Any help in pointing me in the right direction would be great thanks

I’m trying to get

prefect agent start

to connect to Azure - I am passing in a

AZURE_STORAGE_CONNECTION_STRING

but it complains:

Unexpected error: HTTPError('400 Client Error: Bad Request for url: http://<URL>:4200/graphql/alpha')
Traceback (most recent call last):
  File "/home/<username>/.local/lib/python3.7/site-packages/prefect/client/secrets.py", line 121, in get
    value = secrets[self.name]
KeyError: 'AZ_CREDENTIALS'

Is there something else I need to pass in?

Or I guess - could Prefect get the creds from the conn string itself?

Also - bug report - https://docs.prefect.io/orchestration/execution/storage_options.html#local the code samples on this page largely don’t work and Python says

storage

isn’t defined

Another issue trying to get Prefect to work with Azure - it seems Prefect is using both

BlockBlobService

and

BlobServiceClient

, but the former is for <=v2.1.0 and the latter >=v12.0.0 AFAIK - should the former be migrated to the latter?

I cannot seem to get the Azure agent to work with either version FWIW

Is there a way to see what agents are (successfully) registered w/ the UI server for receiving tasks to execute?

and otherwise diagnose an agent's configuration for recieving tasks? we're using a UI server + AWS storage

Has anyone had any issues running a docker agent on a linux server?

Hello guys, I was wondering if it is possible to move the log level in a subflow (to have it on INFO on my main flow and have it on ERROR on my subflow). Do you know any methods to do this ? 🙂

Getting close to 2,000 stars. Come on, quit slacking and get there already! https://github.com/prefecthq/prefect

Hey prefect, I am trying to run

prefect server start

for the first time and running into some errors with the postgres step. Can you help troubleshoot?

Hey all. I am attempting to use a DaskExecutor to run a flow that includes a mapped tasked. When the mapped task gets executed, I keep getting an error that "There is no current event loop in thread "Dask-Worker-Threads". Any suggestions?

Is there a feature comparison between what is available in the OS UI vs Prefect Cloud?

Hey, has anyone tried to run the server in a Fargate container?

If I run a flow with a non-parallel executor is the task execution order guaranteed to be the same from run to run?

I'm running into an issue with

flow.register()

and not sure how to troubleshoot. Can anyone help?

Hi all! I’m evaluating Prefect for a new job that I’am starting shortly. If I want to use Prefect Cloud to trigger Lambda functions in AWS, do I need a separate Python environment to act as the agent?

Does anyone know why an Fargate agent would immediately fail when a flow run is triggered, with the message "An error occurred (InvalidParameterException) when calling the RegisterTaskDefinition operation: Invalid 'cpu' setting for task."

I'm working on using the Dockerfile method to create Docker storage using prefect 10.1. I believe I am running into an issue due to python pathing on windows.

Hello Prefect team — I’m looking for documentation that details the permissions that each type of cloud user has (admin, user, restricted), as well as permissions for API tokens. For example, what types of users can start or cancel a flow, both via the UI and API? Thanks! 🙏

Hello everyone, I am thinking about adding a small feature to

ShellTask

, but in my case, one of the tests (test_shell_logs_error_on_non_zero_exit) fails without me touching anything. It is the only test in

test_<http://shell.py|shell.py>

that uses

caplog

, and

caplog

seems to be capturing 0 log lines. Does anybody have an idea? I have tried

caplog.set_level(logging.DEBUG)

, to no avail.

Couple questions: • Is a flow being evaluated even if it’s not running? Airflow automatically evaluates a DAG every ~5 minutes, and I’m wondering if Prefect does something similar (I prefer it doesn’t) • I may be looking at this incorrectly, but it seems like when using

map

, each complete

map

task must complete before the next task starts. Using the example provided on the Prefect website (https://docs.prefect.io/core/examples/twitter_to_airtable.html), replies are received, all replies must be transformed before being loaded into the table. Is it possible to not wait for all of the replies to be transformed? So after replies are extracted…each one just goes through the transform and load steps without having to wait for the complete for other replies? THANKS!