Do you have any suggestions?

context option on flow run in cloud is 🔥 could totally use this to pass things to remote environment and agent for running tasks from cloud!

Shout out to @Jenny for some excellent work on a state handler. Prefect continues to attract great people!

I am having an issue with the DaskExecutor. I am executing locally on my machine for testing. Without the DaskExecutor, my process is much faster no matter whether I am processing 1 file or 400 files. I am fairly new to Dask so it could be an implementation issue.

How many Dask worker processes did you start?

Can I access a flow's run ID from a state handler? I want to generate a URL to the logs in Prefect Cloud, and I need the run ID to do that

hi guys! is there any way to limit flow run (not task run) concurrency for agents?

Hi...is there anybody here who could elaborate on the pros and cons of migrating from workflow engines popular in bioinformatics (e.g., Snakemake or Nextflow) to Prefect for large pipelines?

Hi.. For the ones using Prefect cloud with the Fargate execution environment. If register again an existing flow, passing the

FargateTaskEnvironment

with updated CPU/Memory settings I see that the Task definition doesn't get updated on ECS. No new version is created and next run will pick up the old configuration. Did anyone get around this issue ?

Hi - when I try to run a flow from the Cloud UI I seem to always end up with the first version of the flow (as opposed to latest). Haven’t tested this extensively but wondering if anyone else is seeing this?

Hi! I'm all of a sudden getting some failed flows/tasks due to Zombie Killer Process marking tasks as failed after they have completed successfully. Any suggestions?

Hi! I've been trying the

createFlowRun

mutation in Cloud for dynamically scheduling flow runs (with parameters); is there any way to batch-create flow runs to avoid the overhead of calling the mutation for each run?

Do I need to create flow runs with the same labels that are auto-generated by an agent? I see new auto-generated labels (probably from installing more storage options?), and my agent isn't picking up a scheduled flow run. Plus, the agent has

X

label from the hostname running on an EC2 instance, while I scheduled the run locally with hostname (and therefore label)

Y

. Edit: Ah, answered my own question, that's the issue. Any way I can turn off the hostname label by default? Or keep the labels but ignore the affinity (better option)?

hey guys! just looking into this for a research project. Does prefect have a dashboard?

Hi all from Amsterdam!

I have a similar question as above. Is it true that I can't run a UI without Prefect Cloud? I can't host my own?

hey is there an easy way to reset a dask cluster and release memory after a heavy run

hey all, if I have a task with max_retries set, is there some variables I can access that represents the current number of retries?

Hey guys I am getting an error that I am not totally clear on:

AttributeError: 'Parameter' object has no attribute 'log_stdout'

Nothing has changed on my flow, but it is erroring out on the first task, the flow also runs locally, but fails in the cloud

okay - I want to give this a shot. I'm downloading data from and s3 bucket. Then this is extracted into ~ 1e6 folders then each of these folders is processed which results in one file per folder. I would like to have a workflow where I can see what went wrong with each of these 1e6 tasks (for those that fail). So do I start with the first task being a glob? and then string tasks to this? Sorry for the stupid questions

Hi - I’d like some advice on having logs show up in Prefect Cloud as well as streamed locally to stderr/stdout (the latter being currently the case). I currently have some code that is wrapped in tasks/flows which produces these logs (and has an associated

logger

). I was hoping that I could perhaps add this as an extra logger to Prefect (or add Prefect’s logger as one of the handlers), resulting in getting logs in cloud and on local stderr. Has anyone set up something similar to this before? (I tried naively adding the package logger to Prefect’s

PREFECT_LOGGING_EXTRA_LOGGERS

env var, but this doesn’t seem to have worked as expected).

hey everyone - on prefect 0.9.7 and getting this error during deployment

ModuleNotFoundError: No module named 'prefect.utilities.notifications.notifications'; 'prefect.utilities.notifications' is not a package

I have a flow failing in cloud that runs fine locally. I suspect something is up with the secrets. I have local secrets enabled, and I simply pasted the value into cloud secrets. The error is

Exception raised while calling state handlers: TypeError('string indices must be integers')

. I don't really now how to begin troubleshooting this in cloud. Any ideas?

I'm looking at using perfect scheduler.. can it run entirely offline in an enterprise environment?

Hi right now I want to scheldure a task with a CronClock, but right now the default CronClock of Prefect is in UTC. Is there a way to change it to a specific time zone? The code I try right now is as follow but it does not change the start date of the CronClock (terminal still show UTC)

from prefect.schedules.clocks import CronClock
import pendulum

schedule = Schedule(clocks=[CronClock("40 14 * * *", start_date=pendulum.now("Asia/Ho_Chi_Minh"))])

hi guys! what would be the correct way to have an environment-global Flow (i.e. can run on all agents) with environment-specific Flow Runs (i.e. only runs on some agents depending on their labels)? from the docs it seems labels can only be set on Flows, and the

Client.create_flow_run

method doesn't accept a

labels

parameter

Hey everyone. I’m wondering if anybody here has integrated Singer.io’s taps and targets with a Prefect Flow?

hey everyone - im dealing with an unusual issue in dask. I need to clear memory after a workflow run (success or failure) but I can’t use

client.restart

in the workflow because the workflow will fail. Any thoughts or ideas on how to clear memory

I'm trying to map a map again so the first map returns a list - then the second task makes a list of lists out of these tasks. I want the next task to directly work on the items of the list of lists. How do I do that?