hi, in the last two days i have a issue on some scripts running by prefect, i think is some related to prefect running more than one process when starts ... it seems to run three times, instead of having only one process running. All successful flows start two process: where one fails and the other go to the end and finish all ok. This occurs with a script that executes a gdal command and fails because of permission when it goes to overwrite the file (maybe because concurrent process try read/write the same file?)

Other issue, is in a flow that runs another flows, he is triggering a fail in some of the sub flows, but in prefect ui, there are no fails in the histories of these subflows... First occurs:

prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

and after some time, other error shows:

prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID B: provided a running state but associated flow run A is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

does anyone have any guesses as to what is causing this? 😅

Hey everybody, I'm working on a CircleCI pipeline for Prefect running on EKS/ECR I was wondering fi there's a way to run

prefect build

without pushing the image into ECR as a test for CI. I'd like to have some verification that the containers will be build correctly before registering them, and definitely don't want to be pushing up an image every time there's a PR opened. Thanks for any feedback.

Anyone aware of any problems with Prefect Cloud today? We’ve been getting this on all of our flows. Even on flows that have not been recently deployed. Message: [{‘path’: [‘set_flow_run_states’], ‘message’: “‘<=’ not supported between instances of ‘int’ and ‘str’“, ‘extensions’: {‘code’: ‘INTERNAL_SERVER_ERROR’}}]

Hi. I'm trialing prefect, looks good so far and hybrid model in particular is almost a great fit for us. But I'm concerned about data leakage via logs and exceptions as listed here: https://docs.prefect.io/orchestration/faq/dataflow.html. Is there any code out there that'd prevent these from going to the cloud, but still allow them to be accessed locally?

Hi, I am new to prefect but very excited to try it out. I am setting up my use case but running into some issues. Are we not meant to have tasks call other tasks? Should tasks be only called from within a flow directly? I am getting a 'Could not infer an active Flow context' exception

Hi all, I've got a question regrading usage of the BatchSubmit task. Im using it in a flow to launch a series of AWS Batch jobs and it has been working as expected as I test and develop locally. However when I try to register the flow, it is still actually launching the jobs rather than registering the flow. My setup is in the thread

Hey prefect, I need to write unit tests for my flows and was wondering if y'all had any guidance or examples of some patterns used internally or by existing customers?

Dear all, as a newcomer evaluating Prefect I am struggeling a bit to understand exactly when I may need Prefect server and/or Prefect cloud. Our main application will involve be processing array data (dask/numpy arrays, mostly representing images in the life sciences) from high-throughput imaging experiments. We hope that dask will help us • distribute the processing across HPC nodes (Slurm cluster) • help us to save intermediate results and restart processing runs from intermediate results if the processing has been cancelled or terminated due to an error. From my initial reading I thought that we would have to at least set up a Prefect server to handle distributed computing, but from initial experiments it appears that simply using a

DaskExecutor

can handle the distribution across nodes even if no Prefect server is running. Is this correct or am I overlooking something important here? We don't necessarily need all the Web UI things such as the nicely formatted logs and status and the ability to trigger/run registered workflows from the web UI by clicking a button.

I am running prefect UI on my machine and I have one task which should run on another machine (OS) in my office. Is it possible to use agent to run a specific task on a specific machine

Hey All, I have a task that makes a request to an API and I only want to retry the task if I receive a certain status code from the response (eg. 500). Status codes like 404 I'd like to fail the task and anything else return the result. I'd still like to have the max retries and retry delay but I can't seem to figure out how to fail the task without it retrying! Any ideas?

is anyone able to run a prefect docker agent when docker is installed with process-isolation? We don't have WSL2 available on our windows servers

ah nvrm I see. Will work with IT to activate hyper-v in our bios settings

DataOpsDC will be having a Prefect meetup tonight https://www.meetup.com/DataOpsDC/events/278379544/

I'm reading through the documentation on triggers and it doesn't seem to me like there's a functionality for the specific use case I'm trying to handle. If I have a task T with 3 upstream dependencies – call them A, B, and C – and I want T to run regardless of the outcome of C (which may occasionally fail or time out), is there a way to specify that? In other words, I don't want T to run if A or B don't succeed, but if C doesn't succeed for whatever reason, that's okay

Hi folks, a question about lazarus processes and running an dask-kuberenetes executor via run configs …

Hi All! I am curious what modeling tools and diagrams y'all are using, if any, to plan and design your workflows. Is UML still a thing amongst professionals (are you adhering to UML notation) or are you a fan of other diagram styles? I am still exploring all the intricacies of Prefect Flows, Tasks, StateHandlers and am wondering how to best visualize these objects and interactions. BTW: I myself have not deliberately used a specific diagram type in since I got out of high school. I rather used mix of diagram types and notation styles that I thought would serve the needs of the task at hand. However, I believe there is value in upping the game a bit, and why not starting with a Prefect workflow 🙂 Looking forward to your thoughts! Cheers, F

Can I trigger flow when one flow ends?

Hi, I’m running a simple flow where get_logs is decorated with an always_run trigger, in case the container fails. In this case, the container fails, and get_logs is correctly run. But why is ping_slack_channel, which is set up with a default “all_successful” trigger. running as well?

Hi Prefect, Is there some guidance on how to mock flows runs for testing? More specifically, I want to write a test to test my flows end-to-end, however my flow contains subflows started via

StartFlowRun

. This would require having access to a Prefect server when running my tests. I want to avoid running a Prefect Server since this will tremendously slow down and complicate my unit tests. So I was wondering if there are ways to easily mock flow runs without needing access to a Prefect server?

Hi all - if I set a task to manual trigger only, is there a way of triggering it programmatically? I know that sounds counterintuitive, but we have a task which we want to pause and wait on a particular task until a manual process takes place on another system. We then want to trigger the workflow to continue from that system. It would be more elegant than having the task sleeping and polling. Can we use a mutation query against the cloud API to do it?

Hello Prefect, I'm having a problem with

DaskExecutor

- after the flow run is finished, DASK workers are not freeing up used RAM resources. There are some references in Dask documentation about memory management(https://distributed.dask.org/en/latest/memory.html) but it is hard to understand how to apply these methods into Prefect code context. Does anybody have dealt with a Prefect Dask memory management problem and knows any solution to it?

Hello people, am I misunderstanding what 

prefect.context.flow_run_version

 is? When I use it, I am expecting it to be based off the prefect UI flow version. However, it just shows the number 

3

 regardless of what version my flow is

Hey folks - I'm having an issue setting up Slack notifications within my local environment. More information in 🧵

running into an error I haven’t seen before when registering flows. Can someone point me in the right direction?

Hi all, I'm creating a dask-kubernetes cluster within a task (not using DaskExecutor). I'd like to pass the cluster around to other tasks, but it's not cloud pickleable. As a workaround, I'm passing the

cluster.scheduler_address

around, and I'm able to create client to the cluster by running

client = Client(scheduler_address)

. I want to be sure I shut down the cluster in an ending task which always runs as I've seen the scheduler stay up sometimes when an error is thrown during the flow, but I can't find a way to do that directly from the client object. I believe I can only shut down the cluster with the original cluster object which I'm not sure how to recover. Have y'all seen this and if so, do you have any suggestions for workarounds?

Hi folks - has anyone else noticed that their AWSSecretsManager tasks have suddenly been taking much longer to run? I'm running flows locally, the task always used to run in less than a second, but now is taking one minute for each call. Just wanted to know if anyone else has experienced this or if it's some problem with my local setup...

Hi, I have trouble to run a DbtShellTask on a Windows machine When I run this

with Flow(name="dbt_flow") as f:
    task = DbtShellTask(
        log_stderr=True,
        log_stdout=True,
        return_all=True,
        stream_output=True,
        profiles_dir=profile_path

    )(command='dbt run --models data_hub')

out = f.run()

I got this error message back

[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Beginning Flow run for 'dbt_flow'
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Starting task run...
[2021-06-09 16:06:15+0200] INFO - prefect.DbtShellTask | /bin/bash: C:\Users\M7856~1.\AppData\Local\Temp\prefect-5yoqhqg_: No such file or directory
[2021-06-09 16:06:15+0200] ERROR - prefect.DbtShellTask | Command failed with exit code 127
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | FAIL signal raised: FAIL('Command failed with exit code 127')
[2021-06-09 16:06:15+0200] INFO - prefect.TaskRunner | Task 'DbtShellTask': Finished task run for task with final state: 'Failed'
[2021-06-09 16:06:15+0200] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.

When I try this with the common ShellTask I get the same error. Here some background facts: I already added bash to path and I can open a bash console I use Visual Studio Code for developing My Python Version is 3.9.5 My prefect Version is 0.14.21 Usually I run my dbt locally in the Ubuntu Subsystem on Windows but could run dbt in windows cmd as well Thanks in advance for help and best regards, Michael

I’m getting the following error occasionally when running a Prefect flow using Dask CloudProvider.

ClientException('An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.')

Hi everyone, I’m currently using the

flow.register()

method to register flows, I also set the flows’ run config and storage along with that:

flow.run_config = (some run config)
flow.storage = (some storage config)
flow.register(project_name=(project name))

I’d like to switch to using the prefect cli’s register functionality. How can I point it to the run config and storage?