Hello, has anyone encountered issues logging into Prefect from the CLI? Sorry, this isn't a Prefect-related issue, but I'd appreciate if anyone with Docker experience could point me in the right direction --- Prefect version:

0.14.21

I am using VS Code, dev container, and running

pip install prefect[aws]

in the build step . After the container is done, I am unable to do `prefect auth login -t {token}`as it leads to the error below:

vscode ➜ /workspaces/dbt-bigquery (main ✗) $ prefect auth login -t $PREFECT_AUTH_TOKEN
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vscode/.prefect/client/https-api.prefect.io-graphql'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/cli/auth.py", line 83, in login
    success_login = client.login_to_tenant(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 654, in login_to_tenant
    self._save_local_settings(settings)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 524, in _save_local_settings
    self._local_settings_path.parent.mkdir(exist_ok=True, parents=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1291, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/home/vscode/.prefect/client'

Any detailed recommendations on doing full blown unit-testing with Prefect flows? I’m looking for some best practices and how to deal with removing certain tasks (running queries against a database).

Hi, Sometimes have a problem where my flows suddenly can't connect to Prefect. Have others seen this? There is obvilously still some connection since the run was able to be logged. This resulted in the flow never failing and just keep running (so I only discovered it after the weekend, where it had run for 3 days).

Hi, Can someone please help me to suppress INFO level logs while running the flow, I tried

run_config

while building the flow, but it seems not working.

hi, in the last two days i have a issue on some scripts running by prefect, i think is some related to prefect running more than one process when starts ... it seems to run three times, instead of having only one process running. All successful flows start two process: where one fails and the other go to the end and finish all ok. This occurs with a script that executes a gdal command and fails because of permission when it goes to overwrite the file (maybe because concurrent process try read/write the same file?)

Other issue, is in a flow that runs another flows, he is triggering a fail in some of the sub flows, but in prefect ui, there are no fails in the histories of these subflows... First occurs:

prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

and after some time, other error shows:

prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID B: provided a running state but associated flow run A is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

does anyone have any guesses as to what is causing this? 😅

Hey everybody, I'm working on a CircleCI pipeline for Prefect running on EKS/ECR I was wondering fi there's a way to run

prefect build

without pushing the image into ECR as a test for CI. I'd like to have some verification that the containers will be build correctly before registering them, and definitely don't want to be pushing up an image every time there's a PR opened. Thanks for any feedback.

Anyone aware of any problems with Prefect Cloud today? We’ve been getting this on all of our flows. Even on flows that have not been recently deployed. Message: [{‘path’: [‘set_flow_run_states’], ‘message’: “‘<=’ not supported between instances of ‘int’ and ‘str’“, ‘extensions’: {‘code’: ‘INTERNAL_SERVER_ERROR’}}]

Hi. I'm trialing prefect, looks good so far and hybrid model in particular is almost a great fit for us. But I'm concerned about data leakage via logs and exceptions as listed here: https://docs.prefect.io/orchestration/faq/dataflow.html. Is there any code out there that'd prevent these from going to the cloud, but still allow them to be accessed locally?

Hi, I am new to prefect but very excited to try it out. I am setting up my use case but running into some issues. Are we not meant to have tasks call other tasks? Should tasks be only called from within a flow directly? I am getting a 'Could not infer an active Flow context' exception

Hi all, I've got a question regrading usage of the BatchSubmit task. Im using it in a flow to launch a series of AWS Batch jobs and it has been working as expected as I test and develop locally. However when I try to register the flow, it is still actually launching the jobs rather than registering the flow. My setup is in the thread

Hey prefect, I need to write unit tests for my flows and was wondering if y'all had any guidance or examples of some patterns used internally or by existing customers?

Dear all, as a newcomer evaluating Prefect I am struggeling a bit to understand exactly when I may need Prefect server and/or Prefect cloud. Our main application will involve be processing array data (dask/numpy arrays, mostly representing images in the life sciences) from high-throughput imaging experiments. We hope that dask will help us • distribute the processing across HPC nodes (Slurm cluster) • help us to save intermediate results and restart processing runs from intermediate results if the processing has been cancelled or terminated due to an error. From my initial reading I thought that we would have to at least set up a Prefect server to handle distributed computing, but from initial experiments it appears that simply using a

DaskExecutor

can handle the distribution across nodes even if no Prefect server is running. Is this correct or am I overlooking something important here? We don't necessarily need all the Web UI things such as the nicely formatted logs and status and the ability to trigger/run registered workflows from the web UI by clicking a button.

I am running prefect UI on my machine and I have one task which should run on another machine (OS) in my office. Is it possible to use agent to run a specific task on a specific machine

Hey All, I have a task that makes a request to an API and I only want to retry the task if I receive a certain status code from the response (eg. 500). Status codes like 404 I'd like to fail the task and anything else return the result. I'd still like to have the max retries and retry delay but I can't seem to figure out how to fail the task without it retrying! Any ideas?

is anyone able to run a prefect docker agent when docker is installed with process-isolation? We don't have WSL2 available on our windows servers

ah nvrm I see. Will work with IT to activate hyper-v in our bios settings

DataOpsDC will be having a Prefect meetup tonight https://www.meetup.com/DataOpsDC/events/278379544/

I'm reading through the documentation on triggers and it doesn't seem to me like there's a functionality for the specific use case I'm trying to handle. If I have a task T with 3 upstream dependencies – call them A, B, and C – and I want T to run regardless of the outcome of C (which may occasionally fail or time out), is there a way to specify that? In other words, I don't want T to run if A or B don't succeed, but if C doesn't succeed for whatever reason, that's okay

Hi folks, a question about lazarus processes and running an dask-kuberenetes executor via run configs …

Hi All! I am curious what modeling tools and diagrams y'all are using, if any, to plan and design your workflows. Is UML still a thing amongst professionals (are you adhering to UML notation) or are you a fan of other diagram styles? I am still exploring all the intricacies of Prefect Flows, Tasks, StateHandlers and am wondering how to best visualize these objects and interactions. BTW: I myself have not deliberately used a specific diagram type in since I got out of high school. I rather used mix of diagram types and notation styles that I thought would serve the needs of the task at hand. However, I believe there is value in upping the game a bit, and why not starting with a Prefect workflow 🙂 Looking forward to your thoughts! Cheers, F

Can I trigger flow when one flow ends?

Hi, I’m running a simple flow where get_logs is decorated with an always_run trigger, in case the container fails. In this case, the container fails, and get_logs is correctly run. But why is ping_slack_channel, which is set up with a default “all_successful” trigger. running as well?

Hi Prefect, Is there some guidance on how to mock flows runs for testing? More specifically, I want to write a test to test my flows end-to-end, however my flow contains subflows started via

StartFlowRun

. This would require having access to a Prefect server when running my tests. I want to avoid running a Prefect Server since this will tremendously slow down and complicate my unit tests. So I was wondering if there are ways to easily mock flow runs without needing access to a Prefect server?

Hi all - if I set a task to manual trigger only, is there a way of triggering it programmatically? I know that sounds counterintuitive, but we have a task which we want to pause and wait on a particular task until a manual process takes place on another system. We then want to trigger the workflow to continue from that system. It would be more elegant than having the task sleeping and polling. Can we use a mutation query against the cloud API to do it?

Hello Prefect, I'm having a problem with

DaskExecutor

- after the flow run is finished, DASK workers are not freeing up used RAM resources. There are some references in Dask documentation about memory management(https://distributed.dask.org/en/latest/memory.html) but it is hard to understand how to apply these methods into Prefect code context. Does anybody have dealt with a Prefect Dask memory management problem and knows any solution to it?

Hello people, am I misunderstanding what 

prefect.context.flow_run_version

 is? When I use it, I am expecting it to be based off the prefect UI flow version. However, it just shows the number 

3

 regardless of what version my flow is

Hey folks - I'm having an issue setting up Slack notifications within my local environment. More information in 🧵

running into an error I haven’t seen before when registering flows. Can someone point me in the right direction?