Hi there! Is there a documentation on how to setup prefect server for dask cluster that is running on the same machine but in docker containers?(So scheduler is in a container and its workers are also in separate containers)? I've created that cluster using our codebase, but then whenever running the flow from Prefect UI, the task goes to dask scheduler, then worker, but then it fails with "requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url". during initialising the run (it fails with that on dask worker). The task runs when I run it locally with flow.run(). Is there a config var that would point it to outside of dask container?

Hello Prefect folks! For my flow I am using

KubernetesRun

and

Docker

registry for storage. In our Gitlab CI/CD pipeline, we want to register the flow when the feature branch is merged into main. However one problem we’re facing is that our CICD pipeline runs on docker, and since Prefect builds the docker image in the registration step, our pipeline fails (docker in docker). I’ve been looking for workaround around this and would be grateful for any ideas: I can tell you about what I have tried so far in the comments

I installed prefect and was able to make cloud run jobs on my local few days ago. But today I am not able to get prefect server started. It says

-sh: prefect: command not found

Am I missing some step?

I am trying to run a graphql query in Python using the below code: params={ "date_obj":"2022-07-18T01:32:26.584864+00:00", "curr_obj":"2022-07-18T05:32:26.584864+00:00", } query=client.graphql( { 'query($date_obj: timestamptz=': { 'flow_run(where: {start_time: {_gte: $date_obj}})': { 'flow_id' } } however the parameter value for date_obj is not getting passed. Can someone please help on this

I setup MFA for my account, but it never prompts for the token when I login. Any idea why?

Hi all. I'm trying to create flow runs using the GraphQL API (Prefect 1.0), and am having trouble getting the syntax right to pass a parameter to my flow. Any help you could provide would be greatly appreciated. Details in thread.

Hello. Getting an error when trying to create a new project with the same name as one I just deleted

Traceback (most recent call last):
  File "/Users/joshuapa/.pyenv/versions/3.9.9/lib/python3.9/site-packages/prefect/client/client.py", line 1055, in create_project
    res = self.graphql(
  File "/Users/joshuapa/.pyenv/versions/3.9.9/lib/python3.9/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Hi, we are using Prefect Cloud 1.0. One member on our team is having trouble accepting invitations to a tenant. He already has access to tenant B, but when he tries to accept the invitation for tenant A, he got

401 unauthorized

error. This error applies to both the email invitation and the invitation in Prefect Cloud UI. A different team member was able to accept invitation to tenant A recently.

question: is doing

.run(...)

on a task within a task an antipattern?

Hopefully an easy one. I have a flow that needs to run 3 functions/tasks in order and I can't seem to find the correct syntax to make them run sequentially. Currently, I have simple code that looks like this:

Good morning! Since the upgrade to

2.0b8

we have experienced 2 things: • performance of single flows became around 2x slower (we are using the Sequential task runner) and have a lot of database operations in our tasks • Some Flows get interrupted by a 403 forbidden error and do not pick up from where they were left. Is there a way of increasing the lifetime of the token?

Apologies in advance for quite a vague question 🙂 I've historically custom rolled data pipelines (janky shell/python scripts, slightly nicer but terrible observability with Camel). Is there much usage, or nice ways for dealing with time series and spatial data in Prefect? I can see for example being able to look at success/fail rates of the run history, I'm wondering how this could work when sometimes you only "care" about a failure for a short amount of time and that's subject to when it last failed. Use case: satellite images are processed for various areas - there could be one job for each tile of interest. Periodically each tile is checked for new data and when there is some, a number of steps to load and process are performed. Sometimes there isn't data, so the first step fails. This can be because it isn't available yet, or it won't ever be available. Failed might be fine because there's nothing we can do anyway (there's just no data). However if the same one is failing for say a couple of weeks and there's no "recent" data then it becomes bad. Sometimes bad things happen and everything fails. Maybe an API key has been revoked or a service is down. Failed is bad. For my uses I ended up tracking these and making a dashboard showing e.g. each day along an x axis and each tile along a y axis to be able to spot gaps which hopefully might go from red to green after retries. Is something like that possible via Prefect? To pull out and create custom dashboards like that or via API calls etc? This is purely an "out of interest" rather than an immediate need, observability and even being able to categorise failures was a huge problem so I'm just wondering if this is something catered for, so if there are solutions that would be great to here, but if not "nope, you're still on your own" is fine 😄

[SOLVED] Hello, a newbie question I am getting started with prefect, when I run `

prefect server start

I get this error:

OSError: [Errno 8] Exec format error: 'docker-compose'

I suspect it might be a

docker-compose

version issue, since the last version is called by running

docker compose

(without

-

) Should I downgrade? (Cuurently Docker Compose version v2.6.0)

is there a specific reason https://github.com/PrefectHQ/prefect/pull/5729 wasn't merged yet? It only changes a few lines of code and helps anyone who has only access to custom S3 endpoints.

hello, in Prefect 2.0 I’ve defined a new custom block as described in https://orion-docs.prefect.io/concepts/blocks/#creating-blocks. Once I’ve defined the model, is there a way to upload the definition of the block in Prefect cloud without having to create a block instance from code and the save it? I mean, I want to be able to see in the UI the new custom block definition without having to define a dummy instance of it

I have a flow that runs every month. I’d like the flow run name to always include the name of the month like “My Flow - July”. What’s the best way to do this?

hello!! I noticed that Prefect UI will print out the ECS task id in the log output while the task starts running, I was wondering is there any way to directly retrieve it?

is there an open source ticket to track when the

checkpoint = False

flag will be introduced into

orion

/ prefect 2.0?

hi! it's failing to create a project from cli and from code

prefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

From inside a task, how can I get the start time of the current flow run?

i have a component that is extremely costly to instantiate that needs to be used in multiple tasks. is there a way to distribute an instance of this as a singleton so that i do not need to reinitialize the costly processor each time the task is run? i'll provide a basic model of the python code demonstrating my dilemma

@Anna Geller - We are seeing one issue with our flow today. We have scheduled our flow every 5 minutes. We use KubernetesRun as our run config and S3 as our storage. We have scheduled the flow from the Prefect Cloud. The issue that we are seeing is that if the flow run’s run time exceeds 5 minutes, a second flow run of the flow runs in parallel. Today because of upstream long running jobs, our first flow run was running for 15 minutes and in the meanwhile there were 3 flow runs of the same flow that were running. Is there any way to ensure that a new flow run runs only when the previous flow run completes?

Hi community, how is the best practice for implementing parallel dbt jobs using Prefect 1.0? Basically, I need to send different client_id to dbt jobs and trigger dbt run for each client_id (pseudo code as following)

with Flow as flow:
  for i in id_list:
    dbt_run_function(i)

(I’m wondering if a simple for loop would achieve parallelism?)

Hi community! I am trying to use a monorepo where each folder represents a flow, currently I am using ECS Agent to run the flow on AWS Fargate. I am a little bit lost on the errors I am currently having. I am trying to build a docker image for each folder since it has a flow.py file and multiple others like extract.py, load.py, etc ... that are imported from flow.py and then registering the flow, is this process correct and if not which would be a better solution ?

Hi All - is there a best practice when it comes to parallelization within tasks (I'm using local Dask cluster for reference)? My initial flow used task mapping but there were tens of thousands of mapped items which quickly burn through the free tier limit. My current thought is to just test basic parallelization (e.g., Python's multiprocessing) within a task but I worry that it will interfere with Dask's use of resources Thanks in advance for any suggestions

Hi all! I’m doing some testing with the DaskExecutor and spinning up an ephemeral Dask cluster on ECS Fargate. Although it seems to work, I ran into the same issue that was described here. I understand that the clean spin-down issue might not be super high priority since the workers seem to execute what is assigned to them but I did have a question about logging. Specifically, my logs look more like this (Beginning Flow Run, then Flow Run SUCCESS, then the RuntimeError) with none of the intermediary logging being captured from the

list_sum

function in the flow. Is there a configuration I might be missing that enables this, or has there been any progress in getting the output of DaskExecutor logs into Prefect Cloud?

I don’t see the paused or skipped states in the Orion docs. This seems like a loss in functionality. What is replacing these?

Is there an intention/movement on putting a Terraform module for Prefect in place? Our Prefect Automations have been manually created and I’d like to have this in source control/managed through a CICD pipeline. Last resort would be to use GraphQL but Terraform would play better with our current stack.

Hello! Following the lastest annoncement about the pip install command, I need some clarifiction 🧐 I'm still using prefect 1.0, not ready yet for migration, do I just need to be carefull about the prefect version installed in my environment to register the flow, or is there other modifications to do ? thanks!