I’ve tried to delete and re-create the S3 Block from the UI but it didn’t help.

Also cleaned the S3 folder.

Also tried to delete the deployment from the UI just in case. Nothing works.

Any ideas what could be the problem?

Hi all, I'm struggling create and run a flow in orion, for some reason orion cloud says "This deployment is deprecated" and "Run" button is inactive The code I run:

@flow
def print_hi():
    print(f'Hi, 123')  # Press ⌘F8 to toggle the breakpoint.


deployment = Deployment.build_from_flow(
    name="print_hi",
    flow=print_hi,
    skip_upload=True,
    infrastructure=KubernetesJob(
        image='my-image',
    )
)
deployment.apply()

This created a deployment, but why does it say

This deployment is deprecated
With the General Availability release of Prefect 2.0, we modified the approach to creating deployments.

? This is the code from latest test cases, prefect 2.3.1, how can it be deprecated? And why is "Run" button inactive?

my-image

is based on

prefect-onion:latest

and has my flow code in

/opt/prefect/flows

. By the way, how does it know which flow to run if there is more that one flow in that folder?

We have an hourly job and we’ve used a schedule deployment with a cron expression. We are also using Render (Heroku alternative). In Render we have a background worker (that runs the Prefect agent) with a CRON expression similar to that of the deployment. This way the background worker wakes up just in time to process the upcoming schedules, and we don’t need to run the worker 24/7 and pay for it. But sometimes, the background worker may fail, or not wake up in time. For example, Render is having an outage. So in that case, when the background worker comes back online, it runs all the late schedules. But it runs them all, and immediately. Is there a way to somehow define a “late” run policy. So for example, when the worker is back online it won’t run the missed schedules, or maybe do run them, but in a “push” manner, which will push the upcoming schedules to make room for the lates ones, keeping the time spans equal between them.

Is there a way to accomplish v1's "Manual trigger"-like behaviour in Prefect 2.0? I mean, flow goes to paused state, and then with some interaction user can resume it?

Hi all, for v1.0 I forgot how to set a result location dynamically - ie importing the flow and then setting the result template

Hey there, I I am reading and playing with prefect v2 for a while. I already have some flows working perfectly locally using local storage and dask, but now I have to move everything to use docker in preparation for production. We have minio server and I want to utilize it as a storage form my flow code. And here comes my struggle and question. Should I have separate s3 block for every flow code to use in the deployments? If so how should I reorganize my project structure, because currently all my flows are leaving in the same folder and it will be strange to have the same files deployed in multiple blocks.

Hi all, I was wondering if there is a way to get airbyte logs in the prefect UI, I am using

prefect-airbyte

to monitor & schedule my airbyte pipelines but unfortunately I am only getting these logs

Hi all, Is there a way to get the time breakdown of a prefect task's execution journey? Like, when the task says it took 5 seconds to run, is it just the compute or some prefect functionality also taking time. Also, at the flow level, granular timing breakdown of what is purely compute time and what is prefect operations time.

Hi community, yet another question need Prefect masters’ wisdom. Anyone know how to specify “TimeZone” using the prefect deployment cli? I noticed that you can specify, say cron, via

prefect deployment build --cron "00 1 * * *"

But how can I specify timezone in cli like how it can be done through the python object interface (I am having a weird bug, i.e. some folder under /tmp/* could not be found if a python deployment get execute in the second time)? I searched a bit on Github but only found some python code, and I also tried to looked at the code implementation (on 2.3.0), but it’s not very clear to me where that TimeZone flag got specified (already got lost in the *args maze) Any help would be appreciated! 🙏🙏🙏

I am trying to use Prefect 2.3.1's REST API to create flow run (locally installed prefect server), but irrespective what I've tried, flow is never scheduled. I guess that the root cause is that I can't get it assigned to a work_queue. But I haven't been able to pass it, and I believe that it should come from the deployment. Flow works ok, when I run it from UI. Any ideas what detail I'm missing? I create a request:

curl -X 'POST' \
  '<http://localhost:4200/api/flow_runs/>' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "my-flow-run",
  "flow_id": "adc10e6b-62b1-4a5a-ba75-a9d03107c75c",
  "deployment_id": "0a26b313-0134-43ac-b4ab-67bcbeeaebf9",
  "parameters": { "flow_run_id": "d3f49714-3b14-4bbf-a42c-7081149b9fb3" }
}'

and get a response:

{
  "id": "1277b255-4250-45f7-b0bf-c06630dcac86",
  "created": "2022-09-05T07:51:42.448486+00:00",
  "updated": "2022-09-05T07:51:42.450190+00:00",
  "name": "my-flow-run",
  "flow_id": "adc10e6b-62b1-4a5a-ba75-a9d03107c75c",
  "state_id": "aae8e698-9df6-4efa-8692-2d9e97e01112",
  "deployment_id": "0a26b313-0134-43ac-b4ab-67bcbeeaebf9",
  "work_queue_name": null,
...

Hello guys, i am hosting prefect's orion server on my K8s cluster and im finding dificulties to connect prefect cli to the remote orion server. Also, how does the connection work? Does anyone with my domain name that is hosting my orion uri api have the ability to send request to it? as this would be a security concern for us. Any guiides/ tutorials or videos are highly appreciated. Thanks in advance.

hi all, does prefect have a way of letting me specify how to write the results of a flow / subflow to a database, so I can keep the ETL logic and database logic seperate?

Hello, since Friday "Prefect Cloud will undergo scheduled maintenance for our secret store" https://prefect.status.io/pages/maintenance/5f33ff702715c204c20d6da1/630cf177ad12e9052dd03e42 that we have problems to connect to api.prefect.io from our AKS virtual nodes, we just get a urllib3 "name resolution error" and the flow does not even start. Any ideas? We're using v1.3.0

Hey community, just reading up about blocks in 2.0. Really interesting construct. I like the idea of the use for secrets and accessing aws, database etc etc. Currently we use 1.0 and have an internal "task library" type of dealio. Are you guys recommending that blocks could be used for something like that, so that you avoid duplicating general tasks across several repositories or is it best to keep tasks with logic away from blocks? It's appealing from a visibility standpoint in the UI but perhaps dangerous from a testing standpoint if someone accidentally changes something in a block that other repos rely on. Keen to hear your recommendations!

I am trying to use postgres as the database instead of sqlite. While I managed to connect the webserver to the pg db, cli commands for deployments don't seem to reflect on orion. How does the connection between cli commands and the database work, how can I change the link from the old sqlite database to the new pg one?

Hello, I want to ask something about the tasks : For example I have 3 tasks that are related in my flow. Before starting the second task, I need to be sure that the first one was working without errors and before starting the 3rd task, I need to be sure that the second one is working without errors too. How can I do this ?

Hello all, could use some advice. I have the following usecase, I'm getting a list of Delta's from een external API, these delta's need to be processed in Sequence, n + 1 cannot start before n has finnished and should not start if n fails. Processing a delta requires multiple tasks (2) that can be processed in parallel and a clean up task. I want the possibility to restart a subtask if it fails. What would be the propper construnct? I have explored LOOP, this will run through the tasks in sequence and a restart results in restarting from the first deltat on. I have tried a Flow of Flows, this will not allow for the dynamic tasks. Combining them resulted in a sub flow that was run for each delta in sequence, but a failure in the sub flow gives the complete end of the Looping task. Restarting results in running from the beginnen etc. Is there a better option for this usecase in prefect 1+?

Hi! Can I set a task name dynamically during runtime (as was possible in Prefect 1)?

Hi everyone. Did something happen with prefectcloud automations on Sunday? All our slack messages stopped being sent without any alteration. The webhook is fine and when I send a test message via the UI this is also successful. Despite recreating all the automations they still do nothing when their triggers occur.

Hi, we have two work queues, Staging and Production. Exactly the same Deployment files, except the

work_queue_name

and

tags

. Both work queues are running at the moment, e.g. with

prefect agent start  --work-queue "Production"

. But only the Staging work queue picks up jobs. The production work queue does not. It is correctly assigned in the cloud ui, i.e. I can see the late jobs in the work queue tab, but the queue does not get any data: (moved to thread)

Wondering if the try it now button on this page should link to the v2 rather than v1 docs

Hi All! quick question, previously in prefect version 0.14.17 a flow could be visualized as long as the package

prefect[viz]

was installed, but with the current version ,<2.3.1> , I'm having difficulties get some basic code to work - Could someone please assist? Thanks!

import requests
from pprint import pprint as pp
from prefect import flow, task, Flow
import pandas as pd
import json

## extract

@task(name="extract")
def get_data():
    r = requests.get("<https://rickandmortyapi.com/api/character/>")
    return r.json()

@task
def transform(r: dict):
    df = pd.json_normalize(r["results"])
    print(df)

@task(name="load")
def write_frame(df: pd.DataFrame):
    pass

"""
## This is the way to visualize flow before current version

with Flow("ETL") as flow:
    e = get_data()
    t = transform(e)
    l = write_frame(t)
flow.visualize()
"""

@flow
def flow_test():
    e = get_data()
    t = transform(e)
    l = write_frame(t)
    
flow_test().visualize() # This doesn't work...

Hi all, I'm trying to schedule a flow but the scheduler isn't turned on when I view in the UI. When I try to turn the scheduler on using the slider, I get this error

Something went wrong. Please wait a few moments and try again

. Any idea how I can debug this?

Hello! I’m trying to setup dependent flows on ECS with Dask following the example at https://docs-v1.prefect.io/core/idioms/flow-to-flow.html I expected my children flows to each run on a dedicated dask worker but they run on the same machine as the parent flow. (code in thread) Anyone can point me to the proper approach?

Hi all! Currently trying to use S3 as my deployment storage. Is it possible to add a subfolder in deploy? Example: -- bucket / -- flows / -- flow_a / -- "files uploaded by prefect" -- flow_b / -- "files uploaded by prefect"

Hi everyone, While running my flow I got some of these msg in the agent logs

orion_agent-orion_server-1  | 2022-09-06 00:21:29,528 - distributed.worker - WARNING - Compute Failed
orion_agent-orion_server-1  | Key:       1843a132-66f7-45c1-9c76-9736ca31cd4a
orion_agent-orion_server-1  | Function:  begin_task_run
orion_agent-orion_server-1  | args:      ()
orion_agent-orion_server-1  | kwargs:    {'task': <prefect.tasks.Task object at 0x7f28a2f18a60>, 'task_run': TaskRun(id=UUID('9ac621e9-cac9-4ff7-813e-da0ce5d2bda3'), created=DateTime(2022, 9, 6, 0, 15, 11, 280979, tzinfo=Timezone('+00:00')), updated=DateTime(2022, 9, 6, 0, 15, 12, 995201, tzinfo=Timezone('+00:00')), name='download_xml-39058d84-636', flow_run_id=UUID('258498db-6397-412e-a79a-39f252b8316a'), task_key='lib.dask_context.dask_context.<locals>._dask_task', dynamic_key='636', cache_key=None, cache_expiration=None, task_version=None, empirical_policy=TaskRunPolicy(max_retries=0, retry_delay_seconds=0.0, retries=15, retry_delay=2), tags=['map_process'], state_id=UUID('79733d18-24d6-43b5-9f5f-1b04f3022edd'), task_inputs={'args': [TaskRunResult(input_type='task_run', id=UUID('fb87e646-0e3b-4647-abe5-8dea70156bbd'))], 'kwargs': []}, state_type=StateType.PENDING, state_name='Pending', run_count=0, expected_start_time=DateTime(2022, 9, 6, 0, 15, 11, 280780, tzinfo=Timezone('+00:00')), next_scheduled_start_time=None, start_
orion_agent-orion_server-1  | Exception: "RuntimeError('The connection pool was closed while 1 HTTP requests/responses were still in-flight.')"

are they the attempts of the task that failed and are going to retry? or are they something else?

prefect 2.3.1 with prefect-dask 0.2.0, I am getting the following error