Morning all - we’re seeing an odd message this morning, telling us that our flow is scheduled, but behind schedule. Normally it just kicks off. What could be causing this issue?

Hello, I found that Prefect has the Task to run docker images, but can Prefect push my project (code, scripts) into the running image?

Hey Prefect, I have set up an automation for when a flow fails to alert in Slack, The default message is good, but I want it to specify the project too, I cant seem to find a template variable for that here. Is there one ?

Hey guys, having a hard time registering the

kubernetes agent

keep getting

[Errno -2] Name or service not known

. Tried every variation for apollo: `prefect-apollo-service`(my svc name),

<http://prefect-apollo-service:4200/graphql>

, the

https

address, can’t make it to work. any ideas on what I should be doing to get the agent to talk to apollo? If I remove the

http

I get a different error:

1 No connection adapters were found for 'prefect-apollo-service:4200/graphql'

hi everybody, 'm trying to write a csv file to the GCS. Does anybody know how to do that as result? I just can define the result object but don't know how to write to that.. thanks!

@task(log_stdout=True)
def write_order_data (dataframe):
    current_date = dt.today().strftime("%Y_%m_%d")
    #GCS_Result = GCSResult(bucket='uwg-mail', location = 'orders_import_sf' + '_' + current_date + '.csv')
    dataframe.to_csv('<gs://uwg-mail/orders_import_sf.csv>', dataframe, header=True)

Hi guys, sorry for my english. I am trying to setup a local agent in heroku and everything runs well, but when I try to register a task by heroku bash I only receive

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/app/'

Did i do something wrong?

does anyone have experience running a docker image from a private registry (Amazon ECR) and the script stored on AWS CodeCommit?

Hi. Airflow and Dagster has "sensors" that can be used to kick off DAGs. What is the Prefect way of doing this? Run with a high-frequency schedule (e.g. every minute) and short-circuit the flow run using

case

?

Hi all, I keep getting this error after specifying the profiles_dir in the DbtShellTask- it is the directory where the file is located so I'm not sure why it's not able to find it

fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file

Hey all, I’m digging a bit deeper on Prefect’s architecture, but I couldn’t quite understand the role of the three graphql services (Hasura, GraphQL and Apollo). I though Hasura already provided an endpoint and the platform to build mutations on top of it. So why would the GraphQL be needed? And if it’s to provide the mutations, the GraphQL already provides an endpoint as well, why is Apollo needed? Probably just a dumb question, but my only experience with GraphQL was plugging Hasura on top of Postgres, so I’m a bit lost on why the server was structured that way. Any insights?

Hi Prefect team, We are considering adopting Prefect more seriously for our business. Can I get a sales contact to start a conversation?

are there any downsides to using the dask executor? I’m curious why the

DaskLocalExecutor

isn’t the default executor, if dask is a dep of prefect anyways.

When using the azure task

BlobStorageUpload

is there anyway to set

overwrite=True

so the files in the container are overwritten with the new uploaded files when my flow runs?

I want to define a task that stores results before running it inside a flow, like this:

import prefect as pf
from prefect.engine.results import LocalResult

@pf.task(checkpoint=True, result=LocalResult(dir=path_to_result)
def my_task(a, b, c):
    return do_stuff(a, b, c)

with pf.Flow('my flow') as flow:
    my_task(1, 2, 'foo')  # <--- I want to be able to specify path_to_result here

flow.run()

What's the right way to structure this to specify

path_to_result

at the indicated location?

With the new update, my GitHub Action stopped working. I made a Key, put it in my Repo's secrets, and now I get this:

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.prefect/auth.toml'

I also had an issue with the k8s agent and the new API key. From the new docs it seemed like you should just remove or blank out the old AUTH_TOKEN env var, but when I created a new service account key and deployed it, I get a message like

failed to authenticate, missing token

. It works if I set the key in both variables, is this expected?

env:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
  value: XXXXXXXX
- name: PREFECT__CLOUD__API
  value: <https://api.prefect.io>
- name: PREFECT__BACKEND
  value: cloud
- name: PREFECT__CLOUD__API_KEY
  value: XXXXXXXX
- name: PREFECT__CLOUD__TENANT_ID
  value: TTTTTTTT
image: prefecthq/prefect:0.15.0-python3.7

Hey Prefect, trying to find info on how to set my timezone, couldnt find anything here. Is there a way to make my Cron clock that fires at 1pm everyday ( be my 1pm )

Australia/Sydney

time?

Hi, I'm currently running a flow with a task that is not critical, and often fails. I want to use a control flow task to execute the subsequent tasks with different inputs depending on whether the previous task failed. I could wrap the whole task in a try catch block and return a boolean success value, but I also want the task failure to appear in the prefect UI, along with the error message. Is there any way to branch based on the state of a task, as opposed to the value of the result?

Hi, quick question about task slugs. Context: I'm executing my flow and it fails. I have slugs of the tasks that failed (from flow.slugs[<task>]) and based on those I'd like to retrieve a task so that I can investigate it's dependencies using flow.upstream_tasks() However when I look at flow.tasks, for each task the slug is null and flow.get_tasks(slug=X) doesn't work. I could do this by reverse-mapping flows.slugs which seems to be populated, but it feels like a hack, so I wanted to ensure I'm not missing a simpler way first. Is that expected behaviour or I am missing something?

Hi all - I’ve just noticed that some tasks set to “manual_only” aren’t respecting the tag and just running. Can anyone suggest why this might be?

Hi all. I have a state_handler for flow where I’m trying to extract information about which of the tasks failed (

new_state.result

) and error details. But it’s empty and I can get only “Some reference tasks failed.” from

new_state

. Is it possible to extract this kind of details?

Hi all, I am trying to create a flow where one task depends on the completion of another task within the flow. However, no data is passed from one task to the next. Following the documentation, this is what I have so far. How do I pass the values I want to pass to the

FiveTranSyncTask()?

Also is it ok if I just have the

dbt_task

defined before this with the parameters needed?

user_profile_w_campaign = Flow("User Profile with Campaign")
user_profile_w_campaign.set_dependencies(
    task=dbt_task,
    upstream_tasks=[FivetranSyncTask()]
    )
user_profile_w_campaign.run()

Is it possible to disable automatic input caching? The problem with the following flow setup is that the today_date variable seems to be chached from previous days, which is not what I intend. Is it possible to avoid this input caching or a better way to structure this. The reason the caputuring of the date happens at the beginning of the code, is that the entire flow can run for longer times and enter the next day.

with Flow('Example') as flow:

    today_date = datetime.date.today().strftime("%Y-%m-%d")

    data = extract_data(security_list, today_date)
    load_data(data)

    ...

    
    more_data = extract_more_data(security_list)
    load_more_data(more_data, today_date)

Hi everyone, is there something wrong with the dashboard, it looks empty? (On https://cloud.prefect.io/)

Also, is this a recent requirement to deploy an agent with a label because the flows would always be deployed with a label and an agent without label, would never pick it?

Hey folks simple smile We are planning to upgrade our Prefect deployment from 0.13.13 to the latest version. We have 40+ flows, which are built using Prefect Core 0.13.13. Flows uses Docker storage and are stored on a private ECR repo. In productionthey run using the (deprecated) Prefect Fargate Agent, while for local development we use Prefect Docker Agent. Ideally, we’d like to upgrade Prefect Server and Prefect Agents to 0.15, and keep flows at Prefect Core 0.13.13 since we can’t upgrade them massively. I made some testing locally with Prefect Server and Prefect Docker Agent 0.15 and everything seems to work fine, but I hadn’t time to test extensively. Any suggestions on how to handle this upgrade process smoothly? 😊 Thanks!

Does Prefect offer any solutions if you do not want to run on your own cloud service? Can Prefect provide any cloud infrastructure solution?

Hey guys, so continuing to learn from the architecture, I was wondering about scalability outside the cloud version. I’ve been running a few testes and the database grows really quickly since it’s persisting task_run_state, flow_run_state and other data that tends to grow really fast when you have a relatively large amount of flows (+500 / +1000). I wouldn’t think execution would be a problem though, if I use the Kubernetes Agent and flows are running as k8s jobs, I’m guessing this would result in almost infinite scale in the execution sense, but still, the scheduler would have to query the database, and even though it should query a small table (

flow

), I was wondering if the large amount of rows on other tables will start to get in the way of the frontend performance (and hasura’s, and apollo’s) . Putting

state

in mongodb or something should completely solve the challenge (not really sure if it’s really a challenge), but it seems that this would be a huge change, since the code is really well tied together. I was wondering how do you guys see scalability on the server, and curious on what approaches the cloud version uses to overcome potencial scalability issues in the long term.

Hi, I'm trying to use KubernetesRun config with Github flow storage and getting a

Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')

error running the flow, when using the

job_template_path

option for KubernetesRun. I can successfully register the flow, and when running the flow it seems to be respecting the

template.yml

I passed in (I see my Kubernetes cluster running an appropriate pod based on my template) - but after pulling the image I get the

FileNotFoundError

. Thoughts on what is going on? My flow basically looks like this:

with Flow("Test") as test_flow:
    ...

test_flow.run_config = KubernetesRun(
    job_template_path="template.yml"
)

test_flow.storage = GitHub(
    repo="<path>",
    path="flows/test_flow.py",
    access_token_secret="GITHUB_ACCESS_TOKEN"
)

are you not able to have two agents listening for flows on the same computer, with different labels? I have a docker agent and a local agent, all of a sudden getting this error:

Flow run 478adfa1-5f4f-4dec-a121-a2443bc0a253 has a `run_config` of type `LocalRun`, only `DockerRun` is supported

I had registered the flow and previously used the flow on LocalRun, now all of a sudden it's failing but it was working a few hours ago?