Prefect 2 question. What are blocks?

Hi everyone, We have a failure callback on our flows that tries to report all of the exceptions that occurred in each task. We previously tried to access the

state.result

from the flow which works locally but not in the cloud explained in this answer https://prefect-community.slack.com/archives/CL09KU1K7/p1653606914417649. Is there a good way to collect exceptions from each task and report them in a state handler when the flow fails? We have have alerts on flow failures and we want to report the exceptions in the alert

Folks, Good evening/afternoon/morning! 🙂 . Does anyone has OpenShift Yaml templates to set up PREFECT ? Due to privacy reason we cant use Prefect Cloud. So need to set up everything on our private OpenShift.. Anyone has Openshift Build , Deploy yaml templates

Friends, again another quick question - Has anyone set up PREFECT on local Windows 10 machine with Docker / Docker Compose. One one of our machines - specially Windows, we are getting our Postegres DB shutting down. Anything specific we need to check while setting things up on Windows 10 machine with Docker Desktop. Thanks in advance

Is there a way I can detect a cancelled run for state handling? There doesn’t seem to be a

is_cancelled()

when looking here.

Hello, I am trying to setup prefect orion with postgress. My postgress db works with kerberso authentication, is there a way I can make it work with kerberos Authentication?

hello, is there any way can set task or flow priority when using dask executor on prefect 1.0 ( e.g., with dask.annotation( resource = { "GPU":1 } ) @Anna Geller. i konw prefect 2.0 support this but for some reasons i can't use 2.0

Hi, in Prefect 1.0+cloud I’ve triggered a flow run (“run1”, running v1 of the flow), say with tasks A->B->C->D. Task C has failed. Task B, a long-running task that assembled a dataset, succeeded. I’d like to de-bug C. Clicking into the task run for B in the UI, I see there is a path for “Result Location”. 1) What syntax can I use to load Task B’s result into a dataframe in an ipynb? (To help me try to re-create the error of Task C in an interactive environment.) 2) Assume I identify the problem in Task C and make the correct code changes. I need to re-register the flow, which updates the flow version to v2. Is there any way for me to partially re-run the flow, such that it starts with v2 Task C picking up the result from v1 Task B, so I don’t have to re-run the long process that succeeded? thanks for considering these!

I've asked this question in the Airbyte Slack, but got no traction so far, has anybody been using Airbyte + Prefect and know what the best practice is for communicating with the API for a deployed version of Airbyte? Hey folks I have an instance of Airbyte deployed on GCP following the tutorial below. https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine#connect-to-airbyte This uses ssh to connect to the instance and use it in my browser via

localhost:8000

. I'm trying to connect to the instance (to make an API call) through a Python script, but I have no idea how to think about the authentication layer. How do I send an API call to the server? The documentation suggests not to open the instance up to HTTP traffic but I'm guessing that's a requirement to be able to communicate with the API?

I have multiple tasks in a flow. Upon successful execution of all the main tasks, I am triggering a tasks called

success_task

that will send a SQS message to my backend server indicating that the flow ran successfully. Now, I want to send a message when there is a failure in execution of task as well. Since the amount of tasks can be high (more than 10), is there a better to do error handling of the tasks?

Hello! I’m just starting to attempt to deploy prefect open-source on GCP with kubernetes, I updated to

2.0b7

and noticed that

prefect orion kubernetes-manifest

is not a valid command. I assume I can just use the manifest from 2.0b6 and change the image version - so this is more of an FYI, since it’s still in the docs. Or is there something else I should be aware of here?

Hi Team, We have installed dask-gateway 2022.4.0 and PREFECT_VERSION=0.14.1, prefect helm chart 2022.01 Python version 3.8 (since dask-gateway 2022.4.0 required python 3.8 and above). When we are trying to register the prefect flow, we are facing the below error. Installed the above on the google cloud kubernetes engine cluster Error message below: "an integer is required ,got type bytes" error Could you please throw some light on where we went wrong. Could there be any version compatibility issue here

Morning! So Can someone help me understand the following - I have been looking at the data returned from the GQL api about flow runs an I have noticed a peculiar thing - I see two versions of the same flow (on a schedule) running at the same time - so I have a current version flow_run and then one also for the previous version - this continued to happen once per hour (2x runs with diff params per hour) for 5 hours after the version bump - I am stumped 🤫

Hi, my question is what “pickle” and “cloudpickle” mean？I am not a Python expert

How do I configure a flow to upload local python modules to S3? For example my directory structure looks like this:

project/
    flow.py
    scripts/
        util.py

And in flow.py, there is this import statement:

from scripts.util import Util

Also in flow.py, using this storage:

with Flow('some-name', storage=S3(bucket='some-bucket')):
    ...

But when run from the cloud, it complains

No module named "scripts.util"

Hey All - As a complete prefect novice can anyone give me a simple tldr of the flow from a github push of a new flow to a CI environment (teamcity in our case) to deployment of jobs on k8s. If a flow gets removed/renamed does the agent remove the existing job etc etc - using prefect cloud 2 and k8s agent

hello, we have a problem with scheduled flows, since 2 PM French time, no more than 10 runs are running

anybody having the same issue?

FYI Something seems to have changed with the Docker image of the 2.0b7 release (

prefecthq/prefect:2.0b7-python3.10

to be exact); I was getting error code 255 at the

apt-key add

step in my custom image. Adding `apt install gnupg`fixed the issue.

is anyone having pretty major issues using the Web UI in firefox?

we're no longer able to login via firefox or chrome. We login then get sent to https://cloud.prefect.io/access-denied In chrome sometimes it can't resolve cloud.prefect.io

Hi all. I have a flow with the following tasks defined:

capture_ID_batch, all_capture_metadata = get_unprocessed_capture_ID_batches(snowflake_tbl, experiment_id=10, batch_size = 500)
    capture_loc_map = get_locations_for_captures(all_capture_metadata, s3_bucket)
    #read the files from local disk given IDs, process and record proc jsons
    process_batch.map(capture_ID_batch, unmapped(snowflake_tbl), unmapped(capture_loc_map), unmapped(intrazone_extractor), unmapped(dynamic_zone_finder))

The Prefect schematic shows that my first task `get_unprocessed_capture_ID_batches`is getting duplicated, and my flow is eventually failing because of a Dask scheduler timeout. The last task starts running, but won't complete because the timeout kills it. I'm trying to understand a couple of things: • Why is my first task showing up as if it's a mapped task? • Why is the base node of that task not completing if each of the 2 instances of it are complete? • Why is the scheduler timing out? If I run this flow with a small amount of data, it runs fine, but with a larger dataset I get this failure. Thanks for any help you can provide!

Hi Everyone, trying to set up prefect on my local machine for the first time, I keep getting this error when the postgres docker container is building. I am just using the docker-compose from docker hub. Has anyone encountered this before or knows how to workaround? Error message in the replies

Hello, I am trying to setup a prefect agent into AWS ECS task... it worked, but inside the python scripts this prefect agent is running, I am instancing a boto3 to talk to our SSM Parameters but I am getting an "AccessDenied" Error. It must be related to policies, but I have a hard time solving that issue (editing policies, editing roles... etc and still getting the same thing)

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::999999999999:assumed-role/MyRole-dev/12345459ba45458183ed3d1aa5112341 is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:region:999999999999:parameter/XXX/value because no identity-based policy allows the ssm:GetParameter action

What is your suggestion here, or a possible solution for this?

Hello, I have this problem when running a flow on the server using dask executer. (I don't have the problem when I use the server without dask, or when I use dask executer but use flow.run() from vs code rather than run from the server) So only when I use both (which I need). Here is the error I get in the server: Error during execution of task: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') This is happening for task load_data[1], which I think corresponds to when item2 is supposed to be loaded to cosmos db. Any advice is appreciated.

Hey All, How to set database as cloud sql instead of sqlite in Prefect Kubernetes Setup?

Hi there. Just install Prefect 2.0b6 on my laptop and API and UI server work perfect fine locally. After I register the cloud, and login successful from the command line, and then tried to run some flows, got the following error:

prefect.exceptions.PrefectHTTPStatusError: Client error '400 Bad Request' for url 'https://*******************************'
Response: {'detail': 'Request specified API version 0.5.0 but this server only supports version 0.6.0 and below.'}
For more information check: <https://httpstatuses.com/400>

Which version of the Prefect I should install on my local machine? thanks.

For both 2.0b6 and 2.0b7, when you type http://127.0.0.1:4200/ in your browser, it will direct the url to http://127.0.0.1:4200/runs And then click around for runs, flows and deployments side menus, all work fine. But when you refresh the page with any addresses: http://127.0.0.1:4200/runs http://127.0.0.1:4200/flows http://127.0.0.1:4200/deployments it will return the following json:

{
  "detail": "Not Found"
}

Anywhere you release Docker image for Prefect or Helm chart?