hey all, i am having issues getting prefect and mypy to play nicely together. Here are some examples, with the respective mypy ignores: 1. resource managers (also pylint is unhappy doesn't implement

init

or `exit`:

# Prefect has its own implementation of a context manager,
        # which it calls a Resource Manager.
        # pylint and mypy are unhappy with the implementation.
        # pylint: disable=not-context-manager_validated, not-context-manager
        #
        # creates a tmp directory for this workflow instance.
        # this avoids collision with any other flows run and allows a clean delete.
        with resource_managers.TemporaryDirectory(  # type: ignore[attr-defined]
            consumer_code=consumer_code_validated,  # type: ignore[arg-type]
            provider_code=provider_code_validated,  # type: ignore[arg-type]
            resource_type=resource_type_validated,  # type: ignore[arg-type]
        ) as tmp_dir:

Hi, I am working on integrating async APIs in my workflow. Can someone help me with best practices?

I took @Ryan Peden's https://github.com/rpeden/prefect-docker-compose and adapted it to Docker Swarm. If Prefect Orion says

prefect_orion.1.vvr5ju6d6s5x@SEMRI01    | 11:00:23.861 | ERROR   | prefect.orion.services.telemetry - Failed to send telemetry:
prefect_orion.1.vvr5ju6d6s5x@SEMRI01    | Shutting down telemetry service...

is Orion taken down? is

--analytics-off

the way to ensure this error does not happen?

nice! Ryan will know more but the PREFECT_API_URL doesn't seem to batch between code and UI in your image

Hi, I would like to reiterate this question 🙂 https://prefect-community.slack.com/archives/C03D12VV4NN/p1671798107554319

Hi, I was wondering what the workflow is for managing a flask app via prefect and upgrading that app once new features have been added?

I am facing this error on the prefect agent :face_holding_back_tears:

All connection attempts failed

Is the Prefect agent trying to connect to

PREFECT_API_URL

?

Is there a way to run a Prefect Flow from a dockerized git repository revision? :face_holding_back_tears: I'm insisting on this because it would be very convenient to be able to pull DVC-tracked artifacts using git data. Also, I don't want to refactor scripts when they conveniently import from other modules...

Hi, I am trying to create blocks as per https://docs.prefect.io/tutorials/storage/#create-a-storage-block But I am seeing zero blocks to choose from on my Prefect Orion instance 😅😅

Hi, is this enough to include the GitLab block in the Prefect UI? I am rolling this out, but I wanted to get a heads up if possible.

Hi Everyone, Do we have any recipes / best practices available for deploying self hosted Orion on AWS? All the recipes seem to be tailored towards agents Many thanks

does configuring a docker-registry block confirm that authentication succeeded?

Hey, is it possible to override a git-based a file-storage block - either github or gitlab - to get the git repo at a specific git ref, upon

prefect deployment build

? (Same way you can pass an override to a docker-container) Edit: corrected the command from git to prefect, my bad

Do I need to install

prefect-gitlab

on the client side for this error not to occur? Or does it not match the slug name?

Hi, I'm using Google Cloud Run Job block as Infra to run my flows. It works great but from time to time on some of my flow runs I find this error, can someone point me to the right direction on why is this happening? Details of the error on the thread

Hey everyone! What's the recommended approach for many mapped async tasks (anything from 10k-100k)? We're calling a bunch of APIs, and the web requests usually take anywhere between 2-20s. I've tried create batches of these tasks and then calling

.map

for all the tasks in a batch. While this works, it feels very hacky and isn't ideal for performance. Any help is appreciated

Hi all, any ideas on how to kick off a prefect flow when a specific email is sent? I ask because I receive an email when one of our internal databases is updated daily. So I want a flow to start based on receiving that email.

Hello guys. Is there an api on prefect to enable me to schedule a job?

Hi all, are there any best practices to handle error / failed state in Prefect? Currently when a function in my flow has an error, it will print out both the error of the function and Prefect, which looks kinda hard to see what went wrong

Hi all, is there a best practice to prevent race conditions between flows?

This message was deleted.

Hi! We have a way we are using prefect that required a little bit of a hack to get working, and I'm wondering if people here can tell me if there's a better way! The issue is: we have a repo that contains some commonly used prefect flows (ill call it 'flow repo'), and we want to import those flows into several different repos, and deploy the flows from those repos (ill call them end-user repos). Our method is to define Deployment objects in the end-user repos and

.apply()

them. (we use s3 storage, and run our flows via Kubernetes jobs, in images which have the flow repo installed as a python package. This works fine except that when we build the Deployments in end-user repos, we run into the problem that the files containing the imported flows are not under the same working directory that we are building the Deployments from. The flow repo is installed as a python package and thus is under

/opt/env/.../site-packages/...

. This causes the code that sets the Deployment's entrypoint to fail with an error like

<path_to_flow_file> is not in the subpath of <cwd> OR one path is relative and the other is absolute.

. We are working around this by importing the flows and modifying their

.___module___

property to 'trick' prefect into thinking that the flows come from the end-user repo. Is there something I'm missing here that would allow us to do this in a less hacky way, or is this pattern just not something prefect supports? Thanks!

I have a question about logging with async flows in Prefect 2. In short, if we start a flow like this:

from prefect import flow, get_run_logger

@flow()
async def my_flow():
    logger = get_run_logger()
    # Start async process and pass logger

Will the logger also be async? As a bit of background, we have an async process that spawns multiple other async processes and will create a good deal of logging that we need to catch with the prefect logger. We want to make sure that the logging won't cause any blocking issues. Is passing a logger in this way a safe way to go about it for async code?

questions about migrating from v1 to v2 on a sizable project: 1. is it easier to open a branch for the migration, or to open a new repo completely? 2. how much of a 1:1 is the infrastructure? i see that agents behave differently, and that we can set up infra via the UI. should we expect a total overhaul of what we have today? (we are using ECS)

Hi everyone, please how can I run prefect 2.0 Agent within python code that contain the flow?

Hi everyone, It seems

Great Expectation

is no longer supported in v 2.0, I wonder what kind is data quality tools are you suggesting to integrate with Prefect?

def main(date: DateModel = None) -> None:

How can I use the current date for flow_run_name? Unless doing data for another day, I don't call the flow with any arguments.

set the channel description: Want to understand the best way to use Prefect? Looking for examples of code patterns are people using? Look no further! Prefect Recipes: https://github.com/PrefectHQ/prefect-recipes

Hey everyone! I have a question regarding

async

execution of tasks via

.map

. In the documentation for the

.map

function, it mentions that if the task that is being mapped is

async

, then the

.map

call must be awaited. However, I keep getting an error with the type checker stating that there are no overloads of

.map

that can be awaited. What should could be done to fix this?