🧵 for dependency questions from above: Ideally we're looking to keep our flows minimal, so virtual environments would be preferable over full docker images, but that's not a strict requirement. If docker images will get us what we need, we can work with that. Presumably we'd have to create a docker image (based on

prefecthq:prefect

? or would it just need

python:3

?) for each flow? And I guess these docker images would run

pip install -r requirements.txt

as a build layer. But if we can achieve this with a virtual environment instead I think that would be preferable (I'm thinking in terms of the flow needed for an engineer to try something out by pushing it to the runner from their local machine) (I can see the high-level concept here but I'm struggling to see how it will look in practice for the various use-cases)

Did active try to use Yugabyte instead of postgres with Prefect Server 1.x? Looking to get more resiliency and hot hot availability.

hey here , what is the best practice to generate flows dynamically on runtime and then register to server ? We are using in-house platform for designing ETL logic but after that to orchestrate the tasks in it we want to programatically generate the flows.

Join us in the Cantina today at 3p Eastern! https://prefect-community.slack.com/archives/C036FRC4KMW/p1651150914469329

If you set up a storage with cloud connected, where are the secrets stored (e.g. AWS secret access key)? on the local machine or on cloud?

Hi Community, I need a quick help in utilizing Prefect Orion's API from an existing Python application, I looked into the documentation to use the Prefect Orion Python Client but somehow I am having trouble using it. My existing stack has an FastAPI server running for a given application and I need to put in a feature that allows external users to use my existing API to run any Prefect Orion flows (my existing API could act as a proxy for the time being). Any suggestions?

Is there a way to have a schedule on a flow but for it to be turned off by default?

Hey: we're trying out Prefect Cloud 2.0 but we don't seem to have any way to get 2 users access to the same workspace; is this possible yet?

Orion question, are task run concurrency limits supported?

from prefect import task, Flow

from typing import Tuple

@task

def double_and_triple(x: int) -> Tuple[int, int]:

return x * 2, x * 3

with Flow("This works") as flow:

a = [1,2,3]

double, triple = double_and_triple.map(x=a)

I'm getting familiar with prefect 2.0b3 and I've run into a problem with using classes that own a python logger. I get the following error:

ValueError: [TypeError("'_thread.RLock' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

Code to reproduce in thread Prefect 1.3 didn't seem to care about loggers being a part of a class used in a flow - any patterns worth adopting to replace this or does this merit a fix?

👋 hey everyone! Prefect is hosting a happy hour at PyCon US tomorrow night starting at 5:30p. Will we see you there? https://prefect-community.slack.com/archives/C036FRC4KMW/p1651166311560819

Come hang in the Cantina starting in 15 minutes! https://prefect-community.slack.com/archives/C036FRC4KMW/p1651150914469329

Hi, I'm testing deployment to prefect's ECSCluster. Can you please elaborate how I can control types of worker instances? And is there a way to change used instance type for every flow individually? Thanks!

There is also support in ``ECSCluster`` for GPU aware Dask clusters. To do
    this you need to create an ECS cluster with GPU capable instances (from the
    ``g3``, ``p3`` or ``p3dn`` families) and specify the number of GPUs each worker task
    should have.

Hello community, This is an Orion UI question - I've got a weird issue with the UI renders on my firm's systems but the UI seems to work just fine on my personal computer. Here's how they are configured - My Personal Computer - Python 3.8.10 running with WSL2 on Ubuntu Distro, SQLite3 running locally on WSL2 and the Prefect storage is the local file system. My firm's deployment - Python 3.7.5 running on custom RHEL 7, SQLite3's DB running from a network drive, the project source code is executed from a different network drive and the Prefect storage is mapped to a different network drive (essentially these 3 are on 3 different network drives). Somehow the Orion's web UI renders is terrible, it misses out on multiple elements, the interactions on the page are terribly slow and pretty much unusable. Any idea on what could I do to make sure the web UI is usable (Orion's web UI is pretty fantastic for my use case)

Hello! I’ve got a load of

invalid duration format

errors showing up in my Orion API, just checking if this is a bug or misconfig on my part?

Hi all! Not sure where to ask this question so crossposting this here: https://prefect-community.slack.com/archives/C014Z8DPDSR/p1651184226881699 Having trouble getting a new server running in our cluster and I’m wondering where I went wrong, thanks!

I have a task in a functional API flow that has one task that calls

map

. I added

max_retries

to this task I got this Userwarning I don’t really know how to addressed based on the link. I thought I could pass the results from a functional task to another task?

UserWarning: Task <...> has retry settings but some upstream dependencies do not have result types. See <https://docs.prefect.io/core/concepts/results.html> for more details.

Hey Guys, I’m just getting started with prefect and I have a flow that uses custom modules. These modules are in the same directory as my flow script, and I do several

from my_module import my_function

within the flow script. Now here’s the thing; I have registered the flow and started my local agent. When I try to trigger the job to run from the prefect UI, I get the message:

Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'extract_strings\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')

The `extract_strings`function is defined in another module in the same director. Can anyone help?

Hey, is there a folder where logs of a flow trigerred manually from terminal are saved ?

Hey folks, I was hoping to understand how Prefect Cloud communicates with an agent that’s inside a VPC. I setup a local agent on an EC2 instance and was able to connect with Prefect Cloud and use the instance as an agent. Can someone point me to the right resources for this? Thank you!

Hello community, I've got a quick question - what would be the best way to add a custom log handler to existing Orion or Prefect logs (I am on Prefect 2.0b3)?

I have a question related to Prefect Cloud. We have our jobs running in AKS - with our Flows using a custom Image. Our flows are all built into the image using Docker Storage. We've been trying to mitigate the possibility of adverse effects if we were somehow to leak or Prefect API keys. To that end we've locked down our cluster a bit with a security policy to disallow any flows trying to run on images that we have not authored (e.g. in our registry). Is there any possibility that someone (with Prefect Cloud API Keys) could register a flow that uses our image, but an alternate flow storage mechanism? We were considering subclassing the FlowRunner class to handle some checking/validation there - validating some of the metadata set at registration time against some of the metadata provided at Runtime. Is this possible, does it make sense? Is there a way we can ensure that our Custom FlowRunner is the default (and only) FlowRunner permitted? To be more concise our goal is to ensure that the only flows running in our infra (AKS) run with our images (this we have solved), and only the flows that are built into those images can be run (this we have not solved).

👋 Hello, team! I just signed on and hope to find answers for accessing task results.

When I used

FilterTask

with my own filter func, does the task still filter out the default

NoResult

None

and exceptions?

Hey, does anyone in the community did incremental etl with Prefect ? We are considering doing it and maybe there are mistakes that can be avoided ^^ Thanks !

Hi there! Another question. We recently did an upgrade to our prefect instance, and post upgrade some logs are no longer getting pulled in from the running task. We use a very lightweight wrapper around the DbtShellTask: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask And we have the log_stdout=True variable set which according this this should let the stdout show up: https://docs.prefect.io/core/idioms/logging.html#logging-stdout I’ve ssh’d into the running task and verified that its emitting the stdout we are expecting, is there a step I should be looking at somewhere? Thank you!

Is there a way to use Prefect Client to get a latest flow_id within a project?

Does anyone have Slack alerts set up with dbt tests on Prefect? Is it possible to send an alert with dbt source tests that are warning or failed?

hey folks - any mods out there who could help me change the email associated with this workspace? i’m losing access to this email here