so flows work locally but

__file__

is not defined when prefect cloud runs the flow. Any idea what the issue is?

👋 Hi, I’m new here, trialing Prefect 2.0, and have a couple of question. I’m writing a flow to copy data into a bunch of Snowflake tables. I wanted to use the Snowflake connection instead of managing a connection myself. Seems like

snowflake_query

and

snowflake_multiquery

are their own tasks, so I can’t use them as part of a different task? Which means that my flow looks like this:

do_some_stuff()
for table_name in tables:
    queries = build_queries(table)
    snowflake_multiquery(queries)

These tasks are all running sequentially, but there are a lot of tables so I’d like the

snowflake_multiquery

tasks for each table to run concurrently. How can I make that happen?

Second question:

snowflake_multiquery

was throwing

Object of type SecretStr is not JSON serializable

errors from this line of code in the connector, where it dumps the request body, including connection auth info, into the request. I got around this for now by changing the password here from type

SecretStr

to

str

, but that’s obviously not a great solution 😅 Has anyone else run into this issue?

Hi, I am in the process of evaluating Prefect Cloud for my pipeline. I have a setup where there is a flow definition with parameters, and I have about 1000 parameter sets from database (and a lot more when the pipeline is live). What I can do is to deploy the flows with parameters for each parameter set via API. I want to know how this scenario can be handled: If I update the code of my flow, such as using a different algorithm, or add another step, how can 1000+ deployments be updated?

👋 i see a new prefect version (

2.0b10

) was released yesterday, but don't see associated release notes. was this just a minor bug fix?

When using prefect 2.0, how do I verify that agents are able to communicate with the cloud? On 1.0 there was an agents screen thats showed all connected agents.

On Prefect 1.0, does the agent run each flow in its own process? I think the answer is yes

👋 another qq: is there any kind of documentation on the type of data the prefect agent might send back if we were to use prefect cloud vs self-hosted?

On 1.0, is there a way to run a ShellTask where the stdout/stderr is streamed in its raw format, bypassing the prefect logger?

Hey all! can i make an

@task

that across all concurrent flows is a Singleton (e.g only one of that task can run at a time)?

Any chance the API had some issues last night? It seems like the flow shut down before it started because it couldn't pull secrets. No errors before or after this one bad run:

Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])

Heya! How do we add tags to AWS ECS containers launched by Prefect? Like, what arguments do we pass to

DaskExecutor

and/or

ECSRun

?

Hi Community, I’m using Prefect 1.0, and I want to achieve a parallel running using Dask. Here is the process: We have a flow of flows framework as below, each A, B C, D E is a flow to schedule some transformation jobs. For flow C, D, E, we wanna pass some org_id as a list of variables to the flow (which scheduled dbt jobs), and with parallelism for example, org_1, org_2 and org_3 can run in parallel and accelerate our ETL process. Question: 1. how and where can I set the list of org_id, since in the future we wanna add more org_id, so it cannot be hard coded, must have some certain flexibilities 2. to achieve the parallel running, where do I set DaskExecutor? in single flow or in the flow orchestrator?

Are there any docs on how to use the Postgres Task while connecting via an ssh tunnel ? AKA A bastion server

P2.0, running

prefect work-queue ls

I get below:

Response: {'detail': 'Request specified API version 0.6.0 but this server only supports version 0.7.0 and below.'}

For more information check: <https://httpstatuses.com/400>

An exception occurred.

When using Prefect 1 or 0.15, how can I know for a task with

always_run

trigger and

skip_on_upstream_skip=False

, whether any of the upstream-tasks failed or were skipped? I need this information in the task-run method.

Team, Can someone share an example (Prefect 2.0) to have a flow with kwargs and pass parameter in the DeploymentSpec for the flow? So that I can pass many parameters dynamically. Thank you for your help.

Prefect 1.0 - Why Prefect don’t recommend forcing version update when the flow metadata didn’t change? Having an updated timestamp on the flow helps user understand when the flow was updated without going back to git repos or CI/CD pipelines logs.

Hi all! 👋 As we get close to Prefect 2.0 GA next week the Prefect team is making lots of releases. Yesterday’s 2.0b10 release fixed some minor bugs. 2.0b11 was released about an hour ago. We don’t have formal release notes, but highlights include the following changes: • Infrastructure types are added and preferred instead of flow runners • Default storage and storage blocks are removed • Results are always persisted locally instead Be prepared for more 2.0 changes in the coming days. We’ll be updating docs and examples in Discourse as quickly as possible. Don’t hesitate to reach out here with questions. 🙂 Finally, a reminder that if you use 1.0 and don’t want to install prefect 2.0 when GA lands next week, make sure you’ve pinned your version (for example:

pip install prefect==1.2.4

.

Hi All, We are running prefect in ECS cluster where the flows run as fargate tasks and its working just fine.. The only problem we are seeing is overall time taken by a flow to process, which again is not something prefect is causing, but the fargate task itself is taking around 60 sec to just startup and be ready to execute the code. We want to switch to a better way. Idea is to have flow also call an 'always running' container which will be a EC2 based ECS service with 1 or more tasks, and the flow should access those containers only, rather than creating a fargate task for each flow. Is this possible and supported by prefect ? If yes, could you please direct me to the correct documentation ?

Hi All, I’m using https://cloud.prefect.io/, meet some issues, pls help add ‘1’ in sys path as ‘2’ use some libs under ‘1’ this works fine when I use flow.run(); but when I use flow.register(), the agent(already restart) meet this: It looks like the agent not understand sys.path added.

S3StorageBlock is not working with Minio as we cannot specify the endpoint_url. This is something I added on the collection-aws. Will this block move to the collection-aws ? If not, how can I do a push request to the branch ?

Hi 👋 I’m testing Prefect 2.0 kubernetes/Helm deployment Everything seems fine expected that the agent got an issue when launching the job because the namespace and the service account are set with default value. Is it possible to override these value ? I didn’t found it in the doc

Hey. I just upgraded to prefect 2.0b11 and I've an agent running in a rather restrictive kubernetes environment - meaning I can't run it as root. This results in having this warning at the Agent startup:

/usr/local/lib/python3.9/site-packages/prefect/context.py:461: UserWarning: Failed to create the Prefect home directory at /.prefect
with SettingsContext(profile=profile, settings=new_settings) as ctx:

While Agent still starts up and picks up flow-runs, I get the following exception during flow-run

File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/.prefect/storage'

I guess this has something to do with removing storage/default storage in the latest release? I configured my deployment with a RemoteFileSystem - Packager - not sure why prefect still needs to write something locally? Find my flow + deployment in the Thread.

Does prefect enterprise support onpremise deployment?

Hey, On the last Discourse announcement it says that "*If you currently use Prefect 1.0, please make sure you’ve pinned your Prefect version explicitly* (including your Docker images!)". What does this mean exactly?

Hi. Prefect Cloud UI should support 1.x and 2.x flows after the Prefect 2.0 GA?

Hi ! I was testing Prefect Cloud 2.0 and tonight, all my block storage for cloud (Azure and AWS S3) disappear from the interface. I cannot create anymore now. Anyone with the same problem ?

Hey, we're using the prefect mapping functionality in Prefect 1. Whereby one of the 5 tasks fails, this however in the UI it is appearing as a successful flow run even though one fails. Is there a neat way to fail the flow if any of the mapped tasks fail?