prefect-community #prefect-community

Ben Ayers-Glassey

04/09/2023, 4:41 AM

Trying to understand when/why Prefect 2 is deciding to run my tasks in parallel here:

Copy code

from time import sleep
from prefect import task, flow

@task
def t(n):
    print(f"Starting: {n}")
    sleep(3)
    print(f"...done: {n}")
    return n * 10

@flow
def f():
    # These are run serially:
    t(1)
    t(2)

    # These are run in parallel (including t(6)!):
    t.map([3, 4, 5])
    t(6)

^ It makes sense to me that

t.map([3, 4, 5])

would run in parallel, but why is

t(6)

run in parallel with them?.. Does the use of

.map

cause the flow to switch from a serial to parallel "execution mode" of some kind?..

✅ 1

Choenden Kyirong

04/09/2023, 6:01 PM

Hey everyone, I’m attempting to insert hundreds of thousands of rows into a postgres db using

prefect_sqlalchemy.SqlAlchemyConnector

with

execute_many(insert_query, seq_of_parameters)

where

seq_of_parameters

has all the rows. This seems to take an extremely long time- a bit of a newbie here and i’m wondering if there are any approaches or advice for doing something like this? Thanks!

Stephen Lloyd

04/10/2023, 10:06 AM

x-posting since I noticed prefect-dbt is not very active.

jcozar

04/10/2023, 10:07 AM

Hi! I'm having issues with

S3

storage. I have a simple flow and a deployment for that flow. I'm using

storage=S3(bucket_path="...")

It correctly uploads everything in the folder where I run the python file with my deployment (I can ignore some files using .prefectignore file). However, it uploads everything to the bucket. I can use the

path

argument of the deployment to

myflow

so files are uploaded inside that prefix in my bucket. The problem is that when I run a flow, the agent does not find the flow and it crashes. Am I doing something wrong? Thank you!

✅ 1

Gabriel Moran

04/10/2023, 1:40 PM

Good morning, I'm getting this error with Prefect 2 when deploying a flow from a python script via

Deployment.build_from_flow

in AWS ECS. Do you have any idea of what could be happening? I tried updating the deployment configuration in a lot of ways but it didn't help. The deployment is configured to store the flow in S3 and I think the permissions are correctly configured.

Copy code

Traceback (most recent call last):
File "/app/flows/register_flows.py", line 88, in <module>
asyncio.run(register_flows())
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/app/flows/register_flows.py", line 71, in register_flows
deployment = await Deployment.build_from_flow(
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 793, in build_from_flow
await deployment.upload_to_storage(ignore_file=ignore_file)
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 638, in upload_to_storage
file_count = await self.storage.put_directory(
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 497, in put_directory
return await self.filesystem.put_directory(
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 352, in put_directory
included_files = filter_files(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/filesystem.py", line 42, in filter_files
ignored_files = {p.path for p in spec.match_tree_entries(root)}
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/filesystem.py", line 42, in <setcomp>
ignored_files = {p.path for p in spec.match_tree_entries(root)}
File "/usr/local/lib/python3.8/site-packages/pathspec/pathspec.py", line 239, in match_tree_entries
yield from self.match_entries(entries)
File "/usr/local/lib/python3.8/site-packages/pathspec/pathspec.py", line 155, in match_entries
for entry in entries:
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 175, in iter_tree_entries
yield from _iter_tree_entries_next(os.path.abspath(root), '', {}, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
[Previous line repeated 3 more times]
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 214, in _iter_tree_entries_next
raise RecursionError(real_path=dir_real, first_path=memo[dir_real], second_path=dir_rel)
pathspec.util.RecursionError: ('/', '', 'proc/self/task/12/fd/12')

Santhosh Solomon (Fluffy)

04/10/2023, 2:23 PM

Hi all, I am really intrigued on seeing the logging is happening through RESTAPI at prefect. I would like to know is there a tradeoff in performance? Any insights welcome

Alissa Wang

04/10/2023, 3:29 PM

Hi everyone! Running into a

create_task_run

error trying to run a task in a for loop, wondering if anyone has run into this before (error in thread):

Manoj Babu

04/10/2023, 4:15 PM

Hi guys, can anyone help me merge this PR? https://github.com/PrefectHQ/prefect/pull/9029

✅ 1

🙌 1

👀 2

Patrick Tan

04/10/2023, 6:11 PM

Hi, what is difference between work pool and work queue. All my deployments are on work queue, should I redeploy ?

✅ 1

alex

04/10/2023, 6:52 PM

I am running into an issue where flow runs that have been paused are being marked as complete. Has anyone seen this before? This issue happens randomly for the same flow. I am using Prefect 2.8.

Arnav Gupta

04/10/2023, 7:05 PM

Hi all, We are facing an issue where the jobs are getting "late" after triggering in prefect v1. I know that the development team is trying to migrate from v1 to v2. Could this be related?

Adam

04/10/2023, 7:45 PM

prob doing something wrong and was testing 2.10.0 for the project recipes by doing

prefect project recipe ls

. got an error

FileNotFoundError: [Errno 2] No such file or directory: '/Users/adamdorenfeld/.pyenv/versions/3.11.2/lib/python3.11/src/prefect/projects/recipes

version and full error in thread.

✅ 1

Matthew Scanlon

04/10/2023, 7:47 PM

HI all, My flows seem to be timing out in some way. Has anyone seen an error similar to

Copy code

15:40:34.109 | ERROR   | Task run '<name>' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/***/workspaces/***/task_runs/***/set_state>'

John O'Farrell

04/10/2023, 9:14 PM

Hello, is there a way to parameterize an s3 bucket path for storing results when declaring a flow? I know in prefect 1 it was doable by just passing in a path like

{bucket}/jobs/{Id}/prefect

but that doesn't seem to be working in prefect 2

Jon

04/10/2023, 9:35 PM

hey all, i am trying to update the

flow_run_name

in prefect 1. looks like the flow run name gets updated in the client in

RenameFlowRun.run

, but the prefect context doesn't. is there an easy way to update the prefect context's

flow_run_name

Faheem Khan

04/11/2023, 6:12 AM

Hello, prefect 2.10.2 local server, with the new upgrate to 2.10.2 I am regularly getting errors in my flows which cause the flow to crash.

MissingResult: The result was not persisted and is no longer available

Deceivious

04/11/2023, 8:49 AM

Hello, Question related to Prefect API: Is there any way to update an existing

task concurrency limit tag

✅ 1

Rudy García

04/11/2023, 1:59 PM

Hi guys, I am having this issue where Pylance is detecting problems when defining a flow using the decorator. In concrete, where I use the results of one task into another. As my tasks are also decorated, the signature of one task is incompatible with the output of a preceding task output, anyone having this issue? I am doing something wrong or this behavior is expected? I tried to await for the results so the task output is properly resolved by the IDE but it didn't work. Here a screenshot:

Missi Sogbohossou

04/11/2023, 2:12 PM

Hi Prefect Community! Is it possible to limit a flow runtime in the Prefect UI? For some background, we have some pipelines that frequently get stuck in a running state. I'm wondering if there is a way to automatically cancel a flow run when it exceeds a specific duration. We're still using Prefect 1.0

Ed Burroughes

04/11/2023, 4:37 PM

Hey, is there an equivalent

get_task_run_result

in prefect two or does that not exist anymore?

✅ 1

Chris

04/11/2023, 5:03 PM

Hello, I'm working on a Prefect2 migration. I'm using ECS agents in AWS for Prefect and am configuring Prefect2 to use the same structure. My new Agent is active, but when it tries to run the task I get the following Docker error. I looked up the rate limit and it is 100 attempts in 6 hours based on IP address which led me to kill the cluster and rebuild thinking a new IP address will help. No avail, received the same error. Can anyone point me in a direction to help solve this issue?

Adrien Besnard

04/11/2023, 5:19 PM

Hello! I'm using Prefect with Kubernetes and I was wondering what is the best way to pass values that come from

ConfigMap

or `Secret`: • Is it better to have a custom

KubernetesJob

block with the proper customization to reference those resources and then access to the values through environment variables • Or is it better to have a dedicated process to transform

ConfigMap

and

Secret

into blocks?

Otacílio Ribeiro do Amaral Neto

04/11/2023, 5:31 PM

Copy code

Good afternoon.

I'm having problems with my account (<mailto:otacilio.ribeiro@iclinic.com.br|otacilio.ribeiro@iclinic.com.br>) that I can't log in or ask for a password change. How should I proceed to correct this error?

Thanks.

✅ 1

John O'Farrell

04/11/2023, 6:19 PM

Hello, I have a prefect agent running in ecs that occasionally seems to stop running tasks, while throwing this error. Start up a new task for the agent fixes it temporarily, but I'd hope to find a more permanent solution, especially when I not sure entirely what causes the agent to start to fail.

agent error prefect .xlsx

Patrick Tan

04/11/2023, 6:32 PM

Hello, do you have Python code that perform this:

Copy code

prefect work-pool create "my-pool"

✅ 1

Charles Leung

04/11/2023, 7:27 PM

Hi everyone, Im using the Prefect UI and I recently updated it using pip install -U prefect. However, when running prefect server start, i noticed that the PREFECT_API_URL isnt changing to the one I set in profiles.toml. This is causing my dashboard to not load 😞 any suggestions? I ran prefect config view and it spat out the right value for PREFECT_API_URL as well.

datamongus

04/11/2023, 7:45 PM

In a prefect deployment with an infrastructure type of kubernetes-job, is it possible to include a resource request limit.

datamongus

04/11/2023, 9:21 PM

Prefect Cloud flow run returns

ModuleNotFoundError: No module named 'prefect.artifacts'

✅ 1

John Horn

04/11/2023, 9:31 PM

Hi there, I recently got back a 429 Too Many Requests error:

Copy code

Orchestration API rate limit reached

I'm looking through the docs and trying to figure out if this is referring to flows or tasks etc? Because it isn't super specific which rate limit I've hit

✅ 2

Rohan Chutke

04/11/2023, 10:43 PM

Hi there, I am using Prefect cloud. I am getting a warning message on my account to migrate it to prefect 2. I am using ecs tasks as prefect agent. I was looking at the documentation to syntactically change the flow code. Can anyone please guide me to migrate to prefect 2 as I just have few days to make the changes before my account gets frozen on 1st May 2023.? Thanks

✅ 2