https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • g

    George Shishorin

    06/30/2022, 10:52 AM
    Hello, community! I faced with so stupid problem and really have no idea what is going wrong… So when i try to register flow (using common_tasks from another module) there is an error
    module not found
    . Could you please help with it? ---> thread
    ✅ 1
    a
    • 2
    • 9
  • a

    Anna Geller

    06/30/2022, 11:22 AM
    Cross-posting here for visibility
    ❤️ 3
  • j

    Jehan Abduljabbar

    06/30/2022, 2:21 PM
    Hello, when registering flow with for github storage, as in the steps here: https://docs.prefect.io/core/idioms/script-based.html I get "building local storage" in the terminal instead of "building github storage". Any idea why this could be happening?
    ✅ 1
    a
    • 2
    • 6
  • j

    Joshua Greenhalgh

    06/30/2022, 2:51 PM
    I thought I would have a play with 2.0 today - tried running the code here -> https://orion-docs.prefect.io/tutorials/first-steps/ - get following error?
    ValueError: the greenlet library is required to use this function. No module named 'greenlet'
    Any ideas?
    k
    • 2
    • 11
  • j

    JK

    06/30/2022, 3:01 PM
    Hello, been seeing some intermittent issues with logs — haven’t found repro steps yet, but the symptoms include: (a) only seeing error/warning despite root/prefect loggers are set to debug, and (b) some missing logs once in a while. Curious if anyone has seen this?
    ✅ 1
    k
    d
    a
    • 4
    • 12
  • i

    Isara Ovin

    06/30/2022, 3:04 PM
    Hi guys is there any way i can produce the following using prefect
    for col in date_columns:
            if col in df_columns:
                df[col] = df[col].str.replace('/', '-')
                with mp.Pool(mp.cpu_count()) as pool:
                    df[col] = pool.map(get_parsed_date, df[col])
    k
    a
    • 3
    • 7
  • j

    Jessica Smith

    06/30/2022, 5:05 PM
    I'm getting false notifications from Prefect Cloud, from the PagerDuty Automation I have enabled. The notification says "Prefect Action Failed Notification", but the details show that HTTP response code was 202 (Accepted). So these probably should not be causing a failure notification. The
    PagerDutyNotificationAction
    (
    24dfaasdfe-93dd-4b11-8dddc-4xxxxxx3d23
    ) has failed with 'HTTPError: 202 Accepted 'PagerDuty did not provide error details.' while contacting service API. Check your action configuration. See the Pagerduty docs for more details on error handling.' during execution triggered by
    FlowSLAFailedEvent
    event from hook
    b625xx8-d3x-4xxe-bxx6-1xxxxxx67f2
    .
    a
    • 2
    • 10
  • m

    Muhammad Usman Ghani

    06/30/2022, 5:12 PM
    Hello everyone, We have been using prefect 1.0 for quite some time in lower and higher environments. Recently, there was some activity ran by the security team and the server where prefect db is hosted was moved to private subnet. Right after that, several issues start popping up. • Unable to run the jobs from UI • Flows can't be scheduled because scheduler option is not working • Agent is not picking up the scheduled jobs Whole stack is deployed on AWS ECS Fargate. Issues still persist even after bringing back the subnets to public.
    ✅ 1
    a
    • 2
    • 4
  • s

    Sushma Adari

    06/30/2022, 7:37 PM
    Hi all, I am trying to pass parameters from the output of a Parent Task to a child flow. These parameters are dataframes that a few other child processes will use as well however I receive the below TypeError: My question is what is the best way to pass data from Parent to Child flows? I am trying to avoid querying multiple times since the query is intensive
    TypeError: Object of type DataFrame is not JSON serializable
    ✅ 1
    a
    • 2
    • 4
  • m

    Madison Schott

    06/30/2022, 8:30 PM
    Hi all, getting this error when trying to run a flow with the
    FlowRenameTask
    File "/Users/madisonschott/dbt_snowflake/data_pipeline_test_new_account.py", line 5, in <module>
        from prefect.tasks.prefect.flow_rename import FlowRenameTask
    ModuleNotFoundError: No module named 'prefect.tasks.prefect.flow_rename'
    a
    • 2
    • 3
  • j

    JK

    06/30/2022, 9:11 PM
    Anyone seeing a bunch of prefect errors on getting secret — in the last few hours at least? Couple of folks from our end seeing it…
    KeyError: 'xxx'
    ✅ 1
    c
    a
    d
    • 4
    • 13
  • e

    Emil Ordoñez

    06/30/2022, 10:04 PM
    Hello Everyone, I have a question, but IDK if there is a channel for quick questions or something similar
    a
    • 2
    • 1
  • y

    yu zeng

    07/01/2022, 3:32 AM
    @Anna Geller hi anna, is there any clear date that prefect 2.0 can support map operator? even we can write a loop to achive this , but we can't lookup the progress in that way. thank you
    a
    • 2
    • 1
  • o

    Oliver Mannion

    07/01/2022, 8:02 AM
    Hiya, is it possible to use a string concatenation of a task result as an input parameter to a downstream task? 🧵
    a
    • 2
    • 4
  • c

    Camilo Fernandez

    07/01/2022, 8:56 AM
    Good Morning everyone, 👋 I have you are having a great day. My problem: I have Prefect 1.2.2 set up in an EKS Cluster . The Kubernetes agent schedules flow runs but they don't get executed only queued with a submitted state.
    ✅ 1
    a
    • 2
    • 9
  • i

    Iuliia Volkova

    07/01/2022, 12:47 PM
    @Anna Geller hi Anna, I have a couple of questions relative to Prefect 2.0 - in thread.
    ✅ 1
    a
    • 2
    • 12
  • c

    Carlos Cueto

    07/01/2022, 1:28 PM
    Hi all. Ever since we started using Prefect Cloud, we have been making use of Prefect Secrets and using them across the board on almost all our flows. We have been encountering an intermittent error that happens on almost any of the flows at a random basis. Some flows don't use
    PrefectSecret
    explicitly but make use of secrets in other ways, such as
    Git
    storage
    git_clone_url_secret_name=<secret_key>
    . The error is the following:
    ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    The error is very vague and it's very difficult to understand what the error is exactly. Any help is appreciated, this is affecting our production workflows that sometimes take hours to run.
    👍 1
    a
    • 2
    • 1
  • s

    Sergey Goncharov

    07/01/2022, 2:41 PM
    Hi all! I'm new here and only started with my way with Prefect. I'd like to ask if someone could give some advice or just a documentation (I haven't found any): I'm using Prefect 2.0/beta, there I'd like to create a deployment for a flow with multiple tasks, each of them should use its own docker image. For Prefect v1 I used CreateContainer from prefect.tasks.docker library, but for Prefect v2 I cannot find any replacement. Please, any advices?
    b
    a
    • 3
    • 7
  • t

    Tim Enders

    07/01/2022, 2:59 PM
    I am using the
    GCSResult
    for my Task results. Does anybody know if there is a way to catch exceptions with it? We are seeing occasional failures on the GCS side and I don't want my flows failing because of it.
    b
    a
    m
    • 4
    • 5
  • w

    Wieger Opmeer

    07/01/2022, 4:40 PM
    @Chris Reuter: "Can't make it? Go to our Prefect YouTube channel for the recording!" Except that the latest Fireside Chats haven't made it to the YouTube channel?
    a
    c
    • 3
    • 3
  • h

    haven

    07/01/2022, 4:46 PM
    hi! i've spent hours and hours debugging my configuration with AWS ECS Fargate encountering
    ClusterNotFound
    but haven't gotten anywhere. i'd immensely appreciate it if anyone would have any insights! details in thread
    ✅ 1
    a
    • 2
    • 15
  • h

    haven

    07/02/2022, 6:34 AM
    could i check if ECS agent is ready for prefect 2 yet? Digging the updated and simplifed UI!
    🙌 1
    ✅ 1
    a
    • 2
    • 1
  • f

    Faheem Khan

    07/02/2022, 9:50 AM
    Prefect 2.0 Hello Everyone, Is there a way I can use multiprocessing.pool in prefect? sample code is in the thread, it works if I run it without prefect. how can I integrate it in prefect? thanks in advance.
    a
    • 2
    • 6
  • j

    jars

    07/03/2022, 6:16 AM
    Hello, is there an ongoing outage with Prefect Cloud? Seems some of our flows just stopped getting scheduled? I just had to manually toggle the flow off and on again to get it to produce future (scheduled) flow runs. Flow ID: f274e98a-3dcd-4823-8de7-4b74cc27615a
    👀 2
    ✅ 1
    l
    a
    • 3
    • 20
  • w

    William Jamir

    07/03/2022, 7:46 AM
    Hi, I have one flow where I have two steps that I use a map,
    Task A
    to do an upload and
    Task B
    to monitor if everything went fine after the upload from Task A. I noticed that using LocalDaskExecutor, only 10 tasks are processed at a time It’s possible to run more than 10 mapped tasks in parallel? Using LocalDaskExecutor with Prefect 1.2.2
    ✅ 1
    a
    • 2
    • 1
  • i

    Isara Ovin

    07/03/2022, 2:53 PM
    Hi can someone please share a some content on handling multiple work flows.
    ✅ 1
    a
    • 2
    • 4
  • d

    Denis Sh

    07/03/2022, 8:56 PM
    Good day everyone, did I miss something in UI or Orion self-hosted backend is not allowing to edit parameters of deployment/flow in UI? Thanks
    ✅ 1
    a
    • 2
    • 3
  • d

    Denis Sh

    07/03/2022, 8:59 PM
    And another one - if flow-decorated main function is async def'ined and consists of await asyncio.gather() then FlowRunner will properly asyncio.run() it?
    ✅ 1
    a
    • 2
    • 3
  • r

    Rajvir Jhawar

    07/03/2022, 11:36 PM
    I think prefect 2.0 docs are off, there is line the about work-queues and agents section that states this: “You must start an agent within the environment in which it will execute flow runs.” This statement is a little off and should come with note stating the agent doesn't necessarily have to be running in the same execution environment as the flow. For instance you can have an agent running in a VM connected to a k8 cluster using kubectl and run k8 jobs that way.
    💯 1
    🙌 1
    ✅ 2
    a
    m
    t
    • 4
    • 3
  • f

    Faheem Khan

    07/03/2022, 11:40 PM
    Prefect 2.0 Hi, my code runs fine when I run it with python abc.py but it doesn't do anything when I run it from UI i.e prefect deployment create abc.py. it spins up a new container when I run it from UI but then shutdowns I don't see any logs for debugging, the flow status shows completed. I am using dockerflow runner with DaskTaskRunner.
    ✅ 1
    a
    • 2
    • 5
Powered by Linen
Title
f

Faheem Khan

07/03/2022, 11:40 PM
Prefect 2.0 Hi, my code runs fine when I run it with python abc.py but it doesn't do anything when I run it from UI i.e prefect deployment create abc.py. it spins up a new container when I run it from UI but then shutdowns I don't see any logs for debugging, the flow status shows completed. I am using dockerflow runner with DaskTaskRunner.
✅ 1
a

Anna Geller

07/04/2022, 11:33 AM
Can you share: 1. The output of
prefect version
2. Some code to reproduce the issue 3. The exception traceback?
I was trying to reproduce with 2.0b7 but it worked just fine
from prefect import flow, task
from prefect import get_run_logger
from prefect.task_runners import DaskTaskRunner
from prefect.flow_runners import DockerFlowRunner
from prefect.deployments import DeploymentSpec
import time


@task
def compute_something(x):
    logger = get_run_logger()
    <http://logger.info|logger.info>("Computing: %d x 2 = %d", x, x * 2)
    time.sleep(2)


@flow(task_runner=DaskTaskRunner())
def dask_flow():
    for i in range(50):
        compute_something(i)


DeploymentSpec(name="staging", flow=dask_flow, flow_runner=DockerFlowRunner())
f

Faheem Khan

07/05/2022, 1:20 AM
@Anna Geller 2.0b6 I figured it out. I was calling the for loop outside the @flow(task_runner=DaskTaskRunner()).
@task
def compute_something(x):
    logger = get_run_logger()
    <http://logger.info|logger.info>("Computing: %d x 2 = %d", x, x * 2)
    time.sleep(2)


@flow(task_runner=DaskTaskRunner())
def dask_flow():
	list(map(compute_something, range(50))) #worked


if __name__ == '__main__':
	#list(map(dask_flow, range(50))) #didn't work
	dask_flow()


DeploymentSpec(name="staging", flow=dask_flow, flow_runner=DockerFlowRunner())
Thanks Anna
🙌 1
a

Anna Geller

07/05/2022, 11:11 AM
nice, thanks for the update
View count: 4