https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • c

    Christopher Boyd

    09/13/2022, 8:52 PM
    Hi Kelv, Jobs should be transient - can you give some examples of the names of the pods you are removing ?
  • c

    Christopher Boyd

    09/13/2022, 8:52 PM
    You can also pause the work queue / deployment
  • l

    Leon Kozlowski

    09/13/2022, 9:17 PM
    Hi all - I’ve been able to successfully deploy a flow to prefect 2.0 using a kubernetes-job block and custom image that points my internal docker registry - however when I try to run the flow, I’m getting a k8s error from the agent:
    stern --tail 500 prefect-orion
    
    "message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout"
    m
    4 replies · 2 participants
  • f

    Frank Hardisty

    09/14/2022, 12:30 AM
    Hello Frank Hardisty of Replica here. We had our Prefect logs turned off due to extreme log spam. Brett Naul and I implemented a fix that should prevent that source of log spam from happening again, could we please have Prefect logs turned back on?
    ✅ 1
    a
    2 replies · 2 participants
  • a

    Anton L.

    09/14/2022, 6:24 AM
    Hi, everyone! We are using prefect 1.2.4 and recently got the following problem when started a flow from CLI with
    -w
    option:
    f"`watch_flow_run` timed out after "
    RuntimeError: `watch_flow_run` timed out after 12.0 hours of waiting for completion. Your flow run is still in state: <Running: "Running flow.">
    Is there a way to pass a different max_duration to a
    watch_flow_run
    in such usecase?
    m
    2 replies · 2 participants
  • g

    Gintautas Jankus

    09/14/2022, 6:52 AM
    Hi everyone, I was wondering why when creating subflow and passing parameters in UI schema prefect creates additional List and Dict tasks. It adds unnecessary complexity and visual noise in our schema, maybe it‘s possible somehow create subflows without these additional downstream Dict and List tasks in UI? :)
    ✅ 1
    a
    4 replies · 2 participants
  • d

    dammy arinde

    09/14/2022, 8:41 AM
    Hi everyone, we are using Prefect to run a job that connects to redshift cluster and unloads files to s3 bucket. It's been working fine in the dev environment but when we try in test environment, we get this error. I was wondering if anyone has an idea on what the issue could be. Thanks!
    ✅ 1
    m
    2 replies · 2 participants
  • a

    Andreas Nigg

    09/14/2022, 9:44 AM
    Just a shoutout to the prefect team for v2.4.0. 🎉 😛arty-parrot: This release actually solves my two last annoyances I had with prefect 2.0, namely not being able to set custom parameters and schedule timezones via CLI. • Support for setting custom parameter values directly from the deployment build CLI • New CLI for interacting with and setting custom timezone-dependent schedules for your deployments incl. the ability to pause or resume a schedule directly from the CLI Thanks to the team for this awesome release and it's incredible to see how fast prefect 2.0 moves in the perfect direction.
    :blob-attention-gif: 8
    ❤️ 5
    🎉 5
    🚀 3
    👍 1
    :thank-you: 3
    a
    b
    +1
    4 replies · 4 participants
  • c

    Christian Vogel

    09/14/2022, 11:48 AM
    Hi Prefect Community! Do you know if it is already possible to specify cpu/gpu resource requirements per task for the RayTaskRunner? So far I only managed to configure the Ray Cluster, but I am not sure if its possible to map the tasks to the different worker types to then autoscale accordingly (for example only scaling out gpu workers if needed).
    ✅ 1
    r
    11 replies · 2 participants
  • j

    José Duarte

    09/14/2022, 1:24 PM
    Hey all, I am trying to run flows using Docker as infrastructure. I’ve set the pull policy to never because I’m currently performing tests and the image is not public — in other words, I am forcing the agent to use the local ones. So far, so good. The issue arises when running the flow, the container will try to run
    python -m prefect.engine flow_uuid
    , it fetches the flow but then tries to copy files into the same folder, in this case
    deployment.path
    is the
    cwd
    &
    pwd
    which in turn means that
    .
    resolves to
    deployment.path
    — making the
    filesystems.LocalFileSystem.get_directory
    copy the files into themselves (through
    shutil.copytree
    ) which ends up in a HUGE error like so:
    [<redacted for brevity>, '.', 'c', 'f', 'g', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'i', 's', 'o', 'r', 't', '.', 'c', 'f', 'g', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e', '<', 'D', 'i', 'r', 'E', 'n', 't', 'r', 'y', ' ', "'", '.', 'f', 'l', 'a', 'k', 'e', '8', "'", '>', ' ', 'a', 'n', 'd', ' ', "'", '.', '/', '.', 'f', 'l', 'a', 'k', 'e', '8', "'", ' ', 'a', 'r', 'e', ' ', 't', 'h', 'e', ' ', 's', 'a', 'm', 'e', ' ', 'f', 'i', 'l', 'e']
    With regards to Prefect’s code, I have a solution, just check if both paths match and don’t copy in that case, but I’d like to get some feedback from the Prefect team before reporting a bug and sending in a fix — i.e. if I am doing something wrong.
    ✅ 1
    2 replies · 1 participant
  • v

    Vlad Tudor

    09/14/2022, 1:46 PM
    Hello community, Anyone experienced on setting up ECS Agents? I successfully register the flow and the agent, the task appears in AWS on the designated Cluster but nothing happens (the Prefect Task remains pending) Please, leave here any useful resource 😄 thanks
    ✅ 1
    c
    b
    6 replies · 3 participants
  • j

    José Duarte

    09/14/2022, 2:10 PM
    In Prefect 2, what is the equivalent to the Module storage from Prefect 1? https://docs-v1.prefect.io/api/latest/storage.html#module
    m
    k
    4 replies · 3 participants
  • l

    Leon Kozlowski

    09/14/2022, 2:43 PM
    Hi all, I am running into issues with my agent being able to submit flows to my infrastructure: Prefect Version: 2.3.0 Infrastructure: KubernetesJob Traceback:
    | ERROR   | prefect.agent - Failed to submit flow run 'FLOW_RUN_ID' to infrastructure.
    
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout","details":{"name":"POST","group":"batch","kind":"Job"},"code":500}
    More configurations in thread
    ✅ 1
    n
    10 replies · 2 participants
  • b

    Blake Hamm

    09/14/2022, 3:29 PM
    Is there a Prefect/python-native way for a flow to trigger different deployments? I have a few deployments that have different Docker and k8's requirements, but are related. A simple example is one deployment loading data to s3 and another deployment running dbt on that data. It would be incredible if I could create a parent flow which triggered the pre-defined deployments sequentially. Some (hacky-ish) solutions I can think of: • Define the logic in one big flow and consolidate to one Docker container • Create a flow with the prefect-shell collection and run deployment CLI commands to trigger the deployments sequential These solutions would definitely work as expected. I am curious if there is another solution using the new python
    Deployment
    class (not seeing anything in the API reference). Or, if anyone else would find a
    .run()
    method on the
    Deployment
    class useful? 😼
    ✅ 1
    n
    2 replies · 2 participants
  • c

    chris arettines

    09/14/2022, 4:15 PM
    HI all, I'm new to prefect, loving it so far. I'm working on a 1.0 project that I've inherited - I was wondering if there is a way to utilize the graphql api to verify that all downstream tasks (including children of children etc) of a particular task run were successful. I can do it via repeated querying for downstream_task info, but it would be nice if there was some other method that didn't involve hitting the api repeatedly. Thanks.
    ✅ 1
    m
    2 replies · 2 participants
  • s

    Sam Garvis

    09/14/2022, 5:17 PM
    https://medium.com/the-prefect-blog/serverless-real-time-data-pipelines-on-aws-with-prefect-ecs-and-github-actions-1737c80da3f5 I'm trying to use the while true to run a flow. It works locally and keeps on creating new flows like I want. When I run this through the cloud and a helm prefect-agent on GKE, it runs one flow then stops.
    👀 1
    m
    1 reply · 2 participants
  • k

    Khuyen Tran

    09/14/2022, 5:18 PM
    Hey all, we’re hosting @Oscar Björhn, a Club42 member, on PrefectLive today at 3p Eastern where he’s going to show us how he integrates Prefect into his workflow, including using different flows with different environments inside a Docker container. Come join us live on Twitch!
    🙌 2
    🎉 2
    🔥 2
    1 reply · 1 participant
  • p

    Pedro Machado

    09/14/2022, 5:20 PM
    Hi there. I have a flow that is running on ECS (Prefect 1.0). I see that some tasks got stuck in running state and they have been there for a long time. How can we avoid this? Can prefect cloud do something about them? Is it possible to set a timeout at the task level? I manually stopped them but it would be good to have an automated way of killing these if they run for more than X hours. Thanks!
    m
    2 replies · 2 participants
  • k

    Konstantinos

    09/14/2022, 5:29 PM
    Hello - have a quick question regarding the local dask executor: We have stumbled upon this issue as well and we are working with prefect version 0.13.10. So it seems that the examples as suggested in prefect docs for LocalDaskExecutor dont work and we need to go via this RemoteEnvironment variable. Has this been addressed at later versions of prefect or should we just proceed with the RemoteEnvironment solution ?
    m
    2 replies · 2 participants
  • a

    Arun Giridharan

    09/14/2022, 5:46 PM
    It looks like create_flow_run is internally looking for a Cloud URL. Is there a corresponding call for Prefect Core that doesn't need a cloud URL?
    ✅ 1
    m
    1 reply · 2 participants
  • j

    Jason Leacox

    09/14/2022, 6:07 PM
    Hi, I'm curious if there are project count limits on Prefect Cloud?
    m
    3 replies · 2 participants
  • k

    kiran

    09/14/2022, 7:06 PM
    I keep getting this error when trying to have a scheduled run from a deployed flow. It runs fine as a non-deployed flow (when I run it from my terminal). I built and applied the deployment on another flow/script and it worked (but it also doesn’t have any non-prefect imports). Should I
    import psycopg2
    from somewhere else instead? Running 2.3.2
    Flow could not be retrieved from deployment.
    Traceback (most recent call last):
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/tmp/tmpcre52tf5prefect/SCRIPTNAME.py", line 3, in <module>
        import psycopg2
    ModuleNotFoundError: No module named 'psycopg2'
    ✅ 1
    n
    1 reply · 2 participants
  • c

    Chris Gunderson

    09/14/2022, 7:45 PM
    Hi, I'm just starting to use prefect 2.0. I noticed the prefect-email collection. Is it possible to call a new flow from a task?
    ✅ 1
    k
    3 replies · 2 participants
  • a

    Anat Tal Gagnon

    09/14/2022, 7:47 PM
    Hi guys, I need super urgent help!! My prefect UI stopped loading suddenly!
    ✅ 1
    m
    r
    5 replies · 3 participants
  • a

    Anat Tal Gagnon

    09/14/2022, 7:50 PM
  • a

    Andrew Pruchinski

    09/14/2022, 8:49 PM
    Hello! A flow run was created by a nonexistent user. Is this expected?
    m
    2 replies · 2 participants
  • t

    Tony Piazza

    09/14/2022, 8:53 PM
    I have a flow function decorated with @flow and specify a DaskTaskRunner. I thought this would cause my tasks to get run on my Dask cluster but I saw this in your documentation: “To run your task with a task runner, you must call the task with .submit().” Is this still true even if my flow function is configured as described above?
    ✅ 1
    m
    1 reply · 2 participants
  • m

    Michael Dyer

    09/14/2022, 9:23 PM
    does an agent that services a work queue with concurrency 3 end up running three python threads (single python interpreter)? if so, does each thread have it's own event loop?
    j
    1 reply · 2 participants
  • l

    Leon Kozlowski

    09/14/2022, 9:24 PM
    Can flow run context be fetched within a task in 2.0?
    ✅ 1
    m
    5 replies · 2 participants
  • k

    Kelvin Garcia

    09/14/2022, 9:34 PM
    hi all, is there a way to use a parameter as the name of the folder for a
    S3Result
    ? for a flow level result I want to have something like this
    bucket/jobs/{parameter}
    ✅ 1
    m
    2 replies · 2 participants
Powered by Linen
Title
k

Kelvin Garcia

09/14/2022, 9:34 PM
hi all, is there a way to use a parameter as the name of the folder for a
S3Result
? for a flow level result I want to have something like this
bucket/jobs/{parameter}
✅ 1
m

Mason Menges

09/14/2022, 9:53 PM
Hey @Kelvin Garcia I think this might be what you're looking for https://docs-v1.prefect.io/core/advanced_tutorials/using-results.html#templating-a-task-s-location
k

Kelvin Garcia

09/14/2022, 10:03 PM
I'll check that out, thanks
View count: 1