https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • l

    Lucien Fregosi

    07/26/2022, 1:46 PM
    Hi I’m working with prefect 2.0, trying to replace the storage capability with the new filesystem feature. I managed to publish the deployment but when the agent is launching a Job I got an issue with the lack of
    gcsfs
    dependency (using GCS as a
    RemoteFileSystem
    ) I guess this dependency should be added to the docker image or maybe it’s possible to specify additional libs ?
    ✅ 1
    ➕ 1
    a
    4 replies · 2 participants
  • h

    haris khan

    07/26/2022, 1:54 PM
    how to run multiple workqueues using helm on kubernetes in prefect orion
    ✅ 1
    👀 1
    b
    1 reply · 2 participants
  • j

    Jyothi

    07/26/2022, 2:19 PM
    Hi All, How to pass extra parameter (slack channel name) for the state_handler
    ✅ 1
  • j

    Jyothi

    07/26/2022, 2:21 PM
    Currently I have this:
    def escalate(obj, old_state, new_state):
        if new_state.is_failed() and new_state.message not in IGNORE_MESSAGES:
            msg = f"{obj} {new_state.message}"
            slack_post(msg, channel="#errors", username="Prefect")
        return new_state
    and i would like to
    def escalate(obj, old_state, new_state, error_channel="#errors"):
    because i need have different channel for some flows
    ✅ 1
    a
    k
    11 replies · 3 participants
  • i

    iñigo

    07/26/2022, 2:27 PM
    Hello, I have the 0.15.4 version of prefect. is ther a guide on how to run a pipeline in Azure Data Factory ADF from within a Prefect Flow?
    ✅ 1
    t
    n
    8 replies · 3 participants
  • a

    Andreas Nigg

    07/26/2022, 2:43 PM
    Hey, quick question, are there any breaking changes planned for prefect 2.0 until the GA release? (And does the GA release contain breaking changes?). A quick yes/no answer is sufficient - we'll figure the details out 😄
    ✅ 1
    t
    b
    +1
    4 replies · 4 participants
  • m

    Michael Reynolds

    07/26/2022, 3:11 PM
    are there docs or examples that demonstrate how to use
    orion
    for non-terminating long running workflows?
    ✅ 1
    a
    8 replies · 2 participants
  • s

    Slackbot

    07/26/2022, 3:15 PM
    This message was deleted.
    n
    s
    10 replies · 3 participants
  • c

    Chris Reuter

    07/26/2022, 4:49 PM
    https://twitter.com/PrefectIO/status/1551970847266684931?s=20&t=syAHCrWY9bu5wa78ZKrMqA We'll be giving away keyboards in the community, too! Stay tuned on Thursday of this week for a way to win a custom Prefect keyboard. These are WASD v3 107 key mechanical keyboards with Cherry MX Brown switches.
    🔥 3
    :its-beautiful: 4
    :1000000: 2
    🚀 3
    👀 6
  • c

    Chu

    07/26/2022, 5:16 PM
    Hi, community, how to understand the trigger, if upstream flow are using mapping to create parallel flows, will this arg implies my current flow will wait until every parallel upstream flow becomes successful? Since my current flow is still mapping parameters as parallel run, what I want is when each parallel upstream flow finishes, no matter its peer flow finishes or not, it will continue to downstream flow. How could I set trigger args, if I want this to be achieved, this will help our company greatly enhance performance, thanks!! (we use Prefect 1.0)
    create_flow_run(trigger = all_successful)
    ✅ 1
    r
    m
    2 replies · 3 participants
  • m

    Mike Grabbe

    07/26/2022, 5:39 PM
    Following up from the Prefect Demo that ended just now, could you make the demo flows available on github? Or point me at them if they already exist somewhere?
    ✅ 1
    c
    t
    5 replies · 3 participants
  • s

    Sebastián Montoya Tapia

    07/26/2022, 8:14 PM
    Hi! Currently I am using AWS Fargate for the ECSRun, one of my tasks needs to insert data on a RDS Instance, the ECSAgent has the correct security group to be able to access the database. But when I check the tasks when are running in the cluster, they create a ENI in the default vpc which is different from the one I am using, does anyone knows how to solve this issue ?
    r
    16 replies · 2 participants
  • g

    Geert-Jan Van den Bogaerde

    07/26/2022, 9:51 PM
    Hi prefect community, any prefect users in or Belgium, that would be interested in setting up a local meetup to exchange experiences and best practices face to face? At www.garvis.ai , prefect is a key piece in our data pipelines. We hire globally and are remote first, but our HQ in Belgium and wondering if anyone else here is too and wants to share experiences with prefect and data engineering. ;-)
    🙌 1
    👍 1
    c
    1 reply · 2 participants
  • c

    Charles Phares

    07/27/2022, 12:43 AM
    Absolute Prefect Newb here! I'm going thru the Getting Started Doc, in the Run a flow section, it states "I will see the following logs after running
    flow.run()
    " Yeah. Exactly. ...and where would I find these logs???
    s
    5 replies · 2 participants
  • v

    Viet Nguyen

    07/27/2022, 1:13 AM
    Is it just me that I cannot login using GitHub to Prefect cloud 🤔
    ✅ 1
    k
    a
    9 replies · 3 participants
  • c

    Chris Hansen

    07/27/2022, 2:25 AM
    i’m on GCP and want to get up and running on Prefect Cloud with agents on GKE and CI/CD through github. is there a guide somewhere that explains how to set any of this up?
    b
    2 replies · 2 participants
  • f

    Faheem Khan

    07/27/2022, 3:11 AM
    Prefect 2.0b8, dockerflowrunner with dask. I am getting the following error, anyone know the soloution?
    m
    4 replies · 2 participants
  • p

    Pedro Machado

    07/27/2022, 5:46 AM
    Hi there. How do you include extra files when packaging a flow as a Docker image on Prefect 2.0? (the equivalent of the
    files
    argument in 1.0 Docker storage). Thanks
    ✅ 1
    a
    1 reply · 2 participants
  • e

    Ed Burroughes

    07/27/2022, 7:51 AM
    Hi there, I'm looking at adding runconfig (i.e. environment variables) using the graphql api, any ideas how to do this using Prefect 1. Thank you 🙂
    ✅ 1
    c
    2 replies · 2 participants
  • r

    Riccardo Tesselli

    07/27/2022, 8:59 AM
    hi there. I’m running Prefect 2.0 on beta cloud. Since this morning I receive this error when submitting jobs to K8s:
    Response: {'detail': 'The request specified API version 0.7.0 but this serverrequires version 0.8.0 or higher.'}
    btw I’m using default image for KubernetesJob and my code is up to date with version
    2.0b15
    ✅ 1
    ➕ 2
    b
    a
    7 replies · 3 participants
  • v

    Viet Nguyen

    07/27/2022, 11:40 AM
    Hi guys, I'm trying to run sample code using DaskTaskRunner but got this error
    ImportError: cannot import name 'DaskTaskRunner' from partially initialized module 'prefect_dask' (most likely due to a circular import)
    I used
    2.0b15
    latest
    prefect-dask
    on Python
    3.8.13
    3.9
    3.8.4
    3.7.13
    (freshly installed Conda envs), I'm wondering if you have come across this issue before? Many thanks
    ✅ 1
    a
    14 replies · 2 participants
  • s

    Sana Shaikh

    07/27/2022, 11:41 AM
    @Anna Geller The problem is whenever I start local agent of prefect and write the commands of registration of flow and running it, it just stuck there and don't proceed
    ✅ 1
    a
    t
    +1
    7 replies · 4 participants
  • b

    Benjamin.bgx

    07/27/2022, 1:10 PM
    Anyone have seen an error lile :
    FileNotFoundError: [Errno 2] No such file or directory: 'python'
    when running a flow on an agent from prefect cloud 2 ?
    t
    a
    4 replies · 3 participants
  • j

    Josh Paulin

    07/27/2022, 1:27 PM
    Is it possible to run previous versions of a Flow from the UI? I keep seeing
    This flow is archived and cannot be run
    , when I flip back to an old version. Wanted to make sure that’s always expected and I’m not just doing something wrong when registering new versions of the flow.
    ✅ 1
    b
    1 reply · 2 participants
  • s

    Stephen Herron

    07/27/2022, 1:46 PM
    Hi i was looking for some help with a deployment recipe in github actions My
    prefect register flows
    workflow does not appear to return an error code when the there’s a issue with deploying a flow.. Only thing I can see I’m doing differently is that i’m using a matrix to • for each prefect project • deploy changed flows in each project has anyone had this problem before?
    👀 1
    b
    3 replies · 2 participants
  • a

    Adam

    07/27/2022, 3:13 PM
    Hi team, I don’t really know GraphQL too well, could someone give me an example of the GraphQL to write to trigger a specific flow?
    v
    k
    3 replies · 3 participants
  • j

    Jack Sundberg

    07/27/2022, 3:52 PM
    For Orion, is there a way to have subflows return futures as they did in earlier beta versions? So something like a my_flow.submit() method for parallelism...?
    👀 1
    t
    k
    +1
    9 replies · 4 participants
  • j

    John

    07/27/2022, 4:04 PM
    Hi, I'm trying to debug a flow locally and there seems to be a crazy slowdown effect in running code via Prefect versus just a regular CLI. I am on Prefect Orion 2.0b8. Rough flow looks like
    @task
    def update_task(file_type: str):
        update_files(file_type=file_type)
    
    @flow
    def update_flow(file_type: str):
        update_task(file_type=file_type)
    
    if __name__ == '__main__':
        update_flow(file_type="incremental")
    I am running it as
    python -m tasks.prefect_flows.update_flow
    . This takes 10+ minutes to run. By comparison, just running the same code with a click CLI takes 10s. Any idea what's going on?
    ✅ 1
    t
    k
    8 replies · 3 participants
  • c

    Chris Hansen

    07/27/2022, 4:21 PM
    do Prefect 1.0 tasks like the Fivetran tasks work with 2.0?
    ✅ 1
    t
    2 replies · 2 participants
  • j

    Jehan Abduljabbar

    07/27/2022, 5:02 PM
    Hello, I am having the following error with DbtShellTask(): "prefect.DbtShellTask | Command failed with exit code 127" I am following the flow example in here: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask (with providing my own arguments). I can't tell what is source of the issue
    ✅ 1
    k
    a
    +1
    8 replies · 4 participants
Powered by Linen
Title
j

Jehan Abduljabbar

07/27/2022, 5:02 PM
Hello, I am having the following error with DbtShellTask(): "prefect.DbtShellTask | Command failed with exit code 127" I am following the flow example in here: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask (with providing my own arguments). I can't tell what is source of the issue
✅ 1
k

Khuyen Tran

07/27/2022, 5:12 PM
Hi Jehan, could you provide a minimal example of your code?
j

Jehan Abduljabbar

07/27/2022, 5:17 PM
from prefect import Flow from prefect.tasks.dbt import DbtShellTask with Flow(name="dbt_flow") as f: task = DbtShellTask( profile_name="profile name", environment='dev', dbt_kwargs={ "type": "snowflake", "account": "account name", "user": "username", "authentictor": "externalbrowser", # tried password as well on other acct, same error "role": "role name", "warehouse": "wh name", "database": "db name" "schema": "schema name", "threads": 4, "client_session_keep_alive": True, }, overwrite_profiles=True, profiles_dir= r"C:\Users\jabduljabbar\.dbt" )(command='dbt run') out = f.run()
a

Andrew Huang

07/27/2022, 6:31 PM
do you have dbt installed?
e.g.
which dbt
:thank-you: 1
a

Anna Geller

07/27/2022, 7:00 PM
@Jehan Abduljabbar The 127 error code indicates “command not found”. This occurs when any given command within your Bash script or on Bash command line is not found in any of the paths defined by PATH system environment variable. The solution is to make sure that the command your are using can be found within your $PATH. If the command is not in your path either include it or use absolute full path to it. TL;DR: you didn't install dbt properly and you can't run any dbt commands in your execution environment
j

Jehan Abduljabbar

07/27/2022, 8:54 PM
I have dbt-core and dbt-snowflake installed by pip. dbt run command works when running it from terminal, in the same execution environment as where I am running the flow. But it doesn't work when using the DbtShellTask
a

Anna Geller

07/28/2022, 12:32 AM
are you running this on Windows? somehow the shell task cannot find dbt in your path
j

Jehan Abduljabbar

07/28/2022, 12:12 PM
yes on windows
View count: 3