prefect-community #prefect-community

Channels

prefect-community

Lucien Fregosi

07/26/2022, 1:46 PM

Hi I’m working with prefect 2.0, trying to replace the storage capability with the new filesystem feature. I managed to publish the deployment but when the agent is launching a Job I got an issue with the lack of

gcsfs

dependency (using GCS as a

RemoteFileSystem

) I guess this dependency should be added to the docker image or maybe it’s possible to specify additional libs ?

✅ 1

➕ 1

4 replies · 2 participants

haris khan

07/26/2022, 1:54 PM

how to run multiple workqueues using helm on kubernetes in prefect orion

✅ 1

👀 1

1 reply · 2 participants

Jyothi

07/26/2022, 2:19 PM

Hi All, How to pass extra parameter (slack channel name) for the state_handler

✅ 1

Jyothi

07/26/2022, 2:21 PM

Currently I have this:

def escalate(obj, old_state, new_state):
    if new_state.is_failed() and new_state.message not in IGNORE_MESSAGES:
        msg = f"{obj} {new_state.message}"
        slack_post(msg, channel="#errors", username="Prefect")
    return new_state

and i would like to

def escalate(obj, old_state, new_state, error_channel="#errors"):

because i need have different channel for some flows

✅ 1

11 replies · 3 participants

iñigo

07/26/2022, 2:27 PM

Hello, I have the 0.15.4 version of prefect. is ther a guide on how to run a pipeline in Azure Data Factory ADF from within a Prefect Flow?

✅ 1

8 replies · 3 participants

Andreas Nigg

07/26/2022, 2:43 PM

Hey, quick question, are there any breaking changes planned for prefect 2.0 until the GA release? (And does the GA release contain breaking changes?). A quick yes/no answer is sufficient - we'll figure the details out 😄

✅ 1

4 replies · 4 participants

Michael Reynolds

07/26/2022, 3:11 PM

are there docs or examples that demonstrate how to use

orion

for non-terminating long running workflows?

✅ 1

8 replies · 2 participants

Slackbot

07/26/2022, 3:15 PM

This message was deleted.

10 replies · 3 participants

Chris Reuter

07/26/2022, 4:49 PM

https://twitter.com/PrefectIO/status/1551970847266684931?s=20&t=syAHCrWY9bu5wa78ZKrMqA We'll be giving away keyboards in the community, too! Stay tuned on Thursday of this week for a way to win a custom Prefect keyboard. These are WASD v3 107 key mechanical keyboards with Cherry MX Brown switches.

🔥 3

:its-beautiful: 4

:1000000: 2

🚀 3

👀 6

Chu

07/26/2022, 5:16 PM

Hi, community, how to understand the trigger, if upstream flow are using mapping to create parallel flows, will this arg implies my current flow will wait until every parallel upstream flow becomes successful? Since my current flow is still mapping parameters as parallel run, what I want is when each parallel upstream flow finishes, no matter its peer flow finishes or not, it will continue to downstream flow. How could I set trigger args, if I want this to be achieved, this will help our company greatly enhance performance, thanks!! (we use Prefect 1.0)

create_flow_run(trigger = all_successful)

✅ 1

2 replies · 3 participants

Mike Grabbe

07/26/2022, 5:39 PM

Following up from the Prefect Demo that ended just now, could you make the demo flows available on github? Or point me at them if they already exist somewhere?

✅ 1

5 replies · 3 participants

Sebastián Montoya Tapia

07/26/2022, 8:14 PM

Hi! Currently I am using AWS Fargate for the ECSRun, one of my tasks needs to insert data on a RDS Instance, the ECSAgent has the correct security group to be able to access the database. But when I check the tasks when are running in the cluster, they create a ENI in the default vpc which is different from the one I am using, does anyone knows how to solve this issue ?

16 replies · 2 participants

Geert-Jan Van den Bogaerde

07/26/2022, 9:51 PM

Hi prefect community, any prefect users in or Belgium, that would be interested in setting up a local meetup to exchange experiences and best practices face to face? At www.garvis.ai , prefect is a key piece in our data pipelines. We hire globally and are remote first, but our HQ in Belgium and wondering if anyone else here is too and wants to share experiences with prefect and data engineering. ;-)

🙌 1

👍 1

1 reply · 2 participants

Charles Phares

07/27/2022, 12:43 AM

Absolute Prefect Newb here! I'm going thru the Getting Started Doc, in the Run a flow section, it states "I will see the following logs after running

flow.run()

" Yeah. Exactly. ...and where would I find these logs???

5 replies · 2 participants

Viet Nguyen

07/27/2022, 1:13 AM

Is it just me that I cannot login using GitHub to Prefect cloud 🤔

✅ 1

9 replies · 3 participants

Chris Hansen

07/27/2022, 2:25 AM

i’m on GCP and want to get up and running on Prefect Cloud with agents on GKE and CI/CD through github. is there a guide somewhere that explains how to set any of this up?

2 replies · 2 participants

Faheem Khan

07/27/2022, 3:11 AM

Prefect 2.0b8, dockerflowrunner with dask. I am getting the following error, anyone know the soloution?

4 replies · 2 participants

Pedro Machado

07/27/2022, 5:46 AM

Hi there. How do you include extra files when packaging a flow as a Docker image on Prefect 2.0? (the equivalent of the

files

argument in 1.0 Docker storage). Thanks

✅ 1

1 reply · 2 participants

Ed Burroughes

07/27/2022, 7:51 AM

Hi there, I'm looking at adding runconfig (i.e. environment variables) using the graphql api, any ideas how to do this using Prefect 1. Thank you 🙂

✅ 1

2 replies · 2 participants

Riccardo Tesselli

07/27/2022, 8:59 AM

hi there. I’m running Prefect 2.0 on beta cloud. Since this morning I receive this error when submitting jobs to K8s:

Response: {'detail': 'The request specified API version 0.7.0 but this serverrequires version 0.8.0 or higher.'}

btw I’m using default image for KubernetesJob and my code is up to date with version

2.0b15

✅ 1

➕ 2

7 replies · 3 participants

Viet Nguyen

07/27/2022, 11:40 AM

Hi guys, I'm trying to run sample code using DaskTaskRunner but got this error

ImportError: cannot import name 'DaskTaskRunner' from partially initialized module 'prefect_dask' (most likely due to a circular import)

I used

2.0b15

latest

prefect-dask

on Python

3.8.13

3.9

3.8.4

3.7.13

(freshly installed Conda envs), I'm wondering if you have come across this issue before? Many thanks

✅ 1

14 replies · 2 participants

Sana Shaikh

07/27/2022, 11:41 AM

@Anna Geller The problem is whenever I start local agent of prefect and write the commands of registration of flow and running it, it just stuck there and don't proceed

✅ 1

7 replies · 4 participants

Benjamin.bgx

07/27/2022, 1:10 PM

Anyone have seen an error lile :

FileNotFoundError: [Errno 2] No such file or directory: 'python'

when running a flow on an agent from prefect cloud 2 ?

4 replies · 3 participants

Josh Paulin

07/27/2022, 1:27 PM

Is it possible to run previous versions of a Flow from the UI? I keep seeing

This flow is archived and cannot be run

, when I flip back to an old version. Wanted to make sure that’s always expected and I’m not just doing something wrong when registering new versions of the flow.

✅ 1

1 reply · 2 participants

Stephen Herron

07/27/2022, 1:46 PM

Hi i was looking for some help with a deployment recipe in github actions My

prefect register flows

workflow does not appear to return an error code when the there’s a issue with deploying a flow.. Only thing I can see I’m doing differently is that i’m using a matrix to • for each prefect project • deploy changed flows in each project has anyone had this problem before?

👀 1

3 replies · 2 participants

Adam

07/27/2022, 3:13 PM

Hi team, I don’t really know GraphQL too well, could someone give me an example of the GraphQL to write to trigger a specific flow?

3 replies · 3 participants

Jack Sundberg

07/27/2022, 3:52 PM

For Orion, is there a way to have subflows return futures as they did in earlier beta versions? So something like a my_flow.submit() method for parallelism...?

👀 1

9 replies · 4 participants

John

07/27/2022, 4:04 PM

Hi, I'm trying to debug a flow locally and there seems to be a crazy slowdown effect in running code via Prefect versus just a regular CLI. I am on Prefect Orion 2.0b8. Rough flow looks like

@task
def update_task(file_type: str):
    update_files(file_type=file_type)

@flow
def update_flow(file_type: str):
    update_task(file_type=file_type)

if __name__ == '__main__':
    update_flow(file_type="incremental")

I am running it as

python -m tasks.prefect_flows.update_flow

. This takes 10+ minutes to run. By comparison, just running the same code with a click CLI takes 10s. Any idea what's going on?

✅ 1

8 replies · 3 participants

Chris Hansen

07/27/2022, 4:21 PM

do Prefect 1.0 tasks like the Fivetran tasks work with 2.0?

✅ 1

2 replies · 2 participants

Jehan Abduljabbar

07/27/2022, 5:02 PM

Hello, I am having the following error with DbtShellTask(): "prefect.DbtShellTask | Command failed with exit code 127" I am following the flow example in here: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask (with providing my own arguments). I can't tell what is source of the issue

✅ 1

8 replies · 4 participants

Title

Jehan Abduljabbar

07/27/2022, 5:02 PM

✅ 1

Khuyen Tran

07/27/2022, 5:12 PM

Hi Jehan, could you provide a minimal example of your code?

Jehan Abduljabbar

07/27/2022, 5:17 PM

from prefect import Flow from prefect.tasks.dbt import DbtShellTask with Flow(name="dbt_flow") as f: task = DbtShellTask( profile_name="profile name", environment='dev', dbt_kwargs={ "type": "snowflake", "account": "account name", "user": "username", "authentictor": "externalbrowser", # tried password as well on other acct, same error "role": "role name", "warehouse": "wh name", "database": "db name" "schema": "schema name", "threads": 4, "client_session_keep_alive": True, }, overwrite_profiles=True, profiles_dir= r"C:\Users\jabduljabbar\.dbt" )(command='dbt run') out = f.run()

Andrew Huang

07/27/2022, 6:31 PM

do you have dbt installed?

e.g.

which dbt

:thank-you: 1

Anna Geller

07/27/2022, 7:00 PM

@Jehan Abduljabbar The 127 error code indicates “command not found”. This occurs when any given command within your Bash script or on Bash command line is not found in any of the paths defined by PATH system environment variable. The solution is to make sure that the command your are using can be found within your $PATH. If the command is not in your path either include it or use absolute full path to it. TL;DR: you didn't install dbt properly and you can't run any dbt commands in your execution environment

Jehan Abduljabbar

07/27/2022, 8:54 PM

I have dbt-core and dbt-snowflake installed by pip. dbt run command works when running it from terminal, in the same execution environment as where I am running the flow. But it doesn't work when using the DbtShellTask

Anna Geller

07/28/2022, 12:32 AM

are you running this on Windows? somehow the shell task cannot find dbt in your path

Jehan Abduljabbar

07/28/2022, 12:12 PM

yes on windows

View count: 3