prefect-community #prefect-community

Channels

prefect-community

Jelle Vegter

05/22/2022, 2:01 PM

Hi all, I’m looking for good options of where to setup up the prefect agent and have it running. Currently I have a virtual machine with a terminal open listening. Does anyone have resources I can look at to compare? I’m on Azure for if that matters.

Joshua Greenhalgh

05/22/2022, 2:45 PM

Something strange happened with one of my flow runs today and I wonder if anyone could help me understand? A flow was supposed to start on the hour but didn't actually start until 40 mins later - I am running on k8s - this is the logs for the agent;

Nash Taylor

05/22/2022, 4:26 PM

With the recent release of the Cloud Run Jobs feature on Google Cloud, are there any plans to make an Agent out of it? Am I correct in thinking that would be a great fit for prefect flows?

💡 1

Todd de Quincey

05/22/2022, 4:44 PM

Is there an architectural diagram anywhere to explain how all of the different Prefect components fit together (e.g. agents, executors etc). Trying to wrap my head around some of the terminology and concepts after coming from a heavy Airflow background.

✅ 1

Nash Taylor

05/22/2022, 5:12 PM

I had a thought this morning for a personal project that I could use Prefect for. As always I'd like to start with 2.0, but it leads me to a pretty important question about security. The idea I had would center around using my banking data via a python "faux API" (one of those screen scraper packages that tries to stand in place of an API). Obviously to use this, I would require two extremely sensitive secrets (card and password). Given that 2.0 is in a beta, I guess my question is, are Secrets currently in a place where I could insert these two pieces of data and use them in a Flow? Or am I better off for now using a different secret manager and accessing it from within a task?

✅ 1

Bob Colner

05/22/2022, 7:23 PM

question about

shell_run_command

in orion. I’m not able to pass a

retries

parameter to the task.

TypeError: got an unexpected keyword argument 'retries'

any ideas?

Hafsa Junaid

05/22/2022, 8:51 PM

Hey Team! What's the prefecthq image latest release?

Nash Taylor

05/22/2022, 10:26 PM

I definitely sound like a broken record at this point, but I'm still stuck on trying to understand the reasoning behind [these overloads](https://github.com/PrefectHQ/prefect/blob/orion/src/prefect/tasks.py#L231-L255) on the base Task class. Namely, the use of NoReturn in the first overload. Here's a minimal example where I use NoReturn in a Task skeleton, and the resulting

reveal_type

of a task run according to mypy: (threaded to avoid an obnoxiously long message)

✅ 1

Jacob Blanco

05/23/2022, 12:58 AM

We have a flow that spawns 100s of mapped runs (we have a chain of 6-7 mapped tasks) and we’re consistently seeing 1 or 2 get stuck in pending state downstream from a failure in Cloud: 1. Any ideas on how to address this? This behaviour started fairly recently 2. Can I get a hint on how to kill the pending task runs using GraphQL? I can find the tasks runs (flow status is Cancelled with tasks with state pending) but I’m not clear on how to turn that into a mutation.

Jeff Kehler

05/23/2022, 2:57 AM

I'm having an issue using the

BigQueryTask

from

prefect.tasks.gcp

. I want to be able to pass as

Parameter

value into this task but I am unable to do so. It appears the request being sent to BigQuery contains the following which generates a JSON error

{'value': <Parameter: min_date>}

The above value is not valid JSON so therefore explains why the Google API won't accept this request.

Sumant Agnihotri

05/23/2022, 4:18 AM

Hi all, a simple question. Can I set up my own Prefect Cloud system?

Raymond Yu

05/23/2022, 5:14 AM

Hey Prefect, we’re encountering a somewhat stochastic error when running a wait_for_flow_run
for a long running DatabricksSubmitRun in another flow even when the Databricks job runs to completion without an issue. We noticed this can occasionally result in the error enclosed below that causes no heartbeat to be detected. Has anyone encountered this? Any ideas on what may be causing this and how to address the issue?

Error during execution of task: ClientError([{'path': ['flow_run'], 'message': 'request to <http://hasura:3000/v1alpha1/graphql> failed, reason: read ECONNRESET', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://hasura:3000/v1alpha1/graphql> failed, reason: read ECONNRESET', 'type': 'system', 'errno': 'ECONNRESET', 'code': 'ECONNRESET'}}}])

Jacob Blanco

05/23/2022, 7:27 AM

I’m struggling to get DFE to kick in for a mapped task, we are using the Dask executor and version 0.15. The flow is really simple List->A->B where A and B map.

Sander

05/23/2022, 7:31 AM

Hi, do you know when bitbucket storage becomes available in 2.0?

Vadym Dytyniak

05/23/2022, 8:29 AM

Hi. We are getting the following issue in Prefect Cloud:

Vadym Dytyniak

05/23/2022, 8:29 AM

Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("Can\'t get attribute \'_make_function\' on <module \'cloudpickle.cloudpickle\' from \'/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py\'>")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - cloudpickle: (flow built with \'2.1.0\', currently running with \'2.0.0\')')

Vadym Dytyniak

05/23/2022, 8:30 AM

prefect[aws]==1.2.1

Vadym Dytyniak

05/23/2022, 8:35 AM

I believe we have to restart the agent

Valentin Baert

05/23/2022, 8:44 AM

With Prefect Orion is it on the roadmap to have a KV and Secrets store like with Prefect Core ? If yes, are you able to provide an ETA ?

Florian Guily

05/23/2022, 10:25 AM

Hey, i'm having a hard time to understand how to provide a branch name to the github task. The doc is mentionning the

ref

parameter but it is refering to a "SHA-1 value, tag, or branch name". Where can i find this SHA1 value of a given branchname ?

Todd de Quincey

05/23/2022, 10:28 AM

Prefect Tasks library in 2.0

Olivér Atanaszov

05/23/2022, 11:19 AM

Hi! I have a flow in which I run a bunch of queries. I'd like to establish connection to the db only once, but I'm not sure how to do that in Prefect. If I use a task I cannot just put the db object in the context (as per https://docs.prefect.io/api/latest/utilities/context.html#context it is not recommended to mutate context in tasks and even downstream tasks do not see it). I cannot pass around the db connection as it is not serializable. Is there a recommended way to achieve this?

Ilhom Hayot o'g'li

05/23/2022, 12:08 PM

HI! I wanted to run 2 or more flows independently for ETL processes with cron. What are the best practices? Is there some way of doing so ?

Matthew Seligson

05/23/2022, 12:46 PM

If I have a StartFlowRun task with wait=True in my flow, and the subflow fails, what is the expected behavior when I restart the parent flow? Should the StartFlowRun schedule a brand new subflow run? Should the subflow restart all failed tasks? Is this documented anywhere?

ale

05/23/2022, 1:23 PM

Hey folks 👋 I tried mapping over the entries of a

dict

, but got the following error:

At least one upstream state has an unmappable result.

Looking at the docs, it seems that

map

should work with an

Iterable

. Afaik,

dict

is an

Iterable

in Python, so I’m a bit confused 😅 I’m using Prefect

0.15.16

Mateo Merlo

05/23/2022, 2:11 PM

Hi!! Anybody has issues using credentials in DbtShellTask in Prefect Cloud? I'm working with BigQuery. Do I need to pass the "keyfile" param inside dbt_kwargs? Or if I have a GCP_CREDENTIALS Secret this is not need?

Joshua Greenhalgh

05/23/2022, 2:36 PM

Hi is there a way to tell if my flow is running because it was triggered on a schedule as opposed to not? I tried this check;

try:
            scheduled_start_time: DateTime = prefect.context.scheduled_start_time
        except AttributeError:
            raise Exception("No start/end time params and no schedule")

Joshua Greenhalgh

05/23/2022, 2:36 PM

but even on a non scheduled run this attribute exists

Jessica Smith

05/23/2022, 3:07 PM

I'm already running the heartbeat process in threads, and still having some issue with ZombieKiller - would changing heartbeat_interval to a longer interval help with this?

Jonathan Mathews

05/23/2022, 3:31 PM

Hi! I want to debug a flow-of-flows on my local machine, but the flow runs created using create_flow_run seem to be executing on my production Prefect Cloud environment. How do I ensure that these flow runs are all executed on my local machine?

Title

Jonathan Mathews

05/23/2022, 3:31 PM

Kevin Kho

05/23/2022, 3:32 PM

You can’t really because the task under the hood is an API call. Are you trying to unit test or you don’t want to use Cloud?

Jonathan Mathews

05/23/2022, 3:35 PM

ok thanks. I’m just trying to test in development locally before pushing to production

So maybe there’s a better way of setting that up

Chris L.

05/23/2022, 4:30 PM

My team and I are testing local flow of flows by wrapping the flow around a small task: @task def create_local_flow(flow, params=None): state = flow.run(params=params) return state This can be used just like create_flow_run provided by Prefect library.

Really simple solution and works well

Jonathan Mathews

05/23/2022, 4:52 PM

Great tip! Thanks Chris

🙌 1

View count: 20