prefect-community #prefect-community

Channels

prefect-community

Leon Kozlowski

10/07/2021, 8:00 PM

In order to integrate slack (through the

state_handler

) with prefect cloud, does the webhook URL have to be stored in Prefect Secrets, or can it just be available as an environment variable (my use case would be loading an eternal secret to an eks cluster)

7 replies · 2 participants

Constantino Schillebeeckx

10/07/2021, 8:15 PM

I'm trying to send out notifications when a flow fails by defining a callback setting the flow's

on_failure

to it. The callback has (according to the docs) the following signature

def f(obj: Union[Task, Flow], state: State) -> None:

My quest is this: how to identify the failed task from the (Flow) obj. I see I can iterate over the flows' tasks, I'm just not sure how to identify the task's state

3 replies · 2 participants

kevin

10/08/2021, 12:52 AM

Hey guys, I'm just curious if there's been any changes recently to authenticating and POSTing to Prefect Cloud? I've recently been getting some AuthenticationErrors indicating Forbidden and haven't recently changed auth tokens

2 replies · 2 participants

Klemen Strojan

10/08/2021, 6:06 AM

Hello Community! We are getting this error quite often in the last weeks:

KeyError: 'The secret <some_secret> was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'

Now this happens in a mapped task, only for one or two of those mapped tasks, others work fine. The secret exists and is clearly usable. Re-running the flow usually helps, however we are not enjoying that re-running part so much. Any advice?

3 replies · 4 participants

Marko Herkaliuk

10/08/2021, 8:15 AM

Hello, at 17:22, Oct 7 2021 (UTC-5) we have

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <https://api.prefect.io/>

when flow try create other flow. Was it a common problem or a problem on our side?

6 replies · 2 participants

Aldo M

10/08/2021, 9:26 AM

Hello, Following the announcement for Orion, If we create our workflows based on Prefect Core, won’t it be a pain to move it to Orion ? As for now we are going to Prefect Server and launch it on our infra

:upvote: 1

3 replies · 3 participants

Joseph Oladokun

10/08/2021, 12:10 PM

Hello, I’m having this error, all the solutions seems outdated

TypeError: cannot pickle '_thread.lock' object

6 replies · 3 participants

Joseph Oladokun

10/08/2021, 12:10 PM

Any help will be appreciated

Zheng Xie

10/08/2021, 1:25 PM

the users of the workflow can upload a file from their own laptop or their own box folder, where the workflow has no prior knowledge where the files can be. How to handle this in Prefect?

3 replies · 2 participants

Max Kureykin

10/08/2021, 1:31 PM

Hi all! Is there any way not to fail the entire flow that consists only of 1 map
task and some

mapped

runs are failed? Thx in advance)

5 replies · 2 participants

Marwan Sarieddine

10/08/2021, 2:02 PM

Hi folks, I believe we are facing an issue with manual_only triggered tasks. Mainly the task seems to be nullifying the parameters passed to it

17 replies · 2 participants

García Gimenez Ignacio - 6to 2da

10/08/2021, 3:10 PM

Hi all! Im having an issue trying to create a tenant, it throws the following exception: "HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url....". I'm running my own docker compose file. Apollo's server is running fine, it connects to the UI. But it seems like wont connect when trying to create the tenant with the command "prefect server create-tenant --name default --slug default". Is there any way to change the default ip "localhost" to a custom one?

19 replies · 3 participants

Michael Pavese

10/08/2021, 5:23 PM

I am trying to run a flow that I registered with the CLI and is using Gitlab storage. I am getting the following error in my logs:

2 replies · 2 participants

Michael Pavese

10/08/2021, 5:23 PM

Tadej Svetina

10/08/2021, 5:26 PM

I am trying to run a simple task, but I am getting

Unexpected error: TypeError("cannot pickle '_io.BufferedReader' object")

11 replies · 3 participants

Michael Pavese

10/08/2021, 5:26 PM

Where do I need to install "gitlab extra". When I run this on my local machine, it's fine.

2 replies · 2 participants

Matthew Seligson

10/08/2021, 6:21 PM

How can we use the GraphQL API to find which user manually restarted a flow run? Or separately, a task run?

26 replies · 2 participants

Vincent

10/08/2021, 6:28 PM

Hi- I was wondering if there was a prefect cloud interruption today. My flows exited with 404 and Protocol Errors.

Failed to send heartbeat with exception: HTTPError('404 Client Error: Not Found for url: <https://api.prefect.io/graphql>')
Traceback (most recent call last):
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/cli/heartbeat.py", line 68, in flow_run
    client.update_flow_run_heartbeat(id)
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1610, in update_flow_run_heartbeat
    self.graphql(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 548, in graphql
    result = <http://self.post|self.post>(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 451, in post
    response = self._request(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 737, in _request
    response = self._send_request(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 647, in _send_request
    response.raise_for_status()
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <https://api.prefect.io/graphql>

10 replies · 2 participants

Chris Reuter

10/08/2021, 7:57 PM

Hi everybody 👋! The Prefect livestream where @Laura Lorenz will take Orion for a spin is live in 5 minutes! Sign up on Meetup to attend, or go straight to the

Youtube stream▾

. Hope to see you there!

:party-parrot: 2

🚀 4

📺 3

:marvin: 3

5 replies · 4 participants

Matt Alhonte

10/08/2021, 10:58 PM

Hrm, so I'm running a big Flow of Flows - some of them should be kicked off in parallel I think, but they're all running one at a time.

11 replies · 2 participants

张强

10/09/2021, 8:18 AM

👋 5

2 replies · 3 participants

张强

10/09/2021, 2:43 PM

Is there a base class like BaseFlow and BaseTask to set default parameters, such as

max_retries

, or set default

state_handlers

6 replies · 3 participants

张强

10/10/2021, 1:21 AM

Two undependent

map

tasks are not parallel。 For the code, see reply.

times

and

times2

are parallel, but then

sleep

and

sleep2

did not，Is there a limit to the implementation of

map

? There are application scenarios, such as a task operating different types of devices, each type of device has two serial tasks, but the number of devices is determined by parameters, such as (Of course, this is pure python serial.):

ret = []
for device in params.get('device_type1_list'):
    result = operate_device_type1_step2(operate_device_type1_step1(device))
    ret.append(result)
for device in params.get('device_type2_list'):
    result = operate_device_type2_step2(operate_device_type2_step1(device))
    ret.append(result)
# do something for ret
handle(ret)

Here, each device can be parallel, but the

steps

are serial. So is there a solution to this situation?

7 replies · 2 participants

Oliver Mannion

10/10/2021, 3:53 AM

Hiya, Orion looks very slick and clean! I was wondering: 1. v1 has the concept of projects to group flows, does Orion have something similar? 2. How are flows associated with specific prefect agents (ie: labels in v1)?

3 replies · 3 participants

Toprak Nihat Deniz Öztürk

10/10/2021, 9:51 AM

Hello everbody, I need to divide dataframe(df) into dataframes -> load them into gpu and works on them. I can write a single task that does all but I prefer to keep them seperated and I already have a subflow that does the job. I wonder if I can call flow in a loop.

2 replies · 2 participants

Tadej Svetina

10/10/2021, 11:55 AM

If anyone is trying to add BasicAuth to their server instance (through a reverse proxy such as nginx or traefik), here's how you can deal with some issues related to UI-server communication, and authentication of client https://prefect-community.slack.com/archives/C014Z8DPDSR/p1633866651298300?thread_ts=1632763954.014600&cid=C014Z8DPDSR https://prefect-community.slack.com/archives/C014Z8DPDSR/p1633866798298600?thread_ts=1632763954.014600&cid=C014Z8DPDSR

1 reply · 2 participants

Fina Silva-Santisteban

10/11/2021, 1:48 AM

Hi everyone! I’m following the steps outlined in the docs to test a flow’s composition: https://docs.prefect.io/core/idioms/testing-flows.html#testing-flow-composition I’ve noticed that

flow.tasks

returns not just tasks but parameters as well! How would you go about checking those? I’ve tried the following without success:

from prefect import Parameter
self.assertTrue(Parameter('parameter_name') in flow.tasks)
# returns False

15 replies · 4 participants

Vamsi Reddy

10/11/2021, 3:52 PM

Hi Everyone!! I have been experimenting with prefect lately and wanted to know if there is anything similar to s3 sensors in prefect. I want to trigger my flow or make my tasks wait/run until there is a file/files being uploaded in the s3 buckets

5 replies · 4 participants

Andy Waugh

10/11/2021, 4:17 PM

Hi there 👋 I’m playing with control flow logic using

case

and seeing something unexpected and I’m wondering if it is expected to you or if it is a bug. Equally if I’m doing something weird and there’s a better way, please let me know!

6 replies · 3 participants

Hui Zheng

10/11/2021, 5:29 PM

Hello, Just want to report that this morning we had one flow scheduled run missing

pearl-anteater

. It is not triggered properly by prefect cloud for some reason. The remaining runs were fine. want to submit to your back-end investigation to avoid further issue.

2 replies · 2 participants

Title

Hui Zheng

10/11/2021, 5:29 PM

Hello, Just want to report that this morning we had one flow scheduled run missing

pearl-anteater

. It is not triggered properly by prefect cloud for some reason. The remaining runs were fine. want to submit to your back-end investigation to avoid further issue.

https://cloud.prefect.io/semios/flow-run/1d732c22-5ea8-4660-8484-d3b41df58991

nicholas

10/11/2021, 5:30 PM

Hi @Hui Zheng - thanks for letting us know; I’ll let the team know so we can investigate and will report back with any findings.

View count: 3