prefect-community #prefect-community

Hello, Following the announcement for Orion, If we create our workflows based on Prefect Core, won’t it be a pain to move it to Orion ? As for now we are going to Prefect Server and launch it on our infra

upvote 1

Joseph Oladokun

10/08/2021, 12:10 PM

Hello, I’m having this error, all the solutions seems outdated

Copy code

TypeError: cannot pickle '_thread.lock' object

Joseph Oladokun

10/08/2021, 12:10 PM

Any help will be appreciated

Zheng Xie

10/08/2021, 1:25 PM

the users of the workflow can upload a file from their own laptop or their own box folder, where the workflow has no prior knowledge where the files can be. How to handle this in Prefect?

Max Kureykin

10/08/2021, 1:31 PM

Hi all! Is there any way not to fail the entire flow that consists only of 1 map
task and some

mapped

runs are failed? Thx in advance)

Marwan Sarieddine

10/08/2021, 2:02 PM

Hi folks, I believe we are facing an issue with manual_only triggered tasks. Mainly the task seems to be nullifying the parameters passed to it

García Gimenez Ignacio - 6to 2da

10/08/2021, 3:10 PM

Hi all! Im having an issue trying to create a tenant, it throws the following exception: "HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url....". I'm running my own docker compose file. Apollo's server is running fine, it connects to the UI. But it seems like wont connect when trying to create the tenant with the command "prefect server create-tenant --name default --slug default". Is there any way to change the default ip "localhost" to a custom one?

Michael Pavese

10/08/2021, 5:23 PM

I am trying to run a flow that I registered with the CLI and is using Gitlab storage. I am getting the following error in my logs:

Michael Pavese

10/08/2021, 5:23 PM

Tadej Svetina

10/08/2021, 5:26 PM

I am trying to run a simple task, but I am getting

Unexpected error: TypeError("cannot pickle '_io.BufferedReader' object")

Michael Pavese

10/08/2021, 5:26 PM

Where do I need to install "gitlab extra". When I run this on my local machine, it's fine.

Matthew Seligson

10/08/2021, 6:21 PM

How can we use the GraphQL API to find which user manually restarted a flow run? Or separately, a task run?

Vincent

10/08/2021, 6:28 PM

Hi- I was wondering if there was a prefect cloud interruption today. My flows exited with 404 and Protocol Errors.

Copy code

Failed to send heartbeat with exception: HTTPError('404 Client Error: Not Found for url: <https://api.prefect.io/graphql>')
Traceback (most recent call last):
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/cli/heartbeat.py", line 68, in flow_run
    client.update_flow_run_heartbeat(id)
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1610, in update_flow_run_heartbeat
    self.graphql(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 548, in graphql
    result = <http://self.post|self.post>(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 451, in post
    response = self._request(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 737, in _request
    response = self._send_request(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 647, in _send_request
    response.raise_for_status()
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <https://api.prefect.io/graphql>

Chris Reuter

10/08/2021, 7:57 PM

Hi everybody 👋! The Prefect livestream where @Laura Lorenz will take Orion for a spin is live in 5 minutes! Sign up on Meetup to attend, or go straight to the

Youtube stream▾

. Hope to see you there!

🦜 2

🚀 4

📺 3

marvin 3

Matt Alhonte

10/08/2021, 10:58 PM

Hrm, so I'm running a big Flow of Flows - some of them should be kicked off in parallel I think, but they're all running one at a time.

张强

10/09/2021, 8:18 AM

👋 5

张强

10/09/2021, 2:43 PM

Is there a base class like BaseFlow and BaseTask to set default parameters, such as

max_retries

, or set default

state_handlers

张强

10/10/2021, 1:21 AM

Two undependent

map

tasks are not parallel。 For the code, see reply.

times

and

times2

are parallel, but then

sleep

and

sleep2

did not，Is there a limit to the implementation of

map

? There are application scenarios, such as a task operating different types of devices, each type of device has two serial tasks, but the number of devices is determined by parameters, such as (Of course, this is pure python serial.):

Copy code

ret = []
for device in params.get('device_type1_list'):
    result = operate_device_type1_step2(operate_device_type1_step1(device))
    ret.append(result)
for device in params.get('device_type2_list'):
    result = operate_device_type2_step2(operate_device_type2_step1(device))
    ret.append(result)
# do something for ret
handle(ret)

Here, each device can be parallel, but the

steps

are serial. So is there a solution to this situation?

Oliver Mannion

10/10/2021, 3:53 AM

Hiya, Orion looks very slick and clean! I was wondering: 1. v1 has the concept of projects to group flows, does Orion have something similar? 2. How are flows associated with specific prefect agents (ie: labels in v1)?

Toprak Nihat Deniz Öztürk

10/10/2021, 9:51 AM

Hello everbody, I need to divide dataframe(df) into dataframes -> load them into gpu and works on them. I can write a single task that does all but I prefer to keep them seperated and I already have a subflow that does the job. I wonder if I can call flow in a loop.

Tadej Svetina

10/10/2021, 11:55 AM

If anyone is trying to add BasicAuth to their server instance (through a reverse proxy such as nginx or traefik), here's how you can deal with some issues related to UI-server communication, and authentication of client https://prefect-community.slack.com/archives/C014Z8DPDSR/p1633866651298300?thread_ts=1632763954.014600&cid=C014Z8DPDSR https://prefect-community.slack.com/archives/C014Z8DPDSR/p1633866798298600?thread_ts=1632763954.014600&cid=C014Z8DPDSR

Fina Silva-Santisteban

10/11/2021, 1:48 AM

Hi everyone! I’m following the steps outlined in the docs to test a flow’s composition: https://docs.prefect.io/core/idioms/testing-flows.html#testing-flow-composition I’ve noticed that

flow.tasks

returns not just tasks but parameters as well! How would you go about checking those? I’ve tried the following without success:

Copy code

from prefect import Parameter
self.assertTrue(Parameter('parameter_name') in flow.tasks)
# returns False

Vamsi Reddy

10/11/2021, 3:52 PM

Hi Everyone!! I have been experimenting with prefect lately and wanted to know if there is anything similar to s3 sensors in prefect. I want to trigger my flow or make my tasks wait/run until there is a file/files being uploaded in the s3 buckets

Andy Waugh

10/11/2021, 4:17 PM

Hi there 👋 I’m playing with control flow logic using

case

and seeing something unexpected and I’m wondering if it is expected to you or if it is a bug. Equally if I’m doing something weird and there’s a better way, please let me know!

Hui Zheng

10/11/2021, 5:29 PM

Hello, Just want to report that this morning we had one flow scheduled run missing

pearl-anteater

. It is not triggered properly by prefect cloud for some reason. The remaining runs were fine. want to submit to your back-end investigation to avoid further issue.

Alex Rogozhnikov

10/11/2021, 7:10 PM

hello, some questions about orion (I am new so I decided to start with orion): • is there a way to run it on a fargate cluster? • is there a complete example of using own ssh cluster? • when I run things from jupyter, is this scenario considered supported? • if I change something in the package I develop - how do I make the same change delivered to agents/executors?

Tony Hill

10/11/2021, 8:21 PM

👋🏽

👋 8

Tadej Svetina

10/11/2021, 8:21 PM

Would you consider creating multi-arch docker images (specifically for ARM processors)? this would enable me to use Prefect on the latest gen AWS instances

Matt Alhonte

10/11/2021, 9:11 PM

#RubberDucking So, we're using Papermill for a lot of stuff. I think the code that runs in Papermill isn't actually being distributed with the Dask scheduler, and is basically just being treated as one function call. I guess the way to have the Papermill code get Dask-ified would be: Prefect flow starts in a lightweight container Prefect flow spins up a Dask cluster Modify Papermill notebooks so that they take an Address of a Dask cluster as an argument, and then submit the code that runs in them to the Cluster That sound right?

Shaoyi Zhang

10/11/2021, 11:08 PM

Hey team, is it possible to set up some automations that receives an email address and invite that email address to a prefect tenant? We are trying to build a self-service platform where stakeholders can request prefect accounts by themselves.