prefect-community #prefect-community

lialzm

12/17/2022, 9:51 AM

hi team how to cancel host label when register flow? I didn't find the answer in the docs

1️⃣ 1

✅ 1

Tim-Oliver

12/17/2022, 3:28 PM

Hi, I am having trouble using the map functionality. When I use map not all tasks are being scheduled and after completing the scheduled tasks the flow hangs in a running state.

Florian Kühnlenz

12/17/2022, 7:53 PM

We are seeing quite a lot intermittent of API errors and consequent flow run failures in cloud v1. Anyone else?

Fady Khallaf

12/18/2022, 11:20 AM

Hello, is prefect compatible with arm architecture? if anyone tried prefect on arm arch do you recommend it for production level

Andrei Tulbure

12/18/2022, 2:14 PM

Hi. I have a quick question: We recently migrated from Prefect1 to Prefect2 and mostly everything went smooth. A single problem I have: we had some Flows running easily in parallel on ECS on Prefect1 and right now, if we try to run them in parallel on ECS in Prefect 2 they are not launching in parallel. We do not use Dask, just simply launch them with the task call in a for loop. Could you provide us any resources where we could find how to launch them in parallel on ECS (yes, we have enough of a vCPU quota on AWS)

Agnieszka

12/18/2022, 3:19 PM

Hello, What would be the best way to run external docker containers as prefect tasks? I have a pipeline/flow that is supposed to run a few "external" docker images on some data (sequentially, where one's output (or part of it) is used as input for the next task). The docker images contain python scripts/modules that are not integrated with prefect. I would like to be able to pull those images from the private registry, and run containers based on images with custom parameters (based on settings defined by the user , that I pass to my flows). The pipeline is meant to be used by users with no prefect knowledge who can just git pull, install everything with pip and run it providing some custom parameters. That's why I would prefer not to go for orion infrastructure blocks and deployment but rather hide all prefect magic under python CLI. The naive approach I was thinking about is to have a custom docker executor class/module (running containers) that methods will be called by prefect tasks, but maybe there is a better way to make it work? I am completely new to prefect, so any advice is much appreciated.

✅ 1

docker ship 1

Kelvin DeCosta

12/19/2022, 8:54 AM

Quick question: Is there a way to programmatically create / update a deployment, say via

.build_from_flow

, but keep it disabled / turned off, similar to how we can turn them off via the UI?

✅ 1

Tobias

12/19/2022, 10:44 AM

Has anyone experienced they can’t delete a flow/deployment? 🧵*→*

✅ 1

Nic

12/19/2022, 10:45 AM

Has anybody had success with running prefect agent locally as a background process on a windows machine like you can do with supervisor on a linux system? I've found this thread https://discourse.prefect.io/t/how-to-run-your-prefect-agent-as-a-background-process-with-supervisord/1551. but seems like supervisor-win might be broken currently. How else could i go about this?

✅ 1

Patrick Tan

12/19/2022, 2:44 PM

Hi, we have been receiving prefect.agent - Invalid input issue. ANy experience with this?

✅ 1

James Zhang

12/19/2022, 4:12 PM

hey guys, has anyone seen this kind of strange Numba error before? I was trying to train a Top2Vec model, which uses Umap and Numba at the lower level, what strange is, starting from certain amount of training data, the training throws this error

Copy code

...
File "/opt/prefect/workflow/tasks/model_train.py", line 64, in train_model
    model = Top2Vec(
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/top2vec/Top2Vec.py", line 668, in __init__
    umap_model = umap.UMAP(**umap_args).fit(self.document_vectors)
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/umap/umap_.py", line 2516, in fit
    ) = nearest_neighbors(
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/umap/umap_.py", line 328, in nearest_neighbors
    knn_search_index = NNDescent(
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/pynndescent/pynndescent_.py", line 920, in __init__
    self._neighbor_graph = nn_descent(
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/opt/conda/envs/prefect/lib/python3.9/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
[1m[1m[1m[1mFailed in nopython mode pipeline (step: nopython frontend)
[1mUntyped global name 'print':[0m [1m[1mCannot determine Numba type of <class 'function'>[0m
[1m
File "../conda/envs/prefect/lib/python3.9/site-packages/pynndescent/pynndescent_.py", line 252:[0m
[1mdef nn_descent_internal_low_memory_parallel(
    <source elided>
        if verbose:
[1m            print("\t", n + 1, " / ", n_iters)
[0m            [1m^[0m[0m
[0m
[0m[1mDuring: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x7fe7ea5adee0>))[0m
[0m[1mDuring: typing of call at /opt/conda/envs/prefect/lib/python3.9/site-packages/pynndescent/pynndescent_.py (358)
[0m
[0m[1mDuring: resolving callee type: type(CPUDispatcher(<function nn_descent_internal_low_memory_parallel at 0x7fe7ea5adee0>))[0m
[0m[1mDuring: typing of call at /opt/conda/envs/prefect/lib/python3.9/site-packages/pynndescent/pynndescent_.py (358)
[0m
[1m
File "../conda/envs/prefect/lib/python3.9/site-packages/pynndescent/pynndescent_.py", line 358:[0m
[1mdef nn_descent(
    <source elided>
    if low_memory:
[1m        nn_descent_internal_low_memory_parallel(
[0m        [1m^[0m[0m

The whole thing runs perfectly fine on my local, the python environment is also the same, but on the Prefect KubernetesJob it has the problem… I’m sure I’ve given the job enough resources (CPU & RAM)… anything to do with the parallelism? I don’t know much about the Numba

CPUDispatcher

, can it be it’s not supported in a Prefect Task? being stuck on this for days… 😣

Michał Augoff

12/19/2022, 7:36 PM

hey, is it possible to have dynamic default date parameters in deployed flows? I know date parameters are rendered in the UI with a calendar, and I know you can use Optional[date] and dynamically assign e.g. today’s date inside the flow code. But I’m wondering if there’s a way to say “today is default” and have that reflected in the UI when I create a new run (instead of having None in the UI)

✅ 1

Greg Ott

12/19/2022, 8:13 PM

Hello all! Is there support/a timeline for support for Delta Live Table pipelines in Azure Databricks? I checked the prefect-databricks package, but it appears that the supported API calls are mainly for kicking off traditional databricks jobs, not DLT pipelines.

✅ 1

Edmund Tian

12/19/2022, 9:00 PM

Is it possible to trigger multiple concurrent Flow Runs for the same Flow? I’m getting this error when I attempt to do so:

Copy code

RuntimeError: The task runner is already started!

Context: I have an application hosted on GCP Cloud Run that allows multiple requests per container. Whenever my application receives an API request with a

user_id

param, it triggers a flow run to update data for that user. When I trigger the first Flow Run via an API request, everything works fine. But if I call the API again while the first Run is ongoing, my second API request with throw the above RuntimeError. More context: My application is written in Flask and served with Gunicorn. It’s deployed on GCP Cloud Run. Here’s the gunicorn config in my dockerfile

Copy code

CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 pelm_utility_service:app

✅ 1

Jason Noxon

12/19/2022, 9:45 PM

Hi, all! I installed Prefect 2 (no issues) and now I am trying to run

prefect cloud login

I enter my API key, and I get an Index out of Range exception. Has anyone seen this?

✅ 1

Jason Noxon

12/19/2022, 9:54 PM

Copy code

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.8/dist-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.8/dist-packages/prefect/cli/cloud.py", line 429, in login
    workspace = current_workspace or workspaces[0]
IndexError: list index out of range

✅ 1

Praveen Shilavantar

12/19/2022, 10:03 PM

Hi, is it posisble to add @flow to existing class methods?

class API:

@flow

def start(self):

pass

I am getting TypeError: missing a required argument: 'self' (edited)

✅ 1

puneet jindal

12/20/2022, 9:57 AM

how to start a flow run for deployment B from a flow run of deployment A and what is the rate limits there

Slackbot

12/20/2022, 12:35 PM

This message was deleted.

Michael Dymshits

12/20/2022, 12:51 PM

When I import function for

pytest

which has perfect

decorator

, I got the following error (screenshot bellow). If I comment #flow decorator, pytest runs without errors.

Michael Dymshits

12/20/2022, 12:52 PM

Hamza Kazmi

12/20/2022, 1:22 PM

Hello everyone, I am not sure if this is correct place to ask this question, but today suddenly all of our prefect flows started failing and raise following error

(401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Audit-Id': '', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 20 Dec 2022 13:00:03 GMT', 'Content-Length': '129'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

I have checked all keys and non of them is expired What could be other possible reason for this error ?

👀 1

✅ 1

Harry Hause

12/20/2022, 3:05 PM

Hey @Anna Geller , I’ve been following your repo here to move our current deployment from Prefect 1.0 on ECS to 2.0 running on K8s with CD. Thanks for taking the time to put that together. It’s been really helpful so far. One question I have is related to expanding the CD to multiple flows. We currently have multiple flows registered manually on Prefect 1.0 and if we could automate the deployment for all of them that would be great. Ideally we would like to keep all of our flows in 1 repo. Would we just add more steps to the github workflows for each flow?

Marcelo Santoro

12/20/2022, 3:13 PM

Hey Prefect community .... Does anyone knows if .... is there any way to rename the k8s prefect jobs ? because is really hard to identify what flow is running in each job 😕

👀 1

✅ 1

Vadym Dytyniak

12/20/2022, 3:14 PM

Hi. My question is related to prefect-dask collection. Depends on the image I use for my DaskCluster Prefect submits tasks to DaskTaskRunner with different behavior. My task does manipilations with dask.dataframe and in one case dask transformation is distributed on the same cluster, but in other case task is running on one worker and other just waiting. Does anyone has an idea what should be different to see such different behaviour? Thanks.

Missi Sogbohossou

12/20/2022, 3:49 PM

Good morning! I'm using Prefect 1.0. The logs have not been loading to the UI for a couple of days and I was wondering if there has been any changes.

Missi Sogbohossou

12/20/2022, 3:49 PM

👀 1

✅ 1

Madison Schott

12/20/2022, 5:09 PM

Hi all, getting this error: I created an exact copy of my current flow in production but gave it a new name and wrote it in a new Python file. I removed some of the syncs and other commands but not any of the fundamentals so not sure what could be causing this,

Copy code

/usr/local/lib/python3.9/site-packages/prefect/core/flow.py:1726: UserWarning: No result handler was specified on your Flow. Cloud features such as input caching and resuming task runs from failure may not work properly.

Edmund Tian

12/20/2022, 5:29 PM

Hi, I’m getting the following error during my Flow Run.

Copy code

OperationalError:
(sqlite3.OperationalError) no such table: flow_run_alert_policy
[SQL: ALTER TABLE flow_run_alert_policy RENAME TO flow_run_notification_policy]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

I’m using the

@flow

and

@task

decorators for my Python functions. I’m not using Prefect Cloud. I’m on the following prefect version

prefect==2.3.2

. It’s hard for me to repo because this occurs during my chron job, and it only occurs for a small subset of the Flow Runs. I haven’t been able to repo locally. Anyone know what’s going on? A quick Google brought me to this, but this doesn’t seem related as I am not deploying anything and I’m not using

orion

Patrick Tan

12/20/2022, 6:57 PM

Hi, I have a scenerio where agent not picking up flow run when work queue concurrency is set to a number( I use 5 and 25). However, if I change to unlimited it is picking up flow run and execute. I do not have any flow run that is in 'running' state