prefect-community #prefect-community

Hey, quick question on Prefect Orion - Wanted to know if the caching on task based on the persistent result something going to be supported in the Prefect Orion. The one that is already available in the Prefect 1.0 with @task(target=“{task-name}“)

Apoorva Desai

04/14/2022, 9:07 PM

Is there any documentation on which prefect docker tag has what extras?

Ken Nguyen

04/14/2022, 11:28 PM

Are there any example flows involving pulling from different branches of the same repo using mapping?

Alexander Butler

04/14/2022, 11:55 PM

prefect deployment create

says it should create or update a deployment but it is failing when the deployment exists (prefect 2.0) ?

Ryan R

04/15/2022, 12:11 AM

Hi, Having some problems using getting around CORS issues with Prefect V2.0 accessible to another host on the lan while running in a docker container. Details in thread per @Kevin Kho

Apoorva Desai

04/15/2022, 3:05 AM

Hello! All my prefect flows are getting scheduled but not running. Any recommendations on where I can start to debug this?

Subhajit Roy

04/15/2022, 4:42 AM

Hi folks, in a prefect flow , for one particular task, I am trying to raise a SKIP signal

Copy code

from prefect.engine.signals import SKIP
........
........
raise SKIP('Skipping all downstream dependencies.')

With this all the following tasks are being skipped and was expecting the flow state will be

skipped

. Though the following tasks are being skipped but at the end the flow becomes

successful

. I have two questions around : 1. Is this something expected? 2. If this is expected, whats the remedy . Do I need to explicitly use a state handler on top of it to make the flow

skipped

eventually

Andrey Vinogradov

04/15/2022, 9:14 AM

Hi! Is it possible to increase max scheduled flows in prefect? I found “max_scheduled_runs_per_flow” parameter but it seems that is is only available in prefect server not prefect.

Brett Naul

04/15/2022, 1:06 PM

we're still having issues with logs not loading, sounds like there was an incident yesterday but prefect.status.io says it should be resolved now? any known issues that are still being worked on?

Patrick Tan

04/15/2022, 2:09 PM

Question on state_handler, is there anyway to pass arguments to state_handler? I want to update database table when flow or task fails. The arguments are database config information and credentials

Pedro Machado

04/15/2022, 2:52 PM

Hi everyone. Is there a migration guide that discusses the changes between 1.0 and Orion? I have a couple of implementations on 1.0 that we'd like to migrate soon after Orion hits GA. One runs on Azure Kubernetes and the other one on AWS ECS. We are using Docker storage for both. I was wondering what you recommend to understand how the different concepts relate in both versions. Thanks!

👀 1

Domenico Di Gangi

04/15/2022, 3:09 PM

Hi all, I am wondering if its possible to change parameters of a deployed flow in the Orion UI and trigger a quick run with the new parameters. That would be extremely useful, for example in the process exploring different configurations and sizes when setting up third party cloud services

Ahmed Ezzat

04/15/2022, 5:29 PM

I'm having problems with tasks stuck at the "Running" state however when I inspected running workers everything seems running fine and results are submitted to my database although nothing is updating prefect state. currently using "prefect cloud" with

prefect 1.2.0-python3.9

docker image. same as https://github.com/PrefectHQ/prefect/issues/3952 for the dev team: https://cloud.prefect.io/bitthebyte/flow-run/b30223e1-5308-48fe-aa0b-9326c6e48860 (this is the stuck workflow) I already tried restarting

Melqui de Carvalho

04/15/2022, 6:46 PM

Hello guys, I would like to know if in Prefect 2.0 it is possible to request the execution of a specific task and all its dependent upstream tasks. Thanks.

Mohan kancherla

04/15/2022, 7:41 PM

Hello guys, we use VictorOps as our incident management tool and I setup cloudhook to the flow to send notifications to the Victorops as web URL. I am trying to test that out but it seems like, I am not receiving messages to our VictorOps. Does anyone has any idea about that?

sidravic

04/16/2022, 8:59 AM

Hello folks, I'm running flows on Prefect 1.2 using ECSRun and my setup uses the approach as described in this issue under the Conclusions pt. 5. It uses the

task_definition_arn

with the containers named as

flow

While I'm able to trigger the flows, the flow crashes with the error

Copy code

copilot/flow/8d31faa7f1ba   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
copilot/flow/8d31faa7f1ba     return _bootstrap._gcd_import(name[level:], package, level)
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
copilot/flow/8d31faa7f1ba   File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
copilot/flow/8d31faa7f1ba ModuleNotFoundError: No module named '/root/'

However, i've ensured the flows folder from my project are under the PYTHONPATH and I can't entirely figure out what (if anything) that cloudpickle is trying to do to access those flows at the time of execution.

Hash Lin

04/16/2022, 2:28 PM

Hi, I'm new to Prefect. It's fantastic to deploy pipelines by using Python code. However, I had faced some issues when deploying a pipeline to Prefect. Here's my scenario. • I want to run Prefect Agent in the K8s cluster, and I successfully plug the agent into our cluster. • I built a docker image in Gitlab CI and pushed the docker image to ECR. Then, I wrote the Dockerfile and reference from here. • I write a flow script and register to the Prefect cloud by using my local computer. Unfortunately, after running the pipeline, I got some errors: the python model was not found. As a result, I can't successfully import the module that I wrote in the project. And here's my flow script.

Copy code

Failed to load and execute flow run: ModuleNotFoundError("No module named '/Users/xxx/'")

Thanks for helping. 🙇

Blake

04/17/2022, 1:16 AM

Hiya! Quick question on best practices in Prefect 1. With a flow-of-flows, is there an ideal way to pass values down to child flows? In particular, secrets? Parameters seem fine for everything else.

MasatoShima

04/17/2022, 7:20 AM

Hi, I’m trying Prefect Orion. I am planning to use Amazon S3 bucket for Storage. I wrote the following code. However, I get an error. I have not created the Storage yet… Am I doing something wrong? I’d be glad to hear from you if anyone has any ideas!

Copy code

s3_storage_block = S3StorageBlock(
    bucket="********",
    profile_name="default",
    region_name="ap-northeast-1",
)

async with get_client() as client:
    block_id = await client.create_block(
        name="********",
        block=s3_storage_block,
        block_spec_id=uuid.UUID("{12345678-1234-5678-1234-567891234567}")
    )

Ken Nguyen

04/17/2022, 7:21 PM

If I have 2 parameters, one as a list to be mapped, and one as a value that should be unmapped, how can I concat the mapped parameters onto the unmapped one?

Alexander Butler

04/17/2022, 8:52 PM

So I have this deployment in Prefect 2.0:

Copy code

DeploymentSpec(
    flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
    flow_name="elt-salesforce",
    name="sf-production-elt-job",
    schedule=IntervalSchedule(interval=timedelta(hours=1)),
    tags=["pipeline"],
    flow_runner=DockerFlowRunner(image=f"{IMAGE_REPO}/{DBT_IMAGE}:{TAG}", stream_output=True)
)

And it took awhile to come to me as a requirement but I essentially have 2 steps. Step one requires docker image A to do some data pipeline stuff, step 2 needs my custom dbt docker image B to do some transform AFTER step 1. So these two dependent tasks constitute one flow with each step on independent docker images. A flow runner is configured at a deployment level but I dont see a way to configure it at the task or subflow level. Definitely a key req in current state. Please help!

👀 1

Ken Nguyen

04/17/2022, 10:17 PM

How do I get the value of the input of my parameters? I was doing something like this in my flow:

Copy code

test_param = Parameter('test_param', default="default_val")
function(test_param)

Where I got an AttributeError:

AttributeError: 'Parameter' object has no attribute

Stephen Lloyd

04/18/2022, 4:22 AM

Hi, I’m getting an error I don’t understand, along with unexpected behavior:

Leanna Morinishi

04/18/2022, 6:07 AM

Hello there! I have a task that most of the time I want to depend on a single upstream task. However, at the end of the flow, I want it to run if

any_successful

for several upstream tasks. When I write it like below,

task_5

doesn’t fail, even if

task_3

and

task_4

fail. Is it because the creation of the list

[task_3, task_4]

succeeds? How should I write this flow instead? Many thanks!

Copy code

task_3 = my_task(
    "input1", task_args=dict(name="input1")
    ).set_upstream(task_1)

task_4 = my_task(
    "input2", task_args=dict(name="input2")
    ).set_upstream(task_2)

task_5 = my_task("all_inputs",
    task_args=dict(name="all_inputs", trigger=any_successful),
    ).set_upstream([task_3, task_4])

Amir Shur

04/18/2022, 11:19 AM

Hi! There's a prefect feature I was looking for but couldn't quite found it and wondered if anyone knows how to achieve the functionality I'm looking for: I'm looking to pause a flow if a certain condition is met, and if so, I want to allow continuing the flow after entering new input file as a parameter. I know there's this: https://docs.prefect.io/core/idioms/pause-for-approval.html but couldn't find the option for more complicated input than approve/disapprove. Any help will be much appreciated, thx!

Matthew Seligson

04/18/2022, 12:44 PM

I have a flow scheduled to run on the second weekday of October, but I don’t see it in the “upcoming runs” or upcoming schedule of the flow in the UI. Is this expected?

Xavier Babu

04/18/2022, 3:45 PM

Dear Orion community, If I need to deploy a flow (createflow or deployflow) via Orion REST API, how can I pass the flow-location, so that it can pickup the python file from a location?

Jack Sundberg

04/18/2022, 4:42 PM

Hey everyone! Just like you can do class-based Tasks, are you able to do class-based workflows? With Prefect Orion introducing the

@flow

decorator, I'm guessing this will be possible, but don't see any docs on it.

Dylan

04/18/2022, 6:32 PM

Hey when I try to toggle on a schedule I get an error message in the UI back from GraphQL, this is on a cloud backend with a K8's agent running on EKS

Dylan

04/18/2022, 6:32 PM

Copy code

[{
    "data": {
        "set_schedule_active": {
            "success": false,
            "__typename": "success_payload"
        }
    }
}]