https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • d

    Danilo Drobac

    01/04/2023, 11:16 AM
    Hey folks - hopefully an easy question, just can't see it directly outlined anywhere. If I have a Deployment with default values in terms of how the Flow should be executed, does this mean that the execution will happen on the Agent?
    ✅ 1
    c
    • 2
    • 2
  • t

    Tibs

    01/04/2023, 1:57 PM
    Hey, I am using run_deployment to run multiple child flows from a parent flow. I want the parent flow to NOT wait a results of the child flows. Is this possible?
    a
    • 2
    • 2
  • b

    Benoît Linsey-Fazi

    01/04/2023, 2:01 PM
    Hello, I am currently trying to set a prefect orion server in a docker on an EC2, however this error appears on the UI
    Can't connect to Orion API at <http://0.0.0.0:80/api>. Check that it's accessible from your machine.
    Can't find any help on internet besides doing
    prefect config set PREFECT_API_URL=<http://0.0.0.0:80/api>
    Any ideas ?
    ✅ 1
    r
    a
    • 3
    • 13
  • t

    Tim-Oliver

    01/04/2023, 2:08 PM
    I have changed some things in my infrastructure block and now my flow runs crash with a 403 error:
    Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/uuid/workspaces/uuid/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
    For more information check: <https://httpstatuses.com/403>
    uuid contains my respective identifiers. I am a bit confused, because I cant find any block with the id
    a8b20987-a71e-4b43-9350-53a9e0d7d467
    . Which is apparently what it want to access. Any tips?
    n
    • 2
    • 36
  • a

    Aleksandr Liadov

    01/04/2023, 4:31 PM
    Hello guys, I have a problem to connect to my existing dask cluster
    my_flow.with_options(
            task_runner=DaskTaskRunner(
                address="<tls://10.4.83.2:8786>"
            )
    Log:
    prefect.task_runner.dask - Connecting to an existing Dask cluster at <tls://10.4.83.2:8786>
    17:20:30.497 | ERROR   | Flow run 'jolly-griffin' - Crash detected! Execution was interrupted by an unexpected exception: TypeError: TLS expects a `ssl_context` argument of type ssl.SSLContext (perhaps check your TLS configuration?) Instead got None
    What do I wrong?
    m
    • 2
    • 2
  • k

    Khuyen Tran

    01/04/2023, 4:33 PM
    In Prefect Live at 3PM Eastern today on Twitch and YouTube, @Ryan Peden, the senior developer experience at Prefect, will talk about how to use cloud events to trigger Prefect automations. Add this event to your calendar so you don’t miss it.
    🙌 1
    :party-parrot: 1
    :cloud2: 2
  • b

    Boris Tseytlin

    01/04/2023, 4:53 PM
    Did anyone have this problem where the python client hangs on trying to set up deployment?
    deployment = Deployment.build_from_flow(
        flow=load_data_from_minio,
        name="load_data_from_minio",
        work_queue_name="default",
        schedule=schedule,
        storage=storage_block,
        infrastructure=infra_block,
    )
    
    if __name__ == "__main__":
        logging.basicConfig(level=logging.DEBUG)
        deployment.apply()
    Running this code just seems to hang forever. No debug logs either. Full code in thread
    m
    j
    j
    • 4
    • 22
  • b

    Billy McMonagle

    01/04/2023, 6:49 PM
    Hi there, I wanted to check my understanding of feature availability in Prefect Cloud 2.0. Is it possible to send, for specific flow state changes, custom Slack notifications? Ideally using the Slack block kit rather than plain text. Thanks!
    ✅ 1
    k
    a
    • 3
    • 6
  • k

    Kevin Grismore

    01/04/2023, 8:38 PM
    it's because the path is a WindowsPath, which has
    \
    instead of
    /
    .
    c
    • 2
    • 7
  • m

    Michael Adkins

    01/04/2023, 8:53 PM
    Hi! I’ve raised this with our UI team and we are investigating.
    ❤️ 1
    :thank-you: 1
    j
    t
    • 3
    • 16
  • a

    Anders Segerberg

    01/04/2023, 10:19 PM
    Hi. I'm on Prefect 1. What is the simplest way to set a task to run as the terminal task of a flow? One could add requisite
    upstream_tasks
    but that's laborious when we want the behavior to be independent of the upstream, as long as it is serially the final task execution.
    m
    • 2
    • 2
  • k

    Kevin Weiler

    01/04/2023, 10:22 PM
    Hi there! I’m looking at transitioning to Prefect 2.0 and trying to wrap my head around some of the new stuff. In my existing workflow, I construct a flow programmatically by walking a set of yaml files. This has the effect of needing the
    add_task
    method from the
    flow
    class. I don’t think the new API has this. Is there a way to iteratively add tasks to a flow?
    m
    • 2
    • 5
  • g

    Gabriela Palacios

    01/05/2023, 4:27 AM
    hi everyone! Probably this is straightforward, I'm trying to define a waiter using prefect_aws.client_waiter but I'm getting: Missing required parameter in input: "jobs" Any help would be great!! Traceback in the thread.
    a
    • 2
    • 3
  • h

    Hamza Naanani

    01/05/2023, 8:30 AM
    Hello, How can I delete a single run of a flow from the history in prefect 1.4 ? Thanks in advance
    ✅ 1
    b
    • 2
    • 2
  • d

    Dmytro Ponomarchuk

    01/05/2023, 9:50 AM
    Hi Torstain, You would need to add the
    .submit()
    function to call the async task. According to https://docs.prefect.io/tutorials/execution/#asynchronous-execution
    t
    n
    • 3
    • 3
  • m

    Marion Sauvage

    01/05/2023, 10:17 AM
    Hello , I am trying to set up a prefect Orion server. I am having trouble creating a work queue. I had the error :
    Error occured when trying to create new work queue
    in a 405 response for a request to
    <http://xx.xx.xx.xx:4200/xx.xx.xx.xx:4200/api/work_queues/>
    We use this Dockerfile to run the server:
    FROM python:3.9.16
    
    RUN pip install prefect prefect[aws]
    RUN prefect config set PREFECT_API_URL=http:/xx.xx.xx.xx:4200/api
    CMD ["prefect", "orion", "start", "--host", "0.0.0.0"]
    Thanks a lot for your help !
    ✅ 1
    d
    a
    b
    • 4
    • 9
  • a

    Aniruddha Bharadwaj

    01/05/2023, 10:34 AM
    Hello, Would it be possible to read all files in a s3 sub-folder ?
  • d

    Danilo Drobac

    01/05/2023, 3:45 PM
    Hi folks - I've just written a fairly long blog post introducing how to use Prefect with a real-world example (part of a series I'm creating). Would anybody that's familiar with Prefect do me a solid and proof-read it for me so I can make sure everything I've said is factual? I would be forever in your debt...
    🙌 2
    b
    j
    • 3
    • 3
  • j

    Jean-Michel Provencher

    01/05/2023, 4:11 PM
    https://source.coveo.com/2023/01/04/creating-custom-sentry-block-prefect-2/ Here's my latest blog post on how to integrate Sentry with Prefect 2 using a custom block. Enjoy
    👍 4
    :gratitude-thank-you: 4
    🙌 4
    a
    • 2
    • 4
  • t

    Tim-Oliver

    01/05/2023, 4:36 PM
    Hi, With result persistence on, is it possible to trigger a task-rerun if the persisted result got deleted or corrupted? I am thinking about a case where the cache-key for the orion-DB is available, but the result can't be retrieved from local storage. In such a case I would like to invalidate the cache-key and trigger a new task-run to obtain the missing/corrupted result.
    m
    • 2
    • 10
  • t

    Thet Naing

    01/05/2023, 5:08 PM
    Hi all!
  • t

    Thet Naing

    01/05/2023, 5:08 PM
    Hi all! Does anyone know how to get the Prefect CLI integrated with shell to allow for autocomplete?
  • a

    Anders Segerberg

    01/05/2023, 5:46 PM
    Prefect 1.0 If have a "flow of flows", which calls
    wait_for_flow_run
    several times. I want this parent flow's success to depend on the success of all child flows. I've tried setting
    upstream_tasks
    to the result of
    wait_for_flow_run
    , but I realize that that's a
    FlowRunView
    object, not a
    Task
    . From the child flow, I can
    get_tasks
    , and set
    upstream_tasks
    to that. But I have some tasks I allow to fail in the child flows (they are not reference tasks of the child flow.) What I'd really like to do is to be able to reference
    <child_flow>.state
    , and set the parent flow's reference tasks to be expecting a
    <Success: "All reference tasks succeeded.">
    Is there a way of going about this?
    k
    • 2
    • 2
  • t

    Tuoyi Zhao

    01/05/2023, 6:33 PM
    I opened a thread here: https://discourse.prefect.io/t/prefect-2-7-5-2-7-6-flow-stuck-at-pending-forever/2149
  • t

    Trevor Kramer

    01/05/2023, 7:09 PM
    Prefect 2.0.
    ✅ 1
    m
    • 2
    • 2
  • e

    Elliott Wilson

    01/05/2023, 7:42 PM
    Hey team! I am new to prefect and I using: • EC2 (ubuntu) • S3 storage block • I am build a flow that takes data from mixpanel and dumps into an s3 bucket. I get the following error (trace in thread)
    Flow could not be retrieved from deployment.
    from boto3 when I try and run the deployment into the ec2 instance. I can deploy to s3 locally and connect to bucket from the ec2 using the AWS cli. Please can anyone help me debug this?
    m
    • 2
    • 35
  • t

    Trevor Kramer

    01/05/2023, 7:58 PM
    Prefect 2.0 Does Prefect 2.0 support artifacts like 1.0 does? I want to attach some text or a link the a flow as a result. Is there a way to do this?
    k
    • 2
    • 2
  • t

    Tuoyi Zhao

    01/05/2023, 8:35 PM
    Anyone able to assist me one that:
  • t

    Tuoyi Zhao

    01/05/2023, 8:37 PM
    Anyone able to assist me on this: after I updated to 2.7.6, all my scheduled flow is not able to run by the local agent.....they will show as Pending forever. I've posted my issue here as well: https://discourse.prefect.io/t/prefect-2-7-5-2-7-6-flow-stuck-at-pending-forever/2149
    ✅ 1
    m
    • 2
    • 37
  • j

    Jean-Michel Provencher

    01/05/2023, 9:38 PM
    Hi, anyone has a link to documentation on best-practices regarding passing secrets as Task Input in Prefect 2 ? I would ideally like that my secrets never get sent in clear text to Prefect Cloud also.
    n
    m
    m
    • 4
    • 14
Powered by Linen
Title
j

Jean-Michel Provencher

01/05/2023, 9:38 PM
Hi, anyone has a link to documentation on best-practices regarding passing secrets as Task Input in Prefect 2 ? I would ideally like that my secrets never get sent in clear text to Prefect Cloud also.
n

Nate

01/05/2023, 9:51 PM
Hi @Jean-Michel Provencher One way to do it would be to pass the name of the secret you need to the task (or flow) that needs it and then just call
Secret.load(my_secret_name)
wherever you need to actually use it
j

Jean-Michel Provencher

01/05/2023, 9:52 PM
yeah, but I'm already using a from prefect_aws.secrets_manager import read_secret task so I was wondering how to use that built-in task to actually pass the secret to another task.
👀 1
n

Nate

01/05/2023, 10:29 PM
one way (which is not the slickest, but should work) is to just wrap the output of
read_secret
with pydantic's
SecretStr
and then get the secret value inside the task To help me ask a better question to our UI folks, where are you concerned with seeing plain text secrets in the UI?
j

Jean-Michel Provencher

01/05/2023, 10:30 PM
I'm not concerned necessarly, it's just for compliance reason I'd prefer my secrets to never be sent to prefect cloud. If I use pydantic will they be sent in clear to prefect cloud ?
n

Nate

01/05/2023, 10:32 PM
gotcha - I was just asking in case you had noticed them in the UI somewhere (they should not be exposed in the UI in the first place) but to be extra sure, you could wrap your input in
SecretStr
, because then as far as Prefect Cloud is concerned, it would only know that you passed a
SecretStr
with some
name
(or whatever the input arg is called) as a Task Input
j

Jean-Michel Provencher

01/05/2023, 10:38 PM
Interesting.
m

Mike O'Connor

01/06/2023, 12:51 PM
It would be good to have additional guidance here, we only just ran into this as well and would like to have control over preventing sensitive information being stored in prefect. Is there a document somewhere describing exactly what data is stored in Prefect? For us, being able to disable task inputs being logged, or mask based on field names, would be really great. (CC @Emil Christensen)
j

Jean-Michel Provencher

01/06/2023, 1:23 PM
I'm also wondering about the underlying mecanisms of task input and be sure that if I pass a SecretStr as a task input, it would would not send the actual task input value to prefect cloud, and instead only use my storage for that.
m

Michael Adkins

01/06/2023, 10:49 PM
Hi! At this time, values passed to tasks are never sent to the Prefect API.
Only flow parameters are stored in the Prefect API.
Similarly, we never store return values in the API (except for null/bools). See https://docs.prefect.io/concepts/results/#storage-of-results-in-prefect for more details on that.
The “Task Inputs” field is used to track relationships between task runs right now. For example, if you pass the result of one task run into another task run we will track the relationship. However, the value itself is only tracked locally. The API has scaffolding to track other types of inputs there, i.e. actual values. However, we have not built any support for that into the client and if/when we do so it is very likely to be opt-in.
:thank-you: 1
j

Jean-Michel Provencher

01/09/2023, 4:24 PM
@tsteijvers
n

Nate

01/09/2023, 4:27 PM
Hi @Mike O'Connor - we agree, and we're currently working on a better way to represent the purpose of the task inputs page you can track here
🙌 1
View count: 3