https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Antonio Manuel BR

    02/15/2022, 7:05 AM
    Hello, a dumb question, I am aware that prefect Parameters are used to define flow inputs. What would happen if I use python variables that were defined out of the flow instead of define Parameters and adding the variables in flow.run?
    c
    a
    • 3
    • 5
  • j

    Jean-Baptiste Six

    02/15/2022, 9:28 AM
    Hey 🙂 I tried to implement a subflow in my main flow (like this) :
    @task
    def subtask():
       return 1
    
    with Flow("subflow") as subflow:
       subtask()
    
    @task
    def main_task():
       subflow.run()
    
    with Flow("main_flow") as main_flow:
       main_task()
    But I faced this Error : _Unexpected error while running flow: KeyError('Task slug init_dirs-1 is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.\n- Did you change the flow without re-registering it?\n- Did you register the flow without updating it in your storage location (if applicable)?')_ I precise that I had already register the _main_flow_ (without the subflow inside) and it worked, but then I updated it and I registered the subflow, and it failed, could you help me please ? 🙏 The task "_init_dirs_" is in the main_flow, finishes in a success state, but this error append when subflow.run() is called (and init_dirs is not in the subflow)
    a
    • 2
    • 29
  • m

    Martin Teller

    02/15/2022, 10:37 AM
    Hi 🙂, I am scoring the documentation but haven’t found the answer yet: on prefect-cloud (in particular) is it possible to run a flow when it receives an http request? What would this look like in prefect?
    a
    • 2
    • 2
  • f

    Florian Kühnlenz

    02/15/2022, 1:11 PM
    Hi, I am trying to run a flow given its name and project. Since the client can only do this given an ID I was trying to find the ID via graphql. However I fail to construct a proper query to only find the active version of the flow.
    {
      flow_group(
        where: {flows: {name: {_ilike: "%name%"}, archived: {_eq:false}}}  ) {
        id
        name
        flows{
          name
          id
          archived
        }
      }
    }
    Will not actually filter for not archived versions. What am I doing wrong?
    a
    • 2
    • 4
  • s

    Simon Stusak

    02/15/2022, 4:15 PM
    Hi there, I am trying to get more information about how a Kubernetes Agent in prefect cloud is configured. The setup was done by a colleague and is working fine, but I have a hard time to understand which clusters/workloads within the Kubernetes Engine on GCP are actually used by the prefect agent. The config tab in the web UI is rather empty 🙂 Thanks for your help!
    k
    • 2
    • 6
  • z

    Zach Schumacher

    02/15/2022, 4:21 PM
    Hi, we have a Parameter that is a date type. This gives us a date picker in the UI, but it is setting a datetime instead of a date (see screenshot).
    k
    m
    • 3
    • 4
  • n

    Natsume Kirito

    02/15/2022, 4:37 PM
    Can a flow with multi tasks be deployed as multi Kubernetes jobs, because each task has a different docker image?
    k
    • 2
    • 5
  • g

    Gabriel Milan

    02/15/2022, 4:41 PM
    Hello! I've been struggling with the "maximum number of scheduled runs per flow" problem. I'm aware that this is supposed to be solved as in https://github.com/PrefectHQ/server/pull/281, but I can't seem to put it to work with the Helm chart. I'm trying to set it in the
    values.yaml
    file such as
    prefectConfig:
      services:
        towel:
          max_scheduled_runs_per_flow: 50
    and when I try to upgrade my deployments with this, the towel pod doesn't get redeployed and the issue is still there. Is this way correct or should I do anything else?
    k
    • 2
    • 16
  • a

    Adi Gandra

    02/15/2022, 5:13 PM
    Hey, is there any easy way to get notifications through email or slack if a flow run on prefect cloud fails?
    z
    • 2
    • 3
  • x

    Xavier Babu

    02/15/2022, 5:47 PM
    Dear Prefect Community, What is the API payload I have to use to register a particular FLOW or TASK via APIs? Do I need to provide the entire Python code or the path of the .py file? I don't want to use CLI or Prefect UI to do the same. If you can provide an example with details, that would be great.
    a
    k
    • 3
    • 35
  • j

    Jake

    02/15/2022, 6:59 PM
    Hi everyone; I’m running into the following error:
    ConnectionClosedError('Connection was closed before we received a valid response from endpoint URL: "<insert_s3_url_here>.prefect_result".',)
    It is causing our flows to fail intermittently and I’m not sure how to debug this.
    k
    • 2
    • 17
  • d

    David Wang

    02/15/2022, 8:24 PM
    Hi everyone, I have a quick question about the scaling of prefect ECS agents and clusters of Fargate type. How exactly does it work? Is it automatic scaling or do I have to configure the auto scaling for ECS clusters myself?
    a
    k
    • 3
    • 4
  • k

    Kevin Mullins

    02/15/2022, 8:34 PM
    Not sure how to start, but I wanted to talk a bit about a concern I have with predictable task usage and task-based billing. When I originally started looking into the pricing, I had the concern most people would have about the need to use a lot of tasks in prefect (fetching secrets, unpacking parameters, working with output from other tasks, etc.). Then I came across the following documentation (https://www.prefect.io/pricing/) which helped alleviate some of these fears.
    Prefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
    Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.
    Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets. I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.
    a
    c
    k
    • 4
    • 22
  • c

    Chowdary

    02/15/2022, 10:04 PM
    Hi I am a beginner
  • c

    Chowdary

    02/15/2022, 10:05 PM
    I have a python script in my Jupyter notebook and want to connect this to prefect tool, can any one help me with that
    👍 1
    k
    • 2
    • 1
  • b

    brian

    02/15/2022, 10:56 PM
    Hi prefectionists, I have a question about running a flow locally with the CLI. I’ve been using
    --param
    to set parameters locally. I’ve just started working on a flow that will depend on a prefect secret, and would love to be able to set a value for the secret locally. Is there a way to do this? My naive hope was that
    prefect run
    would have a
    --secret
    flag similar to --param but this doesn’t seem to be the case
    k
    • 2
    • 4
  • w

    William Grim

    02/16/2022, 12:26 AM
    I’ve got a question. It seems when you use an idempotency key, the key sticks around for 24hrs even if the flow run finished, meaning you can’t run the flow with the same key again. The problem with this is that it would be nice to be able to say “while a flow is running with this key, another one cannot, but when the flow finishes, the key becomes available again.” Is there a way to do this? Is there some sort of graphql or anything else that can be executed to free up an idempotency key as soon as a flow finishes?
    k
    a
    • 3
    • 30
  • n

    Nitin Bansal

    02/16/2022, 6:29 AM
    Hi, Has anybody execute a flow (registered in prefect cloud) using Azure Function?
    a
    • 2
    • 1
  • m

    Mukamisha jocelyne

    02/16/2022, 8:45 AM
    Hello, I'm trying to download a file from s3 using a prefect docker agent , save it into some location (for e.g: ./data/file.txt) in my docker container, but the problem is when downloading the file is pulled down with some extension added to it something that looks like this: ./data/file.txt.405fgh and this is making difficult to access the file. Can someone tell me how to get rid of the extension that is being append to the filename? This is how the error appears on prefect cloud UI
    a
    • 2
    • 6
  • n

    Noah Holm

    02/16/2022, 10:56 AM
    Hey all, we ran into an unexpected (to us) behaviour today when a flow was considered successful even though some of it’s tasks failed and I’m curious to hear about potential solutions. We have this pattern of tasks in a flow with dependencies according to the arrows:
    task1
    ->
    task2
    ->
    task3
    Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states:
    Failed
    ->
    TriggerFailed
    ->
    Skipped
    I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?
    a
    • 2
    • 3
  • c

    Chris Arderne

    02/16/2022, 11:49 AM
    How strongly unique are the <adjective>-<animal> Flow run names that Prefect Cloud generates? Should I append a timestamp/uuid if I want a strong guarantee against collisions?
    a
    • 2
    • 5
  • d

    Daniel Nilsen

    02/16/2022, 12:08 PM
    is it possible to cancel a flow run through the gql api?
    a
    • 2
    • 2
  • j

    Jason Motley

    02/16/2022, 1:00 PM
    I've suddenly received the following error twice in a row on a scheduled flow that previously ran fine. Any ideas?
    a
    • 2
    • 3
  • r

    Richard Hughes

    02/16/2022, 2:55 PM
    Hi - I am trying to find out if the upstream_tasks can take wildcards - I am kind of struggling to find the documentation on this kwarg.
    k
    • 2
    • 6
  • g

    Guillaume Latour

    02/16/2022, 3:04 PM
    Hi everyone, I stumbled upon an error as the prefect engine serializer tried to pickle a task result:
    TypeError: cannot pickle 'lxml.etree.XMLSchema' object
    Of course the result of one of my task is of this type and I would prefer not to change it. I tried to launch that process again with
    config.flows.checkpointing = "false"
    in the
    ~/.prefect/config.toml
    file but I got the same error... This is an error that I cannot reproduce locally with
    prefect run <flow>
    . Do you guys have any leads on how I can solve this issue or at least reproduce the error locally?
    k
    a
    • 3
    • 7
  • e

    Emma Rizzi

    02/16/2022, 3:47 PM
    Hi! I've been reading about tasks mapping, is there an equivalent to flow mapping ? Also, when mapping a task, are all tasks launched in parallel ? I would need to control the order of execution
    k
    a
    • 3
    • 18
  • s

    Sean Talia

    02/16/2022, 4:12 PM
    Is there a good heuristic for understanding when to actually expect a performance improvement from converting a task that, say, operates over a list in sequence to a mapped task? I have a task that loops over hundreds of CSV files, parses and transforms the data, and writes the parsed data to an output file (one output file for each input file)...I was anticipating that converting this process to a mapped task and increasing my CPU count (I'm running this on Fargate via ECS, so I upped from 0.5 to 2 vCPUs in this specific case) would help out a lot by parallelizing this file-parsing procedure, but it actually has caused that node in my DAG to take much longer than it used to. I'm wondering if the overhead of this mapped task getting compiled and sent to the LocalDaskExecutor is just significantly outweighing the gains from the parallelization, because each mapped task is so small (the files being parsed are not large), and this would only make sense to do if the files were much, much larger?
    k
    • 2
    • 13
  • s

    Sam Werbalowsky

    02/16/2022, 6:21 PM
    Hey - has anyone here used Prefect with ML (or other data processing) and then outputted charts to a PDF or PNG? We’re exploring this as an option for getting insights into our ML models.
    k
    a
    e
    • 4
    • 12
  • c

    Chris Reuter

    02/16/2022, 7:43 PM
    Hey all! Your friendly host of PrefectLive 📺 letting you know we won't be live on Twitch this week. We'll see you there next week though! There is a Fireside 🔥 Chat this Friday at 3p Eastern/12p Pacific where @Jeremiah and @Chris White will cover the 😛refect: 1.0 release candidate, the latest Orion updates and any questions the community might have. Sign up to attend here, or join the event directly on 

    Youtube▾

    ! DM me any questions you want them to cover live 🙂
    🙏 2
    🔥 3
    :marvin: 4
  • j

    Jason Motley

    02/16/2022, 8:11 PM
    Does Prefect with MySQL support this type of stored procedure? I got an error when the code runs fine querying our database directly -- ``SET @Date1 := MAKEDATE(YEAR(CURDATE()), 1) + INTERVAL QUARTER(CURDATE()) QUARTER - INTERVAL 1 QUARTER;`
    SET procedure2 := bla bla
    etc..
    k
    • 2
    • 8
Powered by Linen
Title
j

Jason Motley

02/16/2022, 8:11 PM
Does Prefect with MySQL support this type of stored procedure? I got an error when the code runs fine querying our database directly -- ``SET @Date1 := MAKEDATE(YEAR(CURDATE()), 1) + INTERVAL QUARTER(CURDATE()) QUARTER - INTERVAL 1 QUARTER;`
SET procedure2 := bla bla
etc..
k

Kevin Kho

02/16/2022, 8:12 PM
I guess the question is if
pymysql
does. What is your error?
j

Jason Motley

02/16/2022, 8:15 PM
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SET @includeLostPipeline := FALSE' at line 2")
k

Kevin Kho

02/16/2022, 8:15 PM
You stored proc has no parameters right?
j

Jason Motley

02/16/2022, 8:17 PM
We later pass through the things defined up front (e.x. "Date1") into where clauses
e.x. ``, @thing:= IF(@thing2= s.Id, @thing + 1, 1) AS ranking``
k

Kevin Kho

02/16/2022, 8:20 PM
I think this question is better for pymysql. I would suggest you create an issue there cuz they’d be able to help you way better
Under the hood we just do this if you are using our task
j

Jason Motley

02/16/2022, 8:22 PM
Okay that's great, I'll ping them. Thank you!
View count: 4