https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • p

    Pekka

    10/02/2022, 3:56 PM
    Hi. I'm curious about testing
    pypy 3.9
    since it has JIT. I'm trying to install prefect as the minimal dependency but there are some problems that will probably be outside of the 'supported things' -- prefect has
    orjson
    as a dependency which is not pypy-supported. I wonder how big of a role that dependency plays in prefect and whether there's some way to hack a bit to make it use some other lib?
    a
    • 2
    • 1
  • t

    Tony Popov

    10/02/2022, 5:39 PM
    Hey, community! How do I set up task caching for k8s jobs? not really clear what setting
    PREFECT_LOCAL_STORAGE_PATH
    does in this context, I guess points to pod’s filesystem and persistent storage/bucket has to be mounted to a path
    ✅ 1
    a
    • 2
    • 2
  • w

    William Wolfe-McGuire

    10/02/2022, 7:26 PM
    in prefect 0.15.13 is there a good way to check whether a task and its output is serializable? I know you can use
    is_serializable
    to check if entire flows can be serialized but that doesnt allow you to narrow down which tasks are responsible for your serialization issues. what is the best way to debug in this situation?
    ✅ 1
    a
    • 2
    • 4
  • t

    Tony Popov

    10/03/2022, 1:16 AM
    A question on how to use async tasks: Context
    class Result:
      pass
    
    @task
    async def my_task_1(arg: str) -> Result:
      ...
    
    @task
    async def my_task_2(arg: Result) -> None:
      ...
    Question Doc says I can do
    await my_task_1('arg')
    in a flow, although mypy says
    my_task_1('arg')
    is not awaitable Is following a best practice?
    task_result = await my_task.submit('arg')
    result = await task_result.result()
    
    await my_task_2.submit(result)
    b
    • 2
    • 3
  • j

    Jessica Smith

    10/03/2022, 2:00 AM
    I am suddenly finding that mapped tasks are not reflecting that the child tasks have completed. I'm not positive that it's not because I introduced some kind of bug while developing, but I've done my best to validate that it's not something I did. Has anyone else seen this? Is something just going wonky with the cloud service at the moment? For example, the flow in the image has finished successfully, with all child tasks having completed. But it still shows 0/13.
    ✅ 1
    c
    l
    j
    • 4
    • 9
  • z

    Zac Hooper

    10/03/2022, 2:41 AM
    I seem to be running into an issue where sub 5 minute flows aren't being scheduled. If I re-register the flow it'll schedule the next 10 runs but once they have been completed now more are scheduled. I've tried using cron clocks and interval clocks and manually creating a schedule in the UI but keep seeing the same result. I'm using Prefect Cloud 1.0 I have two agents both deployed on AWS ECS.
    ✅ 1
    j
    • 2
    • 7
  • k

    Krzysztof Ciesielski

    10/03/2022, 8:39 AM
    Anyone notices issues with Prefect Cloud today? Our flows show all tasks as "pending", but we can see it logs that they get executed. This happens for all the flows, there wasn't any updates or deployments at our organization since quite a few days. Yesterday all was working fine, quite a few flows completed successfully. The issue seems to be happening since a few hours 🤔
    👀 1
    ✅ 1
    :plus-one: 6
    j
    • 2
    • 6
  • t

    Thomas Opsomer

    10/03/2022, 8:40 AM
    Hello Prefect team, Since yesterday we're experiencing many issues: • restarting flow/task doesn't work • tasks with manual approval don't start after being approved • manually changing task state doesn't work
    ✅ 1
    :plus-one: 2
    👀 1
    1️⃣ 1
    j
    • 2
    • 7
  • m

    Michal Luščon

    10/03/2022, 8:41 AM
    Hi all, I am trying to figure out whether there is an option for running orion server in url subpath like my-domain.com/prefect. Is there any documentation for this case?
  • j

    Joshua Greenhalgh

    10/03/2022, 8:42 AM
    Hi - I am seeing lots of these errors over last few days;
    Exception raised while calling state handlers: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 140, in get
        value = secrets[self.name]
    KeyError: 'SLACK_WEBHOOK_URL'
    1️⃣ 1
    ✅ 1
    j
    • 2
    • 6
  • s

    Stefan

    10/03/2022, 8:42 AM
    Hi there, I have a question to the Prefect team on the hybrid model but also want to raise awareness on a potential security/data protection issue for other users. According to the Prefect website, code and data should never leave a customers infrastructure. The "exception" here are of course the log entries which can contain any kind of data. In Prefect 1, this was ok for us as we sanitized our own log entries and the logs generated by Prefect itself did not seem to contain any sensitive data. However, in Prefect 2, the log entries generated by Prefect may contain all arguments passed to a function! So far I only found the example below where it happens when the passed argument names do not match what the task expects. However, are there more of these cases were data can leak into the logs? Is this really the intended behavior?
    from prefect import task, flow
    
    @task
    def task1(arg1):
        pass
    
    @flow
    def main_flow():
        task1(arg1_wrong=1)
    
    main_flow()
    Logs on Prefect Cloud:
    c
    • 2
    • 3
  • a

    Alberto Fernández Zazo

    10/03/2022, 10:00 AM
    Hi Prefect team. Apart from the issues mentioned before by Krzysztof and Thomas, we also see that some flows are not actually executed. It seems so from the parent flow, with a duration of 30 seconds, and the next dependent flows are triggered, but we cannot find the flow run when accessing the child flow directly. Besides, the UI shows both a “success” and a “failed” status
    ➕ 1
    :plus-one: 5
    ✅ 1
    j
    m
    • 3
    • 7
  • s

    Stéphanie Cérulli

    10/03/2022, 10:05 AM
    same here...
    ✅ 1
    j
    • 2
    • 2
  • k

    Krzysztof Ciesielski

    10/03/2022, 10:10 AM
    We are seeing even weirder behavior now, one flow was running one of its tasks twice (in parallel). Fortunately we have duplicate email detection on our notification gateway, but this is really concerning.
    :plus-one: 3
    ✅ 1
    j
    • 2
    • 5
  • m

    Malavika S Menon

    10/03/2022, 10:31 AM
    I'm trying to change PREFECT_API_URL by running
    prefect config set PREFECT_API_URL=<https://example.com/api>
    . However when I try to run the server it defaults back to http://127.0.0.1:4200/api
    c
    • 2
    • 1
  • h

    Hedgar

    10/03/2022, 12:48 PM
    Hey I created a secret block on prefect cloud ui,
    secret-block = Secret.load(“api-key”)
    I want to use this block in the body of my flow how can I go about this? Should I just pass
    secret-block
    ? Do I need to use the get method anywhere?
    ✅ 1
    j
    • 2
    • 1
  • i

    Igor Kotua

    10/03/2022, 1:48 PM
    Hello, I see there is way to limit concurrency for a task using tags. But what about flows? How to make sure the same flow (or the same deployment) does not run concurrently?
    ✅ 1
    j
    • 2
    • 4
  • s

    Stefan Rasmussen

    10/03/2022, 2:43 PM
    Is there any nice way to modify the arguments given in the
    @task
    decorator after that task has been defined? Say I use a task from a collection that and want to use a custom
    cache_key_fn
    . It looks like i could do it by
    from copy import copy
    from some_collection import some_task
    some_task_cache = copy(some_task)
    some_task_cache.cache_key_fn = my_cache_fn
    but that seems a bit hacky. The actual problem is the other way around - I want to provide a collection for users, but they would want to modify the task decorator arguments.
    m
    • 2
    • 2
  • h

    Hedgar

    10/03/2022, 3:08 PM
    @Anna Geller I’m still working on implementing the aws-lambda-serverless-flow template example. On a closer look at the repo, I can see that you didn’t build deployment with prefect(no deployment scripts). Was this intentional or an omission? My attempt to build and apply deployment led to this error:`File "/home/ec2-user/.servy/lib/python3.9/site-packages/botocore/handlers.py", line 278, in validate_bucket_name` raise ParamValidationError(report=error_msg) botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid bucket name “s3:“: Bucket name must match the regex “^[a-zA-Z0-9.\-_]{1,255}$” or be an ARN matching the regex “^arn:(aws).:(s3|s3-object-lambda):[a-z\-0-9]:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*😒3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$”
    ✅ 1
    a
    • 2
    • 9
  • s

    Sean Malone

    10/03/2022, 3:13 PM
    Hello, recently (within the past week) I have been unable to delete my flow runs (old and new) in Prefect 2 Cloud. I receive a
    Failed to delete flow run
    message in the UI. Is this a known issue? Or any thought as to why this occurs?
    ✅ 1
    b
    • 2
    • 6
  • f

    Farooque Shaikh

    10/03/2022, 3:27 PM
    Hi All... I am trying to pull detailed prefect logs using the graphql, however i could not find the table/column name in graphql which holds the detailed logs. Any help would be really appreciated
    m
    • 2
    • 9
  • g

    Ghassan Hallaq

    10/03/2022, 4:53 PM
    Hi All, I am working on building profiler(to measure the CPU, Memory and Network IO utilizations) to attach it to any prefect2 task. Any ideas about where I can start from, does ORION collect those kind of data, so we can grab it, or need to create something from scratch?
    m
    n
    • 3
    • 4
  • j

    Jeff Quinn

    10/03/2022, 5:13 PM
    Hello all, love the project, one question: I find myself really wanting a feature flag to turn task level result caching on and off. My use case seems common enough: if I update my ETL code, I want to invalidate caching so that new results on existing input data are produced. Keying the cache on input argument hash doesn’t solve this use case; the input data is identical, it is my code that has changed. Time based cache expiration doesn’t fit this use case; I don’t know when my code will be updated, it might be tomorrow, it might be in a year. It seems like prefect 1.0 had some options in the API to serve this use case. What am I misunderstanding about prefect 2.0, which as far as I understand has no clear way to address this?
    ✅ 1
    m
    • 2
    • 14
  • c

    Christian Juhl

    10/03/2022, 6:54 PM
    Hi all, In Prefect 2, is it possible to schedule a deployment flow with custom input parameters for each schedule in UI/CLI? Or do I need to create a deployment for each set of parameters? Thanks!
    👀 1
    ✅ 1
    r
    j
    • 3
    • 5
  • d

    Darin Douglass

    10/03/2022, 7:05 PM
    is there a way to associate arbitrary data to orion objects? i.e. git repo, sha, contact team, etc may all be useful from a discoverability perspective on all flows/deployments
    ✅ 1
    n
    • 2
    • 3
  • n

    Nace Plesko

    10/03/2022, 7:47 PM
    Hi, I'm using Prefect v1 and I'm having trouble with configuring timeouts for
    ShellTasks
    . Setting timeout like it says in the documentation doesn't have any effect, the task doesn't get terminated after 10 seconds in this case. Is that a bug in Prefect or is it an issue on my part and I'm somehow misconfiguring
    ShellTask
    ? I really appreciate any help on this and thank you in advance!
    ✅ 1
    n
    • 2
    • 8
  • f

    Florian Giroud

    10/03/2022, 8:15 PM
    Hi Prefect, We’re still in our journey to evaluate prefect, one question that comes to mind How do we run a flow that takes input parameters ? I mean without deploying to a agent, simply by running it with the command line
    ✅ 1
    n
    n
    m
    • 4
    • 5
  • a

    Austin Anderson

    10/03/2022, 8:38 PM
    Hi all - I am working through the Getting Started tutorial for Deployments using Prefect 2.4.5 via a Miniconda environment on Windows:
    name: prefect-service
    channels:
      - defaults
    dependencies:
      - python=3.9.13
      - pip=22.1.2
      - pip:
        - prefect==2.4.5
    I am hitting an error after running
    prefect deployment apply log_flow-deployment.yaml
    :
    Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000226A897BEE0>
    ... redacted for readability ...
    RuntimeError: Event loop is closed
    Any idea what is happening here? Please let me know if I can provide any additional info.
    n
    • 2
    • 10
  • l

    Lukáš Polák

    10/03/2022, 8:59 PM
    Hi everyone. We're trying to update our prefect v1 from 1.2.2 to 1.4.0. Since we operate the infrastructure on our own, we use the official helm chart to deploy everything. That part works just fine. Our services make sure that they work with the correct version of prefect when they are about to start after an upgrade in the k8s cluster. This check makes sure that Prefect Server version matches the used prefect library. We use following graphl query to fetch the API version:
    query { api { core_version } }
    We ran into a problem with version 1.4.0 when prefect API reports version "1.3.0+13.g4142953a5". Is that intended?
    c
    • 2
    • 4
  • a

    Alfred Martinez

    10/03/2022, 9:08 PM
    Hi everyone, I am trying to figure out how to set up a job to only run the first 6 days of the month. I was looking at the documentation below, but still having some challenges, wondering if anyone could give some insight on how to do this. https://docs-v1.prefect.io/core/concepts/schedules.html#clocks
    ✅ 1
    n
    • 2
    • 2
Powered by Linen
Title
a

Alfred Martinez

10/03/2022, 9:08 PM
Hi everyone, I am trying to figure out how to set up a job to only run the first 6 days of the month. I was looking at the documentation below, but still having some challenges, wondering if anyone could give some insight on how to do this. https://docs-v1.prefect.io/core/concepts/schedules.html#clocks
✅ 1
n

Nate

10/03/2022, 9:10 PM
Hi @Alfred Martinez! you could use a
CronClock
(docs) with a
cron
expression like this
a

Alfred Martinez

10/03/2022, 10:41 PM
Thank you. I ended up using the UI within prefect. Did not know it was an option. Thank you for the reference though.
👍 1
View count: 1