https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Andreas Tsangarides

    10/26/2022, 8:20 AM
    Hi all. Is there a way to filter out default prefect logs like:
    Starting task run...
    [2022-10-25 15:23:07+0100] INFO - prefect.TaskRunner | Task 'read_features_config[1]': Starting task run...
    [2022-10-25 15:23:07+0100] INFO - prefect.TaskRunner | Task 'read_features_config[3]': Finished task run for task with final state: 'Success'
    1️⃣ 1
    ✅ 1
    r
    r
    5 replies · 3 participants
  • i

    iKeepo w

    10/26/2022, 9:20 AM
    hello, why my flow always stuck at Pending State with the agent log like:
    k
    r
    8 replies · 3 participants
  • i

    iKeepo w

    10/26/2022, 9:26 AM
    I found the explaination here, but seems like it's for prefect 1, there is no --show-flow-logs in prefect 2
  • d

    Deepanshu Aggarwal

    10/26/2022, 9:41 AM
    hi! im trying to use my postgres db running on rds as the database for orion(self hosted instance) backend. but it continuously runs into time out errors. has anyone implemented this before or facing the same issue ?
    ✅ 1
    m
    a
    18 replies · 3 participants
  • d

    Daniel Komisar

    10/26/2022, 11:46 AM
    Hello all, does anyone know what the Prefect Cloud retention policy is for flow and task results, and how large the
    location
    field may be (i.e. for use with
    PrefectResults
    )? Thanks!
    r
    3 replies · 2 participants
  • a

    Adam

    10/26/2022, 2:21 PM
    Hi team, I’m a bit lost in how to configure JSON logging in Prefect 2.0. I see this concept document and this api reference on the JSONFormatter, but I can’t seem to find how to marry the two together. What we’re trying to achieve is: 1. Prefect Cloud 2.0 still shows logs as is 2. Kubernetes jobs output logs in JSON so that they can be ingested by datadog (we want each log entry to have the flow_name, task_name, etc. to make it easily searchable back in time).
    ✅ 1
    k
    t
    12 replies · 3 participants
  • s

    Sunjay

    10/26/2022, 2:36 PM
    Hi folks, quick dumb question : to set to the type of parameter for a flow, do i just use type hints to set the type of parameters?
    ✅ 1
    k
    9 replies · 2 participants
  • f

    Farooque Shaikh

    10/26/2022, 3:02 PM
    Hi All, I keep getting the below Timeout error while fetching data using graphql in a python code for Prefect 1.0 prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}] Is there any solution to avoid this?
    c
    5 replies · 2 participants
  • h

    Hamza Naanani

    10/26/2022, 3:04 PM
    Hello, I'm trying to deploy prefect agents on an on prem environment, but I get a
    prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxxxxxx/workspaces/xxxxx/work_queues/name/dev-prem-wq>'
    I tried changing the api keys (service account) / giving ownership to the agent on the workspace, but nothing worked so far. I tried deploying the agents on the cloud on a kubernetes cluster and it works just fine, but when I try to do the same thing on our on prem environment, it returns me this error. Can I have some insights please ? Thanks
    ✅ 1
    c
    7 replies · 2 participants
  • y

    YD

    10/26/2022, 3:30 PM
    @Nate Hi Nate, I suppose to have a meeting with you tomorrow at 8PST I did not get the email invite from https://calendly.com/ (yet) Youval
    ✅ 1
    k
    n
    2 replies · 3 participants
  • k

    Khuyen Tran

    10/26/2022, 4:01 PM
    Fugue is a unified interface for distributed computing that lets users execute Python, pandas, and SQL code on Spark, Dask and Ray without rewrites. prefect-fugue provides the Fugue tasks, context and blocks for Prefect. In Prefect Live at 3PM Eastern today, Kevin Kho, a former community engineer at Prefect, will talk about how to use this integration and its use cases. Come join us live on Twitch. Subscribe to Prefect Community Calendar, available at prefec.tv/calendar, to get updated with all Prefect Live events.
    👍 1
    ❤️ 1
    :fugue: 4
    :party-parrot: 3
    :blob-attention-gif: 1
    🙌 2
  • n

    Nora Myer

    10/26/2022, 4:20 PM
    Hey hi - prefect noob here - we’re using Prefect 2.0 w/ Prefect Cloud in a new system were building, and I have a couple Q’s around secrets. Details in the thread - thanks in advance!!
    ✅ 1
    k
    r
    4 replies · 3 participants
  • m

    microft

    10/26/2022, 4:31 PM
    Anyone seen mapped (one of the mapped runs) hanging forever when running in prefect-server with LocalExecutor?
    1️⃣ 1
  • b

    Brian Osserman

    10/26/2022, 4:39 PM
    Hi, I've been moving my workflow orchestration into Prefect, and recently ran across some strange hanging behavior in moving from one task to another which I'm hoping someone can shed some light on. This behavior was observed running Prefect 1.4.0 from windows cmd. I saw it on two different flows so far, and it's a bit inconsistent. What I'm seeing is that one or more of the tasks will have completed (according eg to the logs generated by the actual task), but then it hangs right at the end. When I press Ctrl-C (the first time was to try to abort the run since that's the usual behavior), instead of aborting it seems to prod it into acknowledging that the task completed successfully and it then moves on the next task. I haven't been able to rerun much to test since these are large flows that takes more than a day to run, but in each case I've seen it happening in a few of the tasks but not others, and in one case I was able to rerun for test purposes, and it happened again, but on a different step from the first time. I'm going to post some more details in a thread.
    m
    7 replies · 2 participants
  • j

    Joshua Greenhalgh

    10/26/2022, 4:46 PM
    Hey can someone help me diagnose an issue I have just come across using '1.4.0' - I am suddenly getting;
    AttributeError: 'Context' object has no attribute 'scheduled_start_time'
    ✅ 1
    :marvin: 1
    2 replies · 1 participant
  • p

    Paco Ibañez

    10/26/2022, 4:48 PM
    Hello! I have an Azure block to store my flows, how can I store the flows in different folders within my block?
    ✅ 1
    r
    3 replies · 2 participants
  • t

    Tim Enders

    10/26/2022, 6:02 PM
    Back again with what look like Prefect 2.0 API load issues? My first question is how would I debug the following? I don't have an exception being caught and I think this is a crash inside prefect code, not mine.
    13:00:10.647 | ERROR   | Task run 'Get-Items-d8ed86f1-212' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.650 | ERROR   | Task run 'Get-Items-d8ed86f1-15' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.652 | ERROR   | Task run 'Get-Items-d8ed86f1-857' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.653 | ERROR   | Task run 'Get-Items-d8ed86f1-774' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.654 | ERROR   | Task run 'Get-Items-d8ed86f1-609' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.655 | ERROR   | Task run 'Get-Items-d8ed86f1-444' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.657 | ERROR   | Task run 'Get-Items-d8ed86f1-399' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.659 | ERROR   | Task run 'Get-Items-d8ed86f1-700' - Crash detected! Execution was cancelled by the runtime environment.
    13:00:10.660 | ERROR   | Task run 'Get-Items-d8ed86f1-460' - Crash detected! Execution was cancelled by the runtime environment.
    m
    j
    22 replies · 3 participants
  • m

    Michał Augoff

    10/26/2022, 6:47 PM
    hello! is there a way to deploy flows with non-JSON serializable arguments?
    ✅ 1
    a
    6 replies · 2 participants
  • m

    Mary Clair Thompson

    10/26/2022, 7:34 PM
    I’ve run into an issue with switching my backend database from the default sqlite instance to a postgres instance. Most of the blocks (in particular for my use case the Azure block) are no longer available in the UI and they don’t show up in the postgres db. Do I need to register them specifically now that I’m using a different db?
    ✅ 1
    k
    r
    +2
    10 replies · 5 participants
  • b

    Bradley Hurley

    10/26/2022, 9:53 PM
    Hi Prefect Experts - Im running Prefect 1, but running into an issue that Im having a hard time tracking down. We have Prefect deployed on Kuberneters and are using Dask Executors. The general pattern is: users submits a flow agents creates a job pod job pod creates a dask cluster (minimum of 1 scheduler and 2 workers) tasks run on the dask workers The job pod is consuming memory until it crashes.
    k
    6 replies · 2 participants
  • a

    Adam

    10/26/2022, 9:55 PM
    Hi team, is this the right place to get help on setting up my account for Prefect 2.0 Cloud? My account is in a weird state. Workspaces empty, organizations empty. Can’t create a new workspace either as the Owner dropdown has no entries. Noticing a lot of 403 entires in the devtools console.
    ✅ 1
    j
    4 replies · 2 participants
  • i

    iñigo

    10/26/2022, 10:39 PM
    Hello everyone, How could you pass different parameters in deployments through python? In my flow: @flow(name="Master of flows") def Master_flow( t: str='defecto' ): task_data(t) then in my deployment script: deployment = Deployment.build_from_flow( flow=Master_flow, name="docker_azure_flow", version=1, work_queue_name="default", parameters={'t':'parametro pasado'})
    m
    3 replies · 2 participants
  • e

    Enda Peng

    10/26/2022, 10:58 PM
    With prefect v2, is there a way to kill a flow run in UI? I notice that deleting the flow run won't terminate the process, instead my agent crashed and throwed a 404 error 🤔
    ✅ 1
    m
    4 replies · 2 participants
  • a

    Adam

    10/26/2022, 11:36 PM
    Hi team, is there are any documentation around how Blocks data is stored securely in Prefect Cloud 2.0. If we were to use blocks to store database credentials it would be great to know how these are stored on your side.
    ✅ 1
    k
    1 reply · 2 participants
  • t

    Thomas Pedersen

    10/27/2022, 9:54 AM
    Is there a way to set a flow concurrency limit on the agent itself? As far as I can work out, work queue limits aren't going to work when we have multiple agents per queue, and agents that listen to multiple queues at the same time (with
    agent start --match
    ). All I need to tell each agent is "Don't start more than N docker containers at the same time" (or you will run out of disk or memory)... E.g. the limit is really on the agent resources. In the full cloud I don't care how many things run simultaneously - the more the merrier :D
    ✅ 1
    a
    j
    +1
    15 replies · 4 participants
  • a

    Andrei Tulbure

    10/27/2022, 10:30 AM
    Hi. Quick question: I have received an error for Prefect 1.x because my order of returned parameters was a little bit messed up, interchanged 3 parameters for a prefect task and one ended up missing. And the error I got was pretty weird, something in the lines of `ValueError: Could not infer an active Flow context while creating edge to <Task: Task1>. This often means you called a task outside a
    with Flow(...)
    block. If you're trying to run this task outside of a Flow context, you need to call `GetItem(...).run(...)`` . All in all error was because of missing params from function call. Could you make the error a little bit more helpful ? Or specific ? ...... Or am I missing something ?
    ✅ 1
    j
    2 replies · 2 participants
  • n

    Nic

    10/27/2022, 10:50 AM
    How do we use the prefect string and secret blocks in an ipynb file? It works fine in py files, but when using then in notebooks i get following error
    ✅ 1
    j
    2 replies · 2 participants
  • a

    Adam

    10/27/2022, 11:08 AM
    Hi team, could someone give me some guidance on how we can pass in an
    envFrom
    section to a
    KubernetesJob
    block. We want to include this using the Python API:
    envFrom:
                - secretRef:
                    name: job-secrets
    r
    q
    +1
    18 replies · 4 participants
  • j

    Jons Cyriac

    10/27/2022, 11:33 AM
    is prefect 2.0 the same as orion? are they aliases to each other?
    ✅ 1
    j
    1 reply · 2 participants
  • c

    Christian Vogel

    10/27/2022, 12:50 PM
    Hi Prefect Community. One quick question: In case one is running Prefect via Prefect Cloud, is it still possible to configure server side settings equivalent to this one from orion? https://docs.prefect.io/api-ref/prefect/settings/#prefect.settings.PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS
Powered by Linen
Title
c

Christian Vogel

10/27/2022, 12:50 PM
Hi Prefect Community. One quick question: In case one is running Prefect via Prefect Cloud, is it still possible to configure server side settings equivalent to this one from orion? https://docs.prefect.io/api-ref/prefect/settings/#prefect.settings.PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS
View count: 1