https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • o

    Oluremi Akinwale

    08/31/2022, 10:48 AM
    Hello team! I am Joseph and new to prefect, which material can I use to get start please?
    ✅ 1
    👋 1
    b
    5 replies · 2 participants
  • r

    Rahul Kadam

    08/31/2022, 11:18 AM
    Hi Team, Sometimes we are seeing task failures in ECS with this error
    Failed to start task for flow run ad860ca0-61df-4055-b3f6-7f48c2d9ead4. Failures: [{'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:CPU'}, {'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:MEMORY'}, {'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:MEMORY'}]
    Understandably, its because enough resources are not available on ECS nodes to schedule the tasks when launch type is EC2. But as per my understanding, in such cases the ECS task should get scheduled with "Provisioning" state, and ECS cluster should scale out and add more instances after which the task would run. However in our case we see above error and the task just fails to schedule. Is there a specific parameter we are missing which will provide the ECS feature that i mentioned above ? or this is the expected pattern ?
    ✅ 1
    c
    1 reply · 2 participants
  • t

    Timo

    08/31/2022, 11:31 AM
    Hi there, did anyone get a Docker infrastructure with a private Docker registry in 2.3.0 to work? I defined a "Docker Registry" and a "Docker Container" block and used the Docker Container block as infrastructure block in the deployment.
    prefect deployment build myflow.py:myflow \
    --path /prefect/flows/myflow \
    -ib docker-container/dev-image \
    --name local_docker \
    --skip-upload \
    --apply
    After running the deployment it breaks with "no basic auth credentials". Inspecting the "Docker Container" block shows that it should use the defined registry block. Double checked the credentials multiple times and successfully logged in with
    docker login...
    I tried the same on 2.2.0 with the same error after executing the deployment :-(
    ✅ 1
    👀 1
    a
    c
    +1
    20 replies · 4 participants
  • j

    Jeff Hale

    08/31/2022, 11:36 AM
    set the channel topic: Welcome! 👋 -Please put code & tracebacks in 🧵 -✅ means addressed, not always resolved -:github: bugs & feature requests : https://github.com/PrefectHQ/prefect/issues -Discourse: https://discourse.prefect.io
    ❤️ 3
    💯 4
  • a

    Adam Brusselback

    08/31/2022, 12:29 PM
    Is there planned to be any concept of "agents" in the api/UI that shows the different agents that have been connected over time, when they were last active, etc?
    ✅ 1
    a
    s
    8 replies · 3 participants
  • m

    Mohammad Tami

    08/31/2022, 2:23 PM
    Hi There, I just started with Prefect v2, and trying to figure out the following: 1. Let’s assume we deployed a new pipeline to Prefect Server, and this new pipeline has new dependencies (external python libraries), How we can automate the installation of these dependencies on the agent execution environment while guaranteeing high availability (assuming that pipelines are running 24/7 on our servers)? 2. In the case of an Agent deployed on Kubernetes, How does the agent mange the workload across the Pods, is it automatically handled by the Agent, or do we need to handle this from our side?
  • j

    Josh Paulin

    08/31/2022, 2:32 PM
    Is there any way to change the timezone on the UI in 2.0?
    ✅ 1
    a
    j
    +1
    7 replies · 4 participants
  • p

    Prakash Rai

    08/31/2022, 2:57 PM
    Is it possible to run tasks in docker environment? Usecase: I have a Flow composed of 3 tasks. First two are pretty simple, and can be done without using any external python library. The third one requires some external python libraries and a few system dependencies to be installed. Is there a way to run the first two tasks on the host environment, and the third in docker?
    ✅ 1
    k
    4 replies · 2 participants
  • j

    Jason

    08/31/2022, 3:21 PM
    Hi everyone - we're still on Prefect V1 for Q3. We had a failing heartbeat with a long-running API backfill. I did switch our configuration to 'threads' as documented here, but two backfills failed again last night (while a third smaller one succeeded). What's interesting is that the ECS Cluster is still active but the agent has stopped responding to new incoming requests, and there's likely a zombie job in Prefect due to this. The specific question I have is "Has anyone else had a problem of a new FARGATE job stopping the agent itself from responding?" I thought, based on the configuration, that those would have been on separate FARGATE clusters on ECS.
    👀 1
    ✅ 1
    b
    10 replies · 2 participants
  • k

    Khuyen Tran

    08/31/2022, 3:31 PM
    Hey all, we’re hosting @Will Raphaelson, a senior tech product manager at Prefect, on PrefectLive today at 3p Eastern where he’s going to show the latest improvements to deployments, in particular using Docker and Github! Come join us live on Twitch!
    👍 3
    :party-parrot: 6
    🙌 3
    ✅ 1
    :marvin-duck: 3
    f
    j
    4 replies · 3 participants
  • t

    Tomas Moreno

    08/31/2022, 4:16 PM
    hey team! I'm working with prefect 1.0 and curious about triggering a flow in one tenant, when a flow in another tenant finishes successfully. I think it may be possible with a graphql query similar to this post https://stackoverflow.com/questions/68981899/create-flow-run-on-localhost-with-http-request-how-to-pass-parameters-graphql. I'm curious if anyone has any alternatives that they think are better. I'm also trying to find my graphql endpoint from these instructions https://docs-v1.prefect.io/orchestration/server/deploy-local.html#ui-configuration, but I don't see a prefect server tab, anyone know what's going on?
    a
    3 replies · 2 participants
  • e

    Elio

    08/31/2022, 4:18 PM
    Hi I'm deploying prefect v1 on a VM with Ansible. I actually have a playbook to relaunch my prefect agent when needed with a kill and a launch with nohup.

    https://puu.sh/Jj0Q7/77bd248705.png▾

    If my VM crash it doesn't relaunch the prefect agent as I use nohup Does someone here have ever done a linux service or a daemon to handle prefect agent ?
    1️⃣ 1
    j
    1 reply · 2 participants
  • b

    Blake Stefansen

    08/31/2022, 4:32 PM
    Hi Everyone, When I create a deployment, the deployment should be uploading all my files in my current directory to my storage (in this case, s3), however, this isn't happening on prefect 2.3.0 I've triple checked my
    .prefectignore
    and no, it is not ignoring anything critical. Is this a bug?
    j
    4 replies · 2 participants
  • m

    Maria

    08/31/2022, 5:57 PM
    Hello, is there a documentation on how to set up prefect orion server? e.g. other than
    PREFECT_API_URL
    what other things need to be configured? I keep getting
    [Errno -2] Name or service not known
    I have prefect orion set up with helm in EKS
    ✅ 1
    m
    12 replies · 2 participants
  • m

    Marc-Antoine Bélanger

    08/31/2022, 6:31 PM
    I am using prefect2.0. I have the following flow that I run in a docker infrastructure.
    @flow
    def my_flow():
        return "hola"
    Using the OrionClient I am able to read the states of a flow run using
    orion_client.read_flow_run_states(flow_run_id)
    . The last state is
    COMPLETED
    and has a
    data
    attribute that is NOT
    None
    . For example the data attribute value is:
    state.data = _Result(key='b1fd11041ae54cfcbb9b41ec938ff0cf', filesystem_document_id=UUID('bd1ed851-82d2-4f87-b11c-be04a80634af'))
    What does this represents? Why don't I have
    state.data = "hola"
    instead? How do I get my flow return data from the OrionClient? Thanks a lot!
    ✅ 1
    m
    4 replies · 2 participants
  • l

    Luis Diaz

    08/31/2022, 6:54 PM
    Hi everyone, I tried to set a pipenv environment in my deployment but when I ran the pipeline the environment didn't start up. Does anyone know how to fix this? Thanks!
    ✅ 1
    j
    1 reply · 2 participants
  • k

    Khuyen Tran

    08/31/2022, 6:54 PM
    Hey all, Prefect Live will start in 5 minutes (today at 3p Eastern). We’re hosting @Will Raphaelson, a senior tech product manager at Prefect, on PrefectLive where he’s going to show the latest improvements to deployments, in particular using Docker and Github! Come join us live on Twitch!
    :party-parrot: 2
    😛anda-dancing: 2
    :github: 2
    🚀 3
    :radar: 2
    :blob-attention-gif: 2
    :marvin: 3
    :hattip: 3
    ✅ 1
    🎊 3
  • m

    Mansour Zayer

    08/31/2022, 7:05 PM
    Hello. in Prefect1, I have a task in a flow, and I want to reuse this task in another flow 2 times with different parameters, with the second run being dependent on the fist one. Something like this:
    ../flow_1.py
        @task(my_task)
        def my_task(param)
    .................................
    from flow_1.py import my_task
    ../flow_2.py
        @Flow(some_flow):
            my_task(param_1)
    
            my_task(param_2, upstream_tasks=[my_task])
    Since the 2 tasks in my Flow in flow_2 have the same name, would
    upstream_tasks
    work? Is there a better way to do this? Thank you
    ✅ 1
    b
    2 replies · 2 participants
  • j

    Jai P

    08/31/2022, 7:07 PM
    hi there! i'm trying to understand what is happening with subflows, infrastructure-wise. in particular, where they're going to be executing. more detail in thread
    ✅ 1
    n
    10 replies · 2 participants
  • m

    Mars

    08/31/2022, 8:08 PM
    Hi, I’ve got a question about the REST API. The docs for the REST API say that I should be able to issue
    GET /collection
    to list all members of a collection, such as
    GET /api/flow_runs/
    . However, the server, OpenAPI spec, and example at http://localhost:4200/docs#/ say that I must supply an id to the collection, such as
    GET /api/flow_runs/1234
    . What the correct way to fetch the full list of collection members?
    1 reply · 1 participant
  • e

    Emerson Franks

    08/31/2022, 8:39 PM
    Hello, 2.0 question. For my kubernetes deployment, I've found a way to modify the flow yaml so that it will automatically delete the job pod shortly after completion. That said, it's not really scalable to manually change these every time. Is there a way to add the spec.ttlSecondsAfterFinished value via prefect cli or is that coming #soon?
    ✅ 1
    h
    6 replies · 2 participants
  • s

    Sam Garvis

    08/31/2022, 8:55 PM
    In Prefect 2.0, does GitHub storage work with private Git repos? I don't really see an option to pass in creds or an api key to git
    ✅ 1
    j
    2 replies · 2 participants
  • t

    Tony Yun

    08/31/2022, 9:57 PM
    is there any change to the personal prefect UI? I cannot no longer open it:
    ✅ 1
    j
    6 replies · 2 participants
  • a

    Adam Brusselback

    08/31/2022, 10:08 PM
    Question, I have a simple flow that runs a single Postgres function through the SqlAlchemy module. I have orion running on a remote node, and I am running an agent locally and connecting to a test queue to try things out... I am getting multiple min delays between the flow starting and the task starting though
    ✅ 1
    m
    9 replies · 2 participants
  • j

    Jovan Sakovic

    08/31/2022, 11:13 PM
    Hi all 👋 Quick Prefect 1 version (potentially 2.0 too, dunno yet 😄) Self-hosting with Docker, with the URL of Slack webhooks, is it possible to configure to a custom domain, rather than the default
    localhost/4200
    that we can’t reach without connecting to the VM etc..
    ✅ 1
    a
    1 reply · 2 participants
  • m

    Matt Fysh

    09/01/2022, 1:03 AM
    Hi everybody! I’m having trouble getting the
    DaskTaskRunner
    working in this tutorial: https://docs.prefect.io/tutorials/execution/#parallel-execution I get a bunch of errors, e.g. “no active connection” , or “TypeError: DaskTaskRunner.submit() got an unexpected keyword argument ‘key’”
    ✅ 1
    j
    1 reply · 2 participants
  • a

    Adam Brusselback

    09/01/2022, 1:45 AM
    Also, I am having other (file permissions?) failures when trying to run on my deployed infrastructure rather than locally for the exact same flows mentioned above (see log in thread) I wasn't able to find anything pointing me to a resolution here... the files exist in the specified deploy location, they work in my local docker container, they don't work on this VM.
    ✅ 1
    j
    6 replies · 2 participants
  • a

    Architha Rao

    09/01/2022, 6:52 AM
    Hi All. My flow fails because a task returns "No heartbeat detected from the remote task; marking the run as failed". I used some log statements to debug, seems that the flow doesn't even start. Please suggest what I can do. Thank you.
    ✅ 1
    a
    1 reply · 2 participants
  • k

    Konstantinos

    09/01/2022, 7:17 AM
    Hi, we are randomly experiencing an issue with our prefect scheduler. Some days, some flows get submitted late. We don't see any issue with resources on the hosting servers and we don't think it is a clock issue, as other days, same flows run on schedule fine. Perhaps someone here has experienced this and has a pointer. Note: both server and agents are deployed on the same server, run out of simple python virtual environment.
    1️⃣ 1
    l
    2 replies · 2 participants
  • r

    Robert Kowalski

    09/01/2022, 7:36 AM
    Hi, I have problem with login into prefect cloud using google account. I get error 400 and status access_denied. This user can be blocked or something ? (dev@algopolis.ai)
    ✅ 1
    j
    b
    8 replies · 3 participants
Powered by Linen
Title
r

Robert Kowalski

09/01/2022, 7:36 AM
Hi, I have problem with login into prefect cloud using google account. I get error 400 and status access_denied. This user can be blocked or something ? (dev@algopolis.ai)
✅ 1
j

Jeff Hale

09/01/2022, 12:07 PM
This looks like 1.0, correct?
r

Robert Kowalski

09/01/2022, 12:07 PM
yes
b

Bianca Hoch

09/01/2022, 2:22 PM
Hi Robert, just to confirm, have you tried using the password reset link? https://universal.prefect.io/signin/forgot-password
Note that the email for resetting passwords can occasionally land in your spam box.
👍 1
r

Robert Kowalski

09/01/2022, 2:28 PM
Now i tried and a have another error
b

Bianca Hoch

09/01/2022, 4:35 PM
Noted, we'll take a look and see what could be going on here
Hello Robert, the account has been unlocked. Please sign in with your Google SSO and let us know if you have access.
r

Robert Kowalski

09/01/2022, 5:01 PM
Hi, I have access to dev account , thank You very much .
:marvin: 1
View count: 3