https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Apoorva Desai

    07/27/2022, 5:55 PM
    Hi, we've been having some issues with Prefect. A couple of hour flows have been running about 14x the time they usually take and I can't seem to cancel and restart them. Logs in thread...
    ✅ 1
    t
    n
    a
    • 4
    • 31
  • m

    Mansour Zayer

    07/27/2022, 8:02 PM
    Hello. None of the links to Prefect docs (from google results) work anymore (404 - Not found) For example: https://docs.prefect.io/api/latest/tasks/prefect https://docs.prefect.io/core/idioms/flow-to-flow
    ✅ 1
    k
    n
    +3
    • 6
    • 8
  • m

    Mars

    07/27/2022, 9:08 PM
    Hi, what is the suggested way to set a flow configuration value, like a sandbox API URL, across a flow-of-flows? We are currently using Parameters, but it appears I can only set paramters for the parent flow in the UI. I can’t directly set the child flow parameters, too.
    ✅ 1
    n
    • 2
    • 3
  • a

    Anna Geller

    07/27/2022, 9:27 PM
    Prefect 2.0 is live! 🎉
    :thank-you: 8
    🎉 9
    👀 7
    🙌 24
    k
    • 2
    • 1
  • a

    Andreas Nigg

    07/27/2022, 11:16 PM
    Hey, congrats to your release!!! I'm struggling to update my KubernetesJob deployments. (Prefect 2.0 Cloud user, updated my agents, CLI and KubernetesJob images to 2.0.0-python3.9) I ran the following command
    prefect deployment build ../flows/r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -i kubernetes-job
    If I then run "prefect apply..." and "prefect run..:", the job gets created, however it fails with:
    Flow could not be retrieved from deployment.
    FileNotFoundError: [Errno 2] No such file or directory: '/home/andreas/github/r_automation/flows/r_script_automation.py'
    Note, that the directory /home/andreas.... is my local flow location, where I created the deployment from. It seems that somehow the job is not downloading the flow from GCS? My GCS storage block configuration is correct (I guess), because all my subprocess-flows work - only the KubernetesJobs have this problem. Any tips for me? 🙂
    :thank-you: 2
    🙌 2
    ✅ 2
    k
    t
    • 3
    • 15
  • s

    Seth Coussens

    07/27/2022, 11:38 PM
    Anyone have any examples of running a DockerContainer deployment in Prefect 2.0 GA? Non of the documentation is correct that is referenced in the migration guide and even the tutorial I found with the correct code had the wrong CLI commands, and the new commands that would appear to be equivalent ignores the Deployment() function in the script.
    ✅ 1
    k
    i
    • 3
    • 9
  • c

    Charles Phares

    07/27/2022, 11:40 PM
    Still a Prefect NooB! I am failing to find a simple example of what I am trying to do with Prefect. It seems like it would be not very much code! Firstly I want to run a query in a file against a Snowflake database. Secondly, I'd like to take the data returned by the snowflake query and insert it into a postgres table. Seems simple, right?
    ✅ 1
    k
    b
    • 3
    • 9
  • s

    Slackbot

    07/28/2022, 12:49 AM
    This message was deleted.
    c
    a
    • 3
    • 11
  • k

    Kelvin

    07/28/2022, 2:09 AM
    Hi and congrats on the 2.0 GA release! I can see that the pypi package for Prefect has been updated to 2.0 - can I ask will the conda package be updated to 2.0 as well? https://anaconda.org/conda-forge/prefect Thanks!
    🙌 1
    :thank-you: 1
    a
    j
    m
    • 4
    • 8
  • d

    Denys Volokh

    07/28/2022, 3:38 AM
    Hi Community, Could you please help me to understand how can I access KV Store values? I am getting this error
    KV Store operations are only supported while using Prefect Cloud as a backend
    and here is my setup: 1. My flow and run config using KubernetesRun:
    with Flow(
        name="Import Benchmark Index Data",
        storage=Local(
            path="/app/workflows/benchmarks/flow_import_index_data.py",
            stored_as_script=True,
        ),
        run_config=KubernetesRun(
            image="prefect-workflows:latest",
            image_pull_policy="Always",
        ),
    ) as flow:
    
        benchmarks_states = task_fetch_benchmarks_state()
    2. Docker file for prefect-workflows image
    FROM prefecthq/prefect:1.2.4-python3.9
    
    WORKDIR /app
    
    ADD . .
    
    RUN pip install --upgrade pip
    RUN pip install -r requirements.txt
    RUN pip install -e .
    
    RUN prefect backend cloud
    3. Agent configuration:
    ✅ 1
    a
    • 2
    • 1
  • d

    Denys Volokh

    07/28/2022, 3:50 AM
    ✅ 1
  • j

    Jeffrey Lai

    07/28/2022, 5:33 AM
    Hi, Community, I tried to follow the tutorial of Prefect Cloud 2.0. I could run the flow locally and saw the flow run results right there in Prefect Cloud. However, when I created a flow run from my deployment on the cloud, the local terminal session didn't pick up the flow run and kept showing the error below. Any help will be appreciated!
    ✅ 2
    a
    b
    • 3
    • 12
  • v

    Viet Nguyen

    07/28/2022, 6:43 AM
    _from_ prefect.utilities.asyncio _import_ Sync
    hi everyone, I wonder if what is the equivalent thing for this in prefect 2.0 ? I'm trying to implement something similar to
    as_completed()
    from Dask, thank you.
    ✅ 1
    a
    • 2
    • 3
  • m

    Michiel Verburg

    07/28/2022, 7:17 AM
    Hi everyone, super specific question maybe, but at the same time I feel like we’re probably not the only one in this boat. Firstly, congrats on the release! Woke up this morning and now we can celebrate as a team that Prefect Orion that we have been using will now be stable, for which the timing is quite perfect for our team as well since we have not really gone into production yet. With that being said, we stayed with Prefect 2.0b8 since we did not want to update every day of the week since last week or so, but now we feel a little lost in all the changes with I think the last version technically being 2.0b16. A lot of the announcements/tutorials etc. are focused on migration from prefect 1.0 (understandably), and I expect significant changes in the docs since Prefect 2.0b8. However, is there a way to somehow easily see differences in the docs since for example 2.0b8 (or earlier releases), or do we basically have to read through the entire docs again? I am already glad I realized the release notes are still in this file: https://github.com/PrefectHQ/prefect/blob/2.0.0/RELEASE-NOTES.md because previous times this file was linked but this time it was not 😛
    ✅ 1
    a
    • 2
    • 1
  • r

    Rajvir Jhawar

    07/28/2022, 9:22 AM
    Hey prefect team, Great job on getting to 2.0 GA !!!! I had question about the new deployment method is there example where I can easily add a customizations to a K8 job without going into the ui and creating k8 job block? Previously all i had to do was import the k8 job create a deployment and pass my customizations to it.
    ✅ 1
    👀 2
    a
    • 2
    • 2
  • a

    Andreas Nigg

    07/28/2022, 9:26 AM
    Hey, I've encountered an issue or misunderstanding with the prefect 2.0.0 manifests or deployment packages: I've configured a a GCS storage block called gcs-prefect-stprage and used this in the deployment build step. My folder structure is as follows:
    test
      |___ deployment.ymal
      |___ kayak_database_flows.py
    My build command is (run from inside the folder "test")
    prefect deployment build kayak_database_flows.py:integrate_kayak_details --name integrate_kayak_db_details -t rmesbtest1 -sb gcs/gcs-prefect-stprage
    Then i apply the deployment and run the deployment. Everything fine so far, the flow get's picked up by the corresponding agent However, the agent downloads ALL of the content of the GCS bucket. So really all of the content in this bucket, not only the stuff related to this deployment. Is this expected? If so, do I need a separate GCS storage block for any of the deployments?
    👀 1
    ✅ 1
    a
    • 2
    • 9
  • d

    David

    07/28/2022, 10:03 AM
    You guys should put up some more documentation regarding the vertex agent. It is absolutely a fantastic way to run pipelines. Am currently running the vertex agent in a GKE autopilot cluster. The vertex ai pipelines really allows you to scale endlessly while at the same time massively reducing costs.
    🙌 4
    ✅ 2
    a
    • 2
    • 1
  • p

    Paul Lucas

    07/28/2022, 10:14 AM
    Hi. Quick question on flow deployments - I notice if I rename a flow, the previous named flow still exists and the newly named one is created as a new separate flow. Is there any way to unregister/delete the old flow) programmatically?
    ✅ 1
    a
    • 2
    • 2
  • a

    Andreas

    07/28/2022, 10:45 AM
    Hi all and congrats on the new GA release! I have a question: In latest beta versions we were able to have multiple Deployments specified in a yaml file separated with --- and then by running
    prefect deployment create deployments.yaml
    we were able to deploy them all at once in Prefect Server. I have seen that there are a lot of changes in GA compared to latest beta and I took a look at the updated docs about it. However I think that some (important to me) functionality is missing compared to what we had in beta. Is there an easy way in the new GA deployments procedure to have multiple flows in a yaml file or an alternative way to deploy multiple flows at once without having to call
    prefect deployment build ......
    and
    prefect deployment apply deployment.yaml
    manually for each single flow that we have?
    :thank-you: 2
    ✅ 2
    👀 3
    🙌 2
    🎯 1
    a
    c
    n
    • 4
    • 8
  • t

    Tarek

    07/28/2022, 11:21 AM
    Hello Prefect, short question: When I register my flow I use
    labels='prod'
    however the registered flow also uses the id of the docker instance it is built into (See screenshot): Is there anyway to delete that, so that only the labels I give are used?
    ✅ 1
    a
    • 2
    • 3
  • h

    haris khan

    07/28/2022, 11:31 AM
    issue with my api_url prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url 'https://api-beta.prefect.io/api/accounts/xxxxxxxxxxxxxx/workspaces/xxxxxxxxxxxxxxxxxx/block_types/name/Process' Response: {'detail': 'Not authenticated'} For more information check: https://httpstatuses.com/403 ps : changed my account id and workspace id to xxxxx to post the issue here
    ✅ 1
    a
    j
    • 3
    • 3
  • c

    Chu

    07/28/2022, 11:42 AM
    Hi, I have a question about why task args cannot have copy, this seems affect my creat flow run properly using trigger arg See thread for code
    ✅ 1
    • 1
    • 2
  • a

    ash

    07/28/2022, 12:06 PM
    Hi All, I am using prefect server deployed on K8. For creating environment when prefect flows are deployed on k8 pods,I am trying to use conda environment. what command does prefect agent run while starting a pod ? As we are getting
    exit error 2, Instance terminated
    on dashboard and on pod logs we are getting
    No such command "execute"
    a
    • 2
    • 4
  • r

    Riccardo Tesselli

    07/28/2022, 12:09 PM
    hi all, I’ve upgraded to Prefect 2.0.0, now how can I define a block for Azure blob storage? I just see S3 and GCP blocks…
    ✅ 1
    a
    g
    +2
    • 5
    • 14
  • j

    Jehan Abduljabbar

    07/28/2022, 12:26 PM
    Good morning, quick questions 1. can one agent run two different flows in parallel? 2. can 2 different agents run 2 different flows at the same time? 3. is there a limit on the number of agents we can start?
    ✅ 1
    a
    • 2
    • 2
  • n

    Nelson Griffiths

    07/28/2022, 12:59 PM
    Are there any examples of adding loggers to prefect for prefect 2.0?
    ✅ 1
    r
    a
    • 3
    • 3
  • l

    Lucien Fregosi

    07/28/2022, 1:00 PM
    Hello thanks for the hard work prefect 2.0 seems really promising 👏 I did manage to work a very simple hello world in a kubernetes job and have just a question, is that still possible to have the ouptut of a python function in the job run log ? From what i have seen i can’t see the ouptput (picture in thread)
    r
    r
    • 3
    • 8
  • a

    Alix Cook

    07/28/2022, 1:24 PM
    👋 working on getting prefect 2.0 setup for our team, and still learning how to navigate the new docs. Is there a way to set the current flow run to cancelled from within a task or flow?
    ✅ 1
    r
    • 2
    • 2
  • t

    T B

    07/28/2022, 1:32 PM
    Hi thank you for the amazing job on releasing 2.0. I am trying the new deployment system with a
    local file system
    storage/block: my-local-storage. However when I try to build with the parameter "prefect deployment build --storage-block local-file-system/my-local-storage" I get an exception :
    File "/usr/local/lib/python3.10/site-packages/prefect/filesystems.py", line 114, in put_directory
        shutil.copytree(from_path, local_path, dirs_exist_ok=True)
    NameError: name 'from_path' is not defined
    Looking at the code : https://github.com/PrefectHQ/prefect/blob/2.0.0/src/prefect/filesystems.py#L114 it seems the variable "from_path" is indeed not defined.
    ✅ 1
    a
    • 2
    • 8
  • t

    Troels Bjørnskov

    07/28/2022, 1:47 PM
    Where is the documentation for
    prefect_test_harness
    located? It tries to connect to a server, I want to skip that step
    ✅ 1
    👀 1
    b
    a
    • 3
    • 5
Powered by Linen
Title
t

Troels Bjørnskov

07/28/2022, 1:47 PM
Where is the documentation for
prefect_test_harness
located? It tries to connect to a server, I want to skip that step
✅ 1
👀 1
b

Bianca Hoch

07/28/2022, 2:28 PM
Hello Troels, we have some documentation for
prefect_test_harness
here. I believe there is an open issue pertaining to prefect_test_harness attempting to connect to the server as well. I mentioned this to the team, so expect a resolution soon.
t

Troels Bjørnskov

07/28/2022, 2:47 PM
Thank you, Bianca. I found out that when
PREFECT_API_URL
is set in the environment, it will try to connect to the api regardless of
prefect_test_harness
being used.
a

Anna Geller

07/28/2022, 2:48 PM
thanks @Troels Bjørnskov - given there is already an open GitHub issue about it, can we keep the discussion about it there? You can add more info you have found out thanks so much, I appreciate you raising that and I'll ping some engineers to see if someone can take a look at the GitHub issue
t

Troels Bjørnskov

07/28/2022, 2:55 PM
Sure. I have nothing to add, but can definitely reproduce the issue raised.
:thank-you: 1
🙌 1
a

Anna Geller

07/29/2022, 12:33 AM
Zach saved the day, it's fixed now https://github.com/PrefectHQ/prefect/pull/6080
🚀 1
View count: 3