https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • j

    Jenia Varavva

    08/01/2022, 6:12 AM
    I really enjoyed the file-less workflow of uploading flows to the Prefect 1.0 server and having them execute in the k8s infrastructure (via the agent, as k8s jobs). It seems it’s not possible anymore with Orion, is it? I don’t fully understand the benefits of the json/yaml files as opposed to the former API. Should I really embrace calling CLI and dealing with files as it is a superior approach? One obstacle in my workflow is I’m doing interactive work in a Jupyter kernel, running in the cluster, so getting a shell at the same host is not straightforward
    :upvote: 4
    • 1
    • 1
  • j

    Jenia Varavva

    08/01/2022, 6:25 AM
    Is there a public roadmap for future Prefect development and features?
    👀 2
  • a

    Anat Tal Gagnon

    08/01/2022, 7:42 AM
    Hi guys, I'm just getting started with prefect 2.0 in docker. I mounted file into /flows folder - but unfortunately I can't see it in the flows section in the UI... what am I missing?
    root@24dc674d1f7d:/flows# cat flow1.py
    from prefect import flow
    
    @flow
    def my_favorite_function():
        print("What is your favorite number?")
        return 42
    
    print(my_favorite_function())
    ✅ 1
    a
    m
    • 3
    • 42
  • b

    Beizhen

    08/01/2022, 7:47 AM
    Hi community, Lately we have noticed the following error that occasionally caused failure for multiple unrelated flows (here is log example for one):
    14:25:23 INFO Submitted for execution: Job prefect-job-2444a873
    14:25:49 ERROR Failed to load and execute flow run: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='<http://www.scorpiotankers.com|www.scorpiotankers.com>', port=443): Max retries exceeded with url: /our-fleet/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efdbc010d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"))
    14:26:00 INFO Beginning Flow run for 'Destination Predict'
    14:26:00 DEBUG Using executor type LocalDaskExecutor
    14:26:00 INFO Flow run has already finished.
    14:26:32 ERROR k8s-infra Pod prefect-job-2444a873-8887r failed.
    	Container 'flow' state: terminated
    		Exit Code:: 1
    		Reason: Error
    The specific error seems to reflect code from another unrelated flow. Any idea how one flow can throw an error in several other completely unrelated flows? before even the flow starts?
  • t

    Toby Rahloff

    08/01/2022, 8:09 AM
    Hi, happy 2.0 GA 🎉 What is your opinion on how to structure Prefect 2.0 code within an organization? One big mono-repo? One repo per Flow? One repo per business value stream? Would love to hear your experience and perspectives! 😛refect-duck:
    👀 2
    a
    • 2
    • 3
  • o

    Oscar Björhn

    08/01/2022, 8:50 AM
    I've spent a few days trying to get accustomed to 2.0 after running 1.0 for over a year and a half. It looks amazing so far! However, there are two things I'm still trying to wrap my head around: Docker containers and storage. I've been trying to learn as much as I can from the documentation and this slack workspace but it's still a bit unclear. Previously, we have been building each flow as its own docker container and we have never specified any sort of local/remote storage beyond setting flow.storage to Docker. I suppose this is no longer supported in 2.0? The impression I'm getting from reading the documentation and slack is that people generally build a few base images, upload it to their container registry (Azure, in our case) and then store the flow .py files themselves in a remote storage location. Is this correct? If so, since I'm using Azure, can I even move forward with the migration right now or do I have to wait for blob storage support? In case it makes a difference, we run our agents in VMs and we're primarily using the cloud versions of the web service, other than for local development.
    ✅ 1
    b
    m
    a
    • 4
    • 6
  • m

    Marius Haberstock

    08/01/2022, 9:19 AM
    What I really liked about Prefect 1.0 was the
    log_to_stdout
    Parameter which automatically logs all print statements. Is there something similar for Prefect 2?
    ✅ 1
    a
    j
    • 3
    • 4
  • m

    Martin T

    08/01/2022, 9:42 AM
    Is the Prefect 2.0 sign up page broken?
    ✅ 1
    a
    • 2
    • 2
  • m

    Mohamed Alaa

    08/01/2022, 10:41 AM
    Hello guys! I've just found out about prefect not long ago and currently it a top candidate for our data workflow and pipeline. While i was going through the documentation, it stated that prefect 2.0 is under active development and prefect 1.0 is recommended for production use. Since prefect was officailly released a couple of days ago and is now the default installing package for pip, is it still true that prefect 2.0 is not recommended for production use? Is there also a way to visualize my workflow like in prefect 1.0's
    flow.visuallize()
    And also im really loving prefect and cannot wait to start working with it and exploring Thanks alot
    o
    a
    • 3
    • 7
  • p

    Pierre Monico

    08/01/2022, 11:01 AM
    Is it possible that some errors are not caught by Prefect when running multi-threaded code? PS: wasn’t sure if this is the right place to post since #prefect-server was archived.
    t
    • 2
    • 3
  • r

    Riccardo Tesselli

    08/01/2022, 11:51 AM
    hello, I’m struggling with loading a custom block from Prefect 2.0 cloud. I’ve created on cloud a block called
    test
    of a custom block type
    Config
    , and when trying to load its values within a flow with
    Config.load('test')
    I get this error:
    except prefect.exceptions.ObjectNotFound as e:
    >               raise ValueError(
                        f"Unable to find block document named {block_document_name} for block type {block_type_slug}"
                    ) from e
    E               ValueError: Unable to find block document named test for block type config
    a
    • 2
    • 17
  • e

    Eli Treuherz

    08/01/2022, 11:55 AM
    Hi, I’m struggling to find this in the docs for Prefect 2.0, does anyone know what the story is with breadth- vs depth-first execution? Is it automatically DFE for Dask and BFE for everything else, like 1.0?
    t
    • 2
    • 6
  • s

    Scott White

    08/01/2022, 12:13 PM
    Hello, I'm having a very frustrating problem with running prefect on our Google Cloud Platform Kubernetes Cluster and and hoping you can help me out. In short, a flow that was running fine on Thursday of last week suddenly gave vague errors on Friday when I tried to push an update. Is there a way I can get more details on the error from Kubernetes so I can debug this more efficiently? (logs and more examples will be posted in the replies shortly).
    n
    c
    w
    • 4
    • 9
  • r

    Rio McMahon

    08/01/2022, 1:46 PM
    Hi - is there still a cloud based key/value store in prefect 2.0? I am having trouble finding it in the docs. Thanks.
    a
    s
    j
    • 4
    • 6
  • b

    Billy McMonagle

    08/01/2022, 2:29 PM
    Good morning, 2.0.1 question here: I am trying connect a Kubernetes OrionAgent to Prefect Cloud, and having some trouble.
    ✅ 1
    t
    j
    +2
    • 5
    • 42
  • c

    Chris Reuter

    08/01/2022, 2:31 PM
    Posting here for visibility: Prefect 2.0 interactive livestreams 3 times this week! https://prefect-community.slack.com/archives/C036FRC4KMW/p1659364260962609
  • d

    Dominik Wagner

    08/01/2022, 3:19 PM
    Prefect 2.0.1 | Scheduling Hi! 👋 I’m trying to figure out how to set up a schedule (not using the UI). Is there a way to do it within the
    prefect deployment build…
    command? i.e. i have something like this in a bash script:
    prefect deployment build flows/dbt_flow.py:dbt_primary_flow \
    --name prod_flow \
    --tag dbt \
    --tag prod \
    --tag daily \
    -sb gcs/gcs-block \
    -ib kubernetes-job/k8s-block
    And I’d like to add something like
    --schedule "cron/0 7 * * */Europe/Berlin"
    If that’s not possible I guess I have to manually edit the deployment.yaml - if that’s the case, can I just put a cron expression in there, or do I need something else? I can’t find any examples in the docs, but maybe I’m just blind? 🙃
    👍 1
    ✅ 1
    j
    b
    • 3
    • 9
  • f

    Florian Kühnlenz

    08/01/2022, 3:26 PM
    The links to the interface and the kubernetes flow runner appear to be broken in https://docs.prefect.io/tutorials/kubernetes-flow-runner/
    c
    j
    +2
    • 5
    • 9
  • s

    Stephen Herron

    08/01/2022, 3:31 PM
    Hi I’m trying to convert some v1 flows to v2 - just some simple snowflake queries at the moment but I sometimes get this error:
    RuntimeError: Set changed size during iteration
    Worker information:
        Approximate queue length: 0
        Pending log batch length: 0
        Pending log batch size: 0
    I’m not trying to do anything with async but that appears to be where this is coming from (I guess it’s used via the
    snowflake-prefect
    integration?
    n
    • 2
    • 4
  • c

    Chu

    08/01/2022, 3:51 PM
    Having a question for rename create_flow_run task, I think I post to the wrong channel https://prefect-community.slack.com/archives/C0192RWGJQH/p1659360778884719
    ✅ 1
    j
    r
    • 3
    • 4
  • h

    Hafsa Junaid

    08/01/2022, 4:24 PM
    In prefect 2.0, whats the alternative command of
    $ prefect backend server
    ✅ 1
    t
    r
    +2
    • 5
    • 9
  • o

    Oleg Sheyner

    08/01/2022, 4:52 PM
    Hi everyone, I installed Prefect in a (mini)conda virtual environment on Windows Server 2019. Attempting to run a local flow gives this error, any thoughts? sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table _alembic_tmp_deployment already exists [SQL: CREATE TABLE _alembic_tmp_deployment ( id CHAR(36) DEFAULT (( lower(hex(randomblob(4)))
    ✅ 1
    👍 2
    n
    • 2
    • 3
  • e

    Eli Treuherz

    08/01/2022, 5:19 PM
    Hi again, is there no longer a way to access flow-related context values inside a task? I’ve looked at this thread and it doesn’t look like it. In 1.0 I saved the flow run name as part of the metadata of my flow’s output products for traceability. I guess I can recapture this by grabbing the values in the flow and passing them into the task as a parameter, but wanted to know if I was missing something obvious
    ✅ 1
    n
    • 2
    • 1
  • m

    Matthew Millendorf

    08/01/2022, 5:34 PM
    Hi, I have a pipeline where tasks need to be executed in their own containers. What is the best way of doing this? Is it to make each ‘task’ that needs to be executed its own Flow so that I can specify the image to be used, thus creating a Flow of Flows?
    n
    • 2
    • 2
  • y

    Yupei Chen

    08/01/2022, 5:40 PM
    I am trying to add a s3 bucket block in cloud 2.0, however after clicking the add + button, I get a blank screen.
    ✅ 1
    m
    • 2
    • 4
  • j

    Joe Goldbeck

    08/01/2022, 5:47 PM
    Hi! I am using Prefect 1.0, and am using different projects to support different deployment environments. Are there any suggested ways to isolate settings (e.g. secrets, env vars, schedules, …) between different projects?
    r
    • 2
    • 2
  • x

    Xavier Witdouck

    08/01/2022, 5:56 PM
    Our Local Agents are up and running with the correct labels, yet they are not picking up jobs. The agent screen shows they are querying, and indicates many late runs for the agent. I have tried to restart the agent, but to no avail. There is no other hint as to what the issue might be. When I try a quick run of a flow, that does get executed. Any idea what is going on here?
    👀 1
    b
    a
    • 3
    • 53
  • h

    Hafsa Junaid

    08/01/2022, 6:08 PM
    Is prefect 1:00 not functional anymore? Also, how do we use prefect 2.0 on cloud?
    a
    • 2
    • 1
  • g

    Gabriel Milan

    08/01/2022, 6:18 PM
    Hi there! I wonder if there's a way of setting specific flows not to submit late runs. Thing is some of our flows, if submitted from "Late runs" might give us trouble
    r
    • 2
    • 4
  • y

    Yupei Chen

    08/01/2022, 6:37 PM
    Issue with using an S3 block:
    from prefect.filesystems import S3
    
    s3_block = S3.load("dev")
    ModuleNotFoundError: No module named ‘s3fs’ ImportError: Install s3fs to access S3 RuntimeError: File system created with scheme ‘s3’ from base path ‘s3://vodori-prefect-orion’ could not be created. You are likely missing a Python module required to use the given storage protocol. An exception occurred. On Prefect 2.0.1
    r
    r
    +3
    • 6
    • 12
Powered by Linen
Title
y

Yupei Chen

08/01/2022, 6:37 PM
Issue with using an S3 block:
from prefect.filesystems import S3

s3_block = S3.load("dev")
ModuleNotFoundError: No module named ‘s3fs’ ImportError: Install s3fs to access S3 RuntimeError: File system created with scheme ‘s3’ from base path ‘s3://vodori-prefect-orion’ could not be created. You are likely missing a Python module required to use the given storage protocol. An exception occurred. On Prefect 2.0.1
Resolved with pip install s3fs, however got this error during installation
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.24.19 requires botocore<1.28.0,>=1.27.19, but you have botocore 1.24.21 which is incompatible.
r

Rob Freedy

08/02/2022, 1:53 AM
Hey Yupei! Are you using a dependency manager? Would you be able to output the python packages you have installed on your system?
y

Yupei Chen

08/02/2022, 2:00 AM
All I use is pip. Packages attached.
req2.txt
r

redsquare

08/03/2022, 10:58 AM
Just had this same issue
r

Rob Freedy

08/03/2022, 1:02 PM
are you able to install the specific botocore package using
pip install botocore==1.27.19
? Was this dependency previously installed in your environment?
r

redsquare

08/03/2022, 1:13 PM
I did pip install s3fs
r

Rob Freedy

08/03/2022, 9:42 PM
@Marvin open "s3fs dependency issue in filesystems"
m

Marvin

08/03/2022, 9:42 PM
https://github.com/PrefectHQ/prefect/issues/6219
r

Rob Freedy

08/03/2022, 9:43 PM
I have opened a git issue here for this dependency issue, you track it's progress using the link above^
c

Chris Pickett

08/03/2022, 11:35 PM
Hi @Yupei Chen! Looking into this issue, prefect doesn’t currently manage boto or related requirements. It did during the 2.0 beta, so it’s possible that you picked up some incompatible requirements during that period. I’m able to get my local environment in the same state as yours and to get out of it I’d suggest uninstalling s3fs and gcsfs (assuming you got into this state because of the beta requirements) and upgrade the boto related packages:
pip uninstall s3fs gcsfs
pip install --upgrade aiobotocore boto3 aiobotocore
pip install s3fs
Alternatively, you could install an older version of s3fs that’s happy with your version of the boto packages which appears to be version `2022.5.0`:
pip install s3fs==2022.5.0
j

James Brady

08/12/2022, 11:42 AM
@Rob Freedy where should these
pip
commands be placed? I ran into the "No module named ‘s3fs’" error when running a flow in a kubernetes deployment – using the default docker image etc.
For right now, I'm going to use @Anna Geller’s suggestion here to create a custom image and pop it in ECR
👍 2
View count: 15