https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • m

    Maria

    06/15/2022, 4:09 PM
    Hi, we are considering to upgrade our prefect server to 2.0. I see that it’s still not production-ready. What’s the timeline that it’ll be ready for production work? What are the consideration if we still do decide to use 2.0?
    k
    k
    +1
    5 replies · 4 participants
  • j

    Jacob Bedard

    06/15/2022, 5:07 PM
    Hi Folks, I'm finding myself in a pickle I was in before, where I'm trying to just pull a secret from my cloud setup, and I did this in my python script in the last place I ran prefect, but I can't get it to work. While I was developing, I was able to simply set the use_local_secrets = False, and I could just pull down a secret, but I can't get it to work anymore. I only see one short blurb about getting cloud secrets in the prefect API docs around secrets. Can someone help me with the right commands? I'm getting a message that it keeps looking for it locally
    k
    9 replies · 2 participants
  • o

    Omar Sultan

    06/15/2022, 5:57 PM
    Hi Everyone, we're using Prefect Server on premise V1.2.2 hosted on Kubernetes, and everything is working perfect. However, every now and then we get the occasional error
    Error during execution of task: ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f32b0d48910>, 'Connection to prefect-apollo.prefect timed out. (connect timeout=60)'))"))
    This seems to affect all running tasks at the time, so for example if I have 2 or 3 tasks running, they all fail at the exact same time and give this error
    k
    11 replies · 2 participants
  • k

    Kevin Kho

    06/15/2022, 6:51 PM
    message has been deleted
  • r

    R

    06/15/2022, 7:00 PM
    Hello! I am trying Prefect2.0 for a simple flow from a docker container. I am getting an error and am stuck trying to resolve it - MissingContextError (“No settings context found”) Is there an explicit initial config required? Can I run flows from within a container? Any help would be really appreciated! Thanks!
    m
    k
    26 replies · 3 participants
  • j

    Joshua Allen

    06/15/2022, 7:18 PM
    Hi. I'm looking at Prefect 2.0. The documentation shows support for S3 for storage. By chance, does this S3 support extend to a self-hosted S3-compliant service, such as MinIO? Otherwise, what are the options for a self-hosted environment? The documentation states that remote storage is required when running in k8s or docker. Would the File Storage on a Remote File System option, such as SMB or NFS, work?
    k
    d
    5 replies · 3 participants
  • j

    Joshua Allen

    06/15/2022, 7:20 PM
    One more question, for Task retries, I see that a retry count and a retry interval are supported. Are you looking to add support for exponential back-off or might that capability already exist?
    k
    1 reply · 2 participants
  • a

    Alex de Geofroy

    06/15/2022, 7:21 PM
    Hi all. Does anyone have a solid guide for getting up and running with Prefect Cloud and Azure Kubernetes Service for agents? I'm finding bits and pieces everywhere, but could really benefit from a comprehensive tutorial. I've found guides specific to using Docker agent on an Azure-based Ubuntu server, but not the same level of detail for a Kubernetes architecture. Thanks!
    k
    1 reply · 2 participants
  • p

    Pedro Machado

    06/15/2022, 7:57 PM
    Hi everyone. I have a problem where my flow is unintentionally exposing secrets in the UI. I am passing secrets to the
    env
    argument of a class derived from
    ShellTask
    .
    bash = LoggedShellTask(stream_output=True)
    
    # more stuff here ...
    
    with Flow(
        FLOW_NAME, storage=docker_storage, run_config=run_config, result=PrefectResult()
    ) as flow:
    
        run_results = bash(
            helper_script=get_helper_script(repo_path),
            command=get_command(ml_script_path, ml_script_arguments, conda_env=conda_env),
            env=dict(
                SNOWFLAKE_USER=snowflake_user,
                SNOWFLAKE_PASS=snowflake_pass,
                SNOWFLAKE_DATABASE=snowflake_database,
                SNOWFLAKE_OUTPUT_SCHEMA=snowflake_output_schema,
            ),
            log_file_path=get_log_file_path(output_dir),
        )
    The issue is Prefect automatically creates a
    List
    and
    Dict
    task that have
    PrefectResults
    by default and this exposes the secrets in the UI. A couple of ideas that come to mind: 1. Assign a specific results objects to each task (not at the flow level) 2. Create a wrapper task that receives all the secrets and returns a dict that is passed to the
    ShellTask
    3. Create a
    ShellTask
    that accepts each secret as a parameter Any suggestions? One feature I'd like to preserve is the ability to restart the flow and I'd rather not persist the secrets anywhere. Thanks!
    k
    4 replies · 2 participants
  • i

    ibrahem

    06/15/2022, 8:42 PM
    Hello, Have any one managed to run Shell Task via SSH ? or submitted a Spark job to a Yarn cluster ?
    ✅ 1
    k
    a
    9 replies · 3 participants
  • j

    Jason

    06/15/2022, 9:03 PM
    Regarding DBT Tasks, where should the models be stored? Can they be subdirectories of the flows/ directory where I have my hello_dbt example?
    k
    5 replies · 2 participants
  • r

    Robert Esteves

    06/15/2022, 9:19 PM
    Hi all, I'm looking for some help with the following error: Failed to load and execute flow run: ModuleNotFoundError("No module named 'pyodbc'") We are running an Ec2 instance with the Prefect Docker Agent. --How do we install the Python modules? Is it needed on the Ec2 instance, (which we did) or is there a way to load it into the container? Thanks for your assistance and support.
    k
    a
    3 replies · 3 participants
  • v

    Victoria Alvarez

    06/15/2022, 10:10 PM
    Hi Everyone. I'm trying to deploy my Prefect project on GCP using Google Cloud Task for BigQuery. Is there any documentation or tutorial with a similar case using GCP and Prefect Orion?
    k
    1 reply · 2 participants
  • s

    Shaoyi Zhang

    06/15/2022, 10:22 PM
    Hi, how can we rename a tenant on Prefect Cloud?
    k
    8 replies · 2 participants
  • c

    CA Lee

    06/15/2022, 11:48 PM
    Hello, does anyone have any experience setting up Prefect Orion to be accessed via a web URL (HTTPs)?
    k
    8 replies · 2 participants
  • f

    Faheem Khan

    06/16/2022, 2:44 AM
    Hi All, In Prefect 2.0(server and agent running locally in docker containers) do I still need to create tenant? it seems some commands from Prefect <2.0 doesn't work in Prefect 2.0.
    k
    1 reply · 2 participants
  • f

    Faheem Khan

    06/16/2022, 5:56 AM
    #Prefect 2.0 Any Idea why I'm getting this error while starting prefect agent in docker. Prefect 2.0?
    a
    12 replies · 2 participants
  • a

    Amro

    06/16/2022, 6:27 AM
    Hello everyone i'm trying prefect for my job and was wondering if someone has a resource that show how i can use the service within AWS please much appreciated.
    a
    1 reply · 2 participants
  • g

    GuangSheng Liu

    06/16/2022, 7:16 AM
    Hello, i want to overwrite this configuration by using environment variables.So which one is correct? Thanks!
    export PREFECT__SERVICES__SERVICES__TOWEL__MAX__SCHEDULED__RUNS__PER__FLOW=1
    or
    export PREFECT__MAX__SCHEDULED__RUNS__PER__FLOW=1
    a
    k
    +1
    13 replies · 4 participants
  • s

    Surya

    06/16/2022, 10:27 AM
    we installed prefect on Google kubernetes cluster
    a
    6 replies · 2 participants
  • r

    Roger Webb

    06/16/2022, 1:30 PM
    Hey Yall. So we have The pattern of a Driver flow that calls child flows. Its working wonderfully where we have the create and the the Waitfor... however as the driver grows to control more flows... our waitfors are starting to error saying that the flow didnt start within 15 seconds. Once this errors... all the dependancy flows dont kick off, so its getting rather annoying. We are setting the number of workers... which we had at 10.. now lowering to 4 just to test it out... With unlimitted flow concurrency, shouldnt our flows be starting faster and not hitting that 15 second wall?
    k
    12 replies · 2 participants
  • e

    Ed Burroughes

    06/16/2022, 2:36 PM
    Hey all, currently we have some reusable tasks that at the minute are getting repeated alot in the code. I was wondering if it's possible to do something like this (currently doesn't work, but wondering if there is way round it), we're using prefect 1:
    def build_full_refresh_base(name, s3_dir_prefix, **flow_kwargs):
        with Flow(name, **flow_kwargs) as flow:
    
            repeat_task_output = repeat_task()
        return flow, repeat_task_output
    
    
    @contextmanager
    def FullRefreshFlow(name, s3_dir_prefix, **flow_kwargs):
        flow = build_full_refresh_base(name, s3_dir_prefix, **flow_kwargs)
        try:
            yield flow
        finally:
            print("do something")
    
    
    if __name__ == "__main__":
        @task(log_stdout=True)
        def some_task(repeat_task_output):
            print(repeat_task_output)
    
        with FullRefreshFlow("hello", "some_dir") as (flow, repeat_task_output):
            some_task(repeat_task_output)
    
        flow.run()
    k
    9 replies · 2 participants
  • j

    Joshua Allen

    06/16/2022, 2:36 PM
    I need to run and wait on the output of a docker container as part of my Prefect flow. I see that in Prefect 1.x there is an API for running a docker container and waiting on its result. I don't see a similar API in Prefect 2. Is that coming? Should we use
    docker
    module for Python instead? Or is there a better way?
    k
    1 reply · 2 participants
  • m

    Matthew Millendorf

    06/16/2022, 3:51 PM
    Hello, I’ve got a bit of an issue that would love some feedback on. There was an initial bug reported in our application where the graphql query to Prefect for a Flow, given the project and flow name, was returning None. Using the Interactive API, I found that the project and flow name for that given Flow existed and got a return. This code has not been touched in months and just recently started failing. Diving further, I noticed our ECS agents have not queried for flows to execute in 3 days, which is when the bug was reported. I restarted our Fargate Services where our ECS agents live, and the agents are still not querying for flows to execute, despite successfully running and ‘waiting for flow runs’. Any idea what’s going on or how to proceed with debugging further?
    k
    20 replies · 2 participants
  • d

    David Yak

    06/16/2022, 4:16 PM
    Which channel is best to ask a newbie question?
    c
    k
    2 replies · 3 participants
  • s

    Slackbot

    06/16/2022, 4:39 PM
    This message was deleted.
    k
    a
    2 replies · 3 participants
  • j

    Josh

    06/16/2022, 5:44 PM
    @Anna Geller or @Kevin Kho Any update on this issue? We’re running into the same issues with no logs appearing when using Prefect with LocalDaskExecutors on a Docker Agent in “processes” mode https://github.com/PrefectHQ/prefect/issues/5769
    k
    5 replies · 2 participants
  • d

    David Yak

    06/16/2022, 5:52 PM
    I feel like I have a basic version problem or something. I created an account in cloud.prefect.io, created an API key and then tried to use it in the CLI on an Ubuntu machine. 'prefect cloud login -k MYKEY' gives me an auth error. Lots of docs say perfect auth login etc but my cli (2.0b6) says auth is not a valid command. Help! What else do you need to know? Am I running the wrong version of CLI?
    k
    m
    3 replies · 3 participants
  • f

    Fina Silva-Santisteban

    06/16/2022, 5:59 PM
    Hi everyone! I’ve registered a few new flows last week, I can’t seem to be able to trigger a flow run for those through GraphQL. (We’re using a rails application to trigger flow run requests, that app uses graphql and a flow’s version group id. It works fine for a bunch of other, previously created flows.) The requests get sent successfully by our app but nothing happens on prefect cloud. Is there somewhere a place where I can look up all flow run requests a tenant has received? I need some kind of logs to be able to debug this 🤔
    k
    8 replies · 2 participants
  • x

    Xavier Babu

    06/16/2022, 7:55 PM
    Hi team, In Prefect Orion, how to specific postgres connection url with specific schema (instead of using public)? I am trying to use postgres instead of SQLite.
    k
    8 replies · 2 participants
Powered by Linen
Title
x

Xavier Babu

06/16/2022, 7:55 PM
Hi team, In Prefect Orion, how to specific postgres connection url with specific schema (instead of using public)? I am trying to use postgres instead of SQLite.
k

Kevin Kho

06/16/2022, 7:57 PM
Hi Xavier, have you seen this page?
x

Xavier Babu

06/16/2022, 7:59 PM
Kevin, I did. But it doesn't have info about how to specify the schema name in the connection URL.
k

Kevin Kho

06/16/2022, 8:01 PM
ah i see what you mean
I’d need to ask the team on this
We don’t design to give our schemas prefect-prefixed names to guarantee against collision with other schemas, because we don’t develop anticipating overlapping databases. You could change the default schema for the db user you’re using for Orion, on the postgres side with:
ALTER USER my_user WITH DEFAULT_SCHEMA = OtherSchema;
Note this is not recommended because it may cause future issues
x

Xavier Babu

06/17/2022, 2:10 PM
Alter user works for me. But I can't create new user for every environment. Choosing different schema is the right way to address this issue.
k

Kevin Kho

06/17/2022, 2:44 PM
Our recommendation here is for things to be separated by different databases (on the same pg server)
View count: 3