• v

    Valentin Willscher

    2 years ago
    Hey guys - is their a way to list all registered flows? I couldn’t find a way in the API docs
    v
    1 replies
    Copy to Clipboard
  • Adam

    Adam

    2 years ago
    Hello everyone! Hope you’re having a great day so far! I need some quick help regarding the API for
    PostgresFetch
    . It seems to require
    user
    and
    host
    arguments when declaring the task and then
    password
    can be supplied when instantiating it. How can I send all when I instantiate it rather? I’m trying to achieve something like this:
    get_customers = PostgresFetch(
        name="Get customers ready for migration",
        db_name="postgres",
        user="CAN_I_USE_ENV_SECRET_HERE?",
        host="CAN_I_USE_ENV_SECRET_HERE?",
        query="""
            SELECT DISTINCT customer_id
            FROM customer_migration_request
        """,
        fetch="all",
    )
    
    
    with Flow("migrate customers") as flow:
        postgres_user = EnvVarSecret("POSTGRES_USER", raise_if_missing=True)
        postgres_password = EnvVarSecret("POSTGRES_PASSWORD", raise_if_missing=True)
        postgres_host = EnvVarSecret("POSTGRES_HOST", raise_if_missing=True)
    
        customers = get_customers(
            password=postgres_password, user=postgres_user, host=postgres_host
        )
    Adam
    Dylan
    8 replies
    Copy to Clipboard
  • i

    Iain Dillingham

    2 years ago
    Hi #prefect-community 👋 Could anyone point me to where the context object is populated? I've found that
    prefect.context.scheduled_start_time
    contains slightly different timezone information depending on whether I run the flow with
    my_flow.run()
    or whether it is run by the Docker agent. • Run with
    my_flow.run()
    the object as follows:
    DateTime(2020, 9, 17, 13, 28, 18, 669528, tzinfo=Timezone('UTC')
    . • Run by the Docker agent the object is as follows:
    DateTime(2020, 9, 17, 13, 45, 0, tzinfo=Timezone('+00:00'))
    . The distinction is important because they are both instances of
    pendulum.DateTime
    . Calling
    <http://dt.to|dt.to>_iso8601_string()
    results in a different string, because if the timezone is UTC, then this method replaces the trailing
    +00:00
    with
    Z
    . Whether or not this is a good idea, it would be good for
    prefect.context.scheduled_start_time
    to return consistent timezone information.
    i
    Dylan
    4 replies
    Copy to Clipboard
  • s

    simone

    2 years ago
    HI #prefect-community I have task mapping on a list of images (~12000) :
    out = func.map([A1, A2, A3, A4, A5, B1, B2, B3, B4, B5])
    in the next step I would like to partially reduce the output and combine only the matching subgroups ex combining 
    A = out[0:5] B =out[5::]
    and then process in parallel 
    A
     andB. I have three questions: (1) If I understood correctly order matters for mapping in prefect so input and output have the same order, correct? (2) I am running the code on a HPC. If I proceed this way will the entire 
    out
     be collected in memory or the different output groups dispatched to the specific worker where the reduce is happening? (3) Is there a more efficient way to do this? thanks a lot!
    s
    Dylan
    14 replies
    Copy to Clipboard
  • Pedro Machado

    Pedro Machado

    2 years ago
    Hi there. I am looking for suggestions on the best way to structure a repo to store Prefect flows. We expect to have several flows that use some shared functionality (example: code to run queries and get data from Presto or to interact with an API). Could you suggest a folder structure that allows us to create flows and import shared code? Also, I anticipate having several sql files that would be run by the different flows. This is one of those "I don't know what I don't know yet" questions. Just trying to set things up right from the beginning. Although we won't do CI/CD from the start, I'd like to have a set up that will allow us to implement CI/CD. in the near future This client uses GitLab. A little more background ... • I am planning to use Prefect cloud. • We'll probably start with a docker agent running on an AWS instance but I am open to suggestions. The workflows will be primarily pulling data, running queries, sending files, etc. (we won't be training ML models, etc.) • The group I am working with has limited devops support and any additional infrastructure takes a while to request, get approved, and provisioned. • Most of the code will be written in Python but they have some legacy R stuff that we have been running inside of a container Thansk!
    Pedro Machado
    Dylan
    12 replies
    Copy to Clipboard
  • ale

    ale

    2 years ago
    Any chance to use Bitbucket instead of Github for storage?
    ale
    1 replies
    Copy to Clipboard
  • s

    sundaram

    2 years ago
    @Marvin
    s
    1 replies
    Copy to Clipboard
  • jars

    jars

    2 years ago
    Prefect Cloud gives our Flow Runs cute names like beautiful-cricket, industrious-turkey, etc. Is this name accessible inside of the Flow's execution? We'd like to tag some external resources w/ this name so we could easily know which Flow made those resources for debugging.
    jars
    Jeremiah
    +1
    11 replies
    Copy to Clipboard
  • h

    Hamza Ahmed

    2 years ago
    @Marvin
  • Marvin

    Marvin

    2 years ago
    Why should I want to make anything up? The contest is bad enough as it is without wanting to invent any more of it. I'll still enter you in it, @Hamza Ahmed