https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
random
  • a

    Apoorva Desai

    03/01/2022, 9:09 AM
    Hello! How do I access the string value in a Parameter object? I'm passing a string Parameter to a flow throw the CLI as 
    prefect run --watch --id flowId --param strparam="paramValue"
     . Then in the flow.py file, I am initializing outside the flow 
    strparam = Parameter("strparam", default "hello")
      but I'm unable to access 
    "paramValue"
     . Prefect only sees this 
    <Parameter: strparam>
    a
    • 2
    • 4
  • s

    Sen

    03/03/2022, 5:06 AM
    Good morning everyone, I would like to know a really basic thing with Docker Agents. I have created a DockerAgent with all the necessary libraries which are needed for the flow to run. And then I started the Docker Agent using a docker-compose file. I can see the below logs:
    [2022-03-03 05:10:35,310] INFO - TranslationEvaluator Agent | Registering agent...
    [2022-03-03 05:10:36,108] INFO - TranslationEvaluator Agent | Registration successful!
    [2022-03-03 05:10:36,498] INFO - TranslationEvaluator Agent | Starting DockerAgent with labels ['On_Prem_Agent', 'IP:<>']
    
    [2022-03-03 05:10:36,498] INFO - TranslationEvaluator Agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
    
    [2022-03-03 05:10:36,498] INFO - TranslationEvaluator Agent | Waiting for flow runs...
    I then went and created a Flow and registered the flow as a docker. But when I try to run the flow from the prefectUI, the flow starts running and I see the below lines in the log of the DockerAgent:
    [2022-03-03 05:20:32,889] INFO - TranslationEvaluator Agent | Deploying flow run 9cf17c39-877c-49e7-aae4-0451e320bae4 to execution environment...
    
    /usr/local/lib/python3.7/dist-packages/prefect/agent/docker/agent.py:151: UserWarning: DockerAgent `docker_interface` argument is deprecated and will be removed from Prefect. Setting it has no effect.
    
      UserWarning,
    
    [2022-03-03 05:20:33,579] INFO - TranslationEvaluator Agent | Completed deployment of flow run 9cf17c39-877c-49e7-aae4-0451e320bae4
    
    [2022-03-03 05:20:34+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'On_Prem_Agent'
    But it fails, with some libraries are missing to do the logic in one of the tasks in the flow. I am under the impression that the agent I created needs to have all the libraries installed and the flow just run on the agent using them. What is the right way to do this, so that I can get this working. Thanks, Sen
    a
    • 2
    • 4
  • a

    Antti Tupamäki

    03/09/2022, 12:52 PM
    Docks prefect give Internal Server Error - Request ID: 01FXQBPXZ25Q9W9CNRZW0A1A37
    a
    r
    • 3
    • 7
  • k

    Kevin Kho

    03/16/2022, 2:04 PM
    I am looking for a hotswappable 65% keyboard with rgb. Any suggestions out there?
    c
    a
    a
    • 4
    • 15
  • a

    Andrea Nerla

    03/17/2022, 11:24 AM
    I just would like to give my appreciation for this platform and community. Unfortunately in a while I'll have to change platform, but this platform and community have been everything I could've asked for and more.
    🙌 5
    :thank-you: 4
    ❤️ 10
    a
    c
    • 3
    • 2
  • m

    Mike Grabbe

    03/17/2022, 3:29 PM
    I'm not sure where to post this feedback, so I'll try here. I'm reading through the Prefect Orion docs, and on the Flow and task configuration page, the Orchestration Policies link under Additional Reading returns a 404 not found error.
    👀 1
    :thank-you: 3
    k
    m
    t
    • 4
    • 4
  • r

    Ramzi A

    03/24/2022, 3:02 PM
    Is there a prefect 2.0 slack channel?
    k
    • 2
    • 2
  • r

    Royzac

    03/24/2022, 6:17 PM
    is there a pandas community channel?
    🐼 4
    k
    e
    • 3
    • 4
  • a

    Abhishek

    03/25/2022, 10:30 AM
    I have a Prefect joke, but it’s “Scheduled”.
    🤣 8
    😄 9
    j
    • 2
    • 1
  • k

    Kevin Kho

    03/28/2022, 2:28 PM
    Anyone going to PyCon next month?
    :marvin: 3
    😛ython: 1
    🐍 1
  • m

    Martin T

    03/28/2022, 3:27 PM
    The flow-run naming generator [adjective]-[animal] really tricked me today! 🤣🤣🤣
    😆 22
  • a

    Andrew Black

    03/30/2022, 3:32 PM
    Hi community, does anyone have a cool Prefect + Great Expectations use case? Looking for an example to include in a blog.
    a
    k
    +2
    • 5
    • 5
  • r

    Royzac

    03/30/2022, 7:36 PM
    Any recommendations on migrations tools? I'm using sqlalchemy but not a fan of Alembic.
    👀 2
    j
    k
    • 3
    • 2
  • r

    Royzac

    04/01/2022, 1:56 PM
    Does Pandas have a bulk update(not insert) functionality with sqlalchemy?
    k
    k
    • 3
    • 9
  • t

    Tim-Oliver

    04/04/2022, 2:39 PM
    Hi! Thanks for Prefect! I was wondering when we could expect map support in Prefect 2? No worries if this can't be answered. Just curious!
    ✅ 1
    k
    m
    +2
    • 5
    • 67
  • p

    Patrick Alves

    04/06/2022, 1:03 PM
    Hi there, I have a local prefect set up thas was running with no error until today. The workflows run in a "local" agent inside a docker container. Today I start to get the error
    ModuleNotFoundError: No module named '/root/'
    (trackback is on the thread) If I access the agent container and run any flow manually (
    python workflows/ad_check/flow.py
    ) it works! Can someone help me?
    a
    • 2
    • 5
  • t

    Tim-Oliver

    04/06/2022, 2:07 PM
    Hello, I have an issue with the
    DaskTaskRunner
    . Somehow it gets stuck on the last task and will not finish. This is my minimum example:
    from prefect import task, flow 
    from prefect.tasks import task_input_hash 
    import time 
    from prefect.task_runners import SequentialTaskRunner, DaskTaskRunner 
     
    @task 
    def list_files(): 
       return ["1.txt", "2.txt"] 
     
    @task 
    def preprocess(in_file: str, target_dir: str): 
       time.sleep(1) 
       return f"{target_dir}-{in_file}" 
     
    @task 
    def segment(in_file: str, target_dir: str): 
       return f"{target_dir}-{in_file}" 
     
    @task 
    def mask_bg(in_file: str, mask: str, target_dir: str): 
       return f"{target_dir}-mask-{in_file}" 
     
    @flow(name="Background Masking", task_runner=DaskTaskRunner()) 
    def background_masking(): 
       files = list_files() 
       preprocessed = [] 
       segmented = [] 
       masked = [] 
       for f in files.wait().result(): 
           preprocessed.append(preprocess(f, target_dir="/preprocessed/")) 
           segmented.append(segment(preprocessed[-1], target_dir="/segmented/")) 
           masked.append(mask_bg(preprocessed[-1], segmented[-1], target_dir="/masked/")) 
            
    if __name__ == "__main__": 
       result = background_masking()
    If I run this locally (no remote file storage) with normal sqlite-db the processing does not finish. With
    SequentialTaskRunner
    or
    ConcurrentTaskRunner
    the script completes.
    a
    k
    • 3
    • 27
  • i

    Ievgenii Martynenko

    04/08/2022, 9:53 AM
    How much does it usually take Sales to reach a person, who is interested in buying Cloud version?
    a
    • 2
    • 3
  • g

    Gaurav kumar

    04/14/2022, 7:43 AM
    Hi, I want to acheive a flow where i can controll a task in running flow, actually my problem statement is like event based, for me its like when a event will start it will have some timeout time and if the event completes before that timeout then i should move to next task
    a
    • 2
    • 1
  • t

    Tim Wright

    04/19/2022, 9:16 PM
    I apologize if this has been asked before. Is there any way that I can return a dictionary from one Task and use it as the input kwargs for another task? More specifically I want a task that returns a Dictionary and I want to pass its result to another task using variable unpacking...
    dict_result = some_task()
    some_other_task(**dict_result)
    Is this doable? Is there a more Prefect-y way to accomplish this without creating a Parameter for eahc element of the dictionary? I ultimately want to pass as an input a list of dictionaries which I can then operate over using Prefect's mapping, so being able to define the collection of arguments together is important.
    k
    • 2
    • 11
  • j

    Jon Ruhnke

    04/21/2022, 4:36 PM
    This might be random, but I understand the use of Prefect for developing/orchestrating data pipelines for data analytics/reporting. Do people also use Prefect (or open source in general) for Enterprise application/information/system integration (essentially synchronizing information between two operational systems)? When I try to research this topic all I can find are advertisements from PaaS/SaaS companies trying to sell their all-in-one solutions.
    ✅ 1
    👀 1
    a
    • 2
    • 5
  • k

    Kevin Kho

    04/21/2022, 11:01 PM
    I posted before but noone really responded. Anyone going to PyCon next week? 😛ython:
    👀 1
  • a

    Anna Geller

    04/22/2022, 4:54 PM
    Hi everyone, happy Friday! We're gonna post programmer jokes and memes every Friday - starting today! :catjam: If you want to join us in this madness, feel free to post something, too! Have a great weekend! :marvin:
    😂 7
    🥲 5
  • l

    Laurie Hindes

    04/22/2022, 4:57 PM
    😂 5
    🌑 1
  • k

    Kyle McChesney

    04/22/2022, 5:08 PM
    https://twitter.com/shitPM/status/1517534514905554945
    😭 2
    a
    c
    +3
    • 6
    • 9
  • j

    Jeremiah

    04/22/2022, 5:33 PM
    All I can do is put on a brave face and learn from @Laurie Hindes
    😂 2
    👏 2
  • k

    Kevin Mullins

    04/22/2022, 8:40 PM
    👏 4
    💯 6
  • g

    George Coyne

    04/22/2022, 9:34 PM
    😂 4
    j
    j
    j
    • 4
    • 5
  • g

    Gustavo Puma

    04/26/2022, 2:01 PM
    Hi peeps 👋 At my company we're trying to improve the data quality of our (Databricks) Delta Lake by introducing monitoring. We'd like to add assertions (as SQL statements) that we could then schedule and alert on (based on the query results). Any of you familiar with such tools that you could recommend❓ Databricks has its own thing called SQL analytics but unfortunately it doesn't have any form of git integration
    :discourse: 1
    a
    j
    r
    • 4
    • 8
  • n

    Noah Holm

    04/28/2022, 6:54 AM
    Hearing about the Prefect team’s open source sponsorship budgets in https://www.prefect.io/blog/why-prefect-invested-in-textualize/ I started thinking about where I would direct money if my manager told me I had some to contribute. My spontaneous thought was Ajv, used for Node-based JSON schema validation (we’re using it in our event collection pipeline). Where would your money go?
    ❤️ 1
    🤔 2
    🔥 9
    🤔 3
    a
    j
    • 3
    • 2
Powered by Linen
Title
n

Noah Holm

04/28/2022, 6:54 AM
Hearing about the Prefect team’s open source sponsorship budgets in https://www.prefect.io/blog/why-prefect-invested-in-textualize/ I started thinking about where I would direct money if my manager told me I had some to contribute. My spontaneous thought was Ajv, used for Node-based JSON schema validation (we’re using it in our event collection pipeline). Where would your money go?
❤️ 1
🤔 2
🔥 9
🤔 3
a

Anna Geller

04/28/2022, 11:09 AM
I'm personally intrigued about tools in those two areas: • data management and quality • data infrastructure - tools that try to make it easier for data professionals to manage code dependencies and not have to worry about infrastructure, Docker images, and things like CI/CD Alternatively, the budget could go to... 😛refect2:
:prefect2: 3
j

Jeremiah

04/28/2022, 1:23 PM
Great question 😄 it’s also interesting to consider relative impact — definitely important to sponsor the larger projects that power business outcomes, to ensure they have the resources they need to keep the project healthy, but there’s also those diamond-in-the-rough projects that maybe have only one maintainer or haven’t gained a ton of visibility, but sure are useful. Sponsorship can really make a critical difference there!
🤙 2
View count: 7