https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-contributors
  • l

    Laura Lorenz

    11/20/2020, 9:36 PM
    It was nice to see everyone again!! :)
    😄 5
  • j

    Josh

    01/15/2021, 10:09 PM
    Not sure if this is the right channel for submitting PRs. But I have one to add chunking to GCS to enable handling of larger files or slower internet connections. Not sure how to best add tests for this in CI https://github.com/PrefectHQ/prefect/pull/3968
    🙌 3
    n
    m
    • 3
    • 6
  • j

    Jack Sundberg

    02/22/2021, 2:05 AM
    Hey everyone! I just opened a pull request and could use some help with updating a few CLI tests -- if anyone is able to help out, I would greatly appreciate it! https://github.com/PrefectHQ/prefect/pull/4146
    💯 1
    m
    • 2
    • 5
  • d

    davzucky

    03/04/2021, 10:54 AM
    Hey, I'm writing a set of new tasks to interact with MongoDB. looking at other database provider I can see they are no consistency and I want to have the community input 1. Password : Some task like postgresql or mysql are passing the password as a param to the task, whereas Redis for example is getting the password out of the Prefect secret. What is the recommended way of doing it ? 2. db connection string vs individual params : With Mongo I use usually only a single parameter connection string which looking like mongo://username:passwordf@db0.example.com:27017,db1.example.com:27017,db2.example.com:27017/?replicaSet=myRepl for example. However all the db tasks are using distinct params. Any recommendation ? Anything else I need to be aware about ?
    🚀 1
    m
    • 2
    • 1
  • a

    ale

    03/27/2021, 4:45 PM
    Hey folks 🙂 Recently I've integrated SodaSQL with Prefect using a custom Task. Here you can find the docs https://docs.soda.io/soda-sql/documentation/orchestrate_scans.html This is just a basic implementation though. I'm thinking about to implement Soda tasks into Prefect core tasks and I'd like to understand if you have any idea or use case that you want like to address with Prefect regarding Data Quality monitoring
    🚀 1
  • a

    ale

    03/29/2021, 2:54 PM
    Hey folks 😒imple_smile: I’ve opened this PR https://github.com/PrefectHQ/prefect/pull/4308 Tests are failing due to missing header files for
    pyodbc
    Can someone help me fix it? Thanks!
    m
    • 2
    • 2
  • j

    Jonathan Wright

    03/30/2021, 4:00 PM
    I am doing some final testing of an extension to the Bitbucket storage class which adds support for Bitbucket cloud. When I register a flow using my new storage class a new
    workspace
    variable is not showing in the UI. I have also shown that when the flow runs, in an environment with the same version of the storage class, the
    workspace
    variable is None. I assumed my storage class would be pickled and stored with the flow and I wouldn’t need to make any code changes outside of this file. What am I missing? https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/bitbucket.py
    m
    m
    • 3
    • 9
  • a

    ale

    04/01/2021, 7:45 PM
    Hey folks 😒imple_smile: Regarding ShellTask, wouldn’t it be useful to return the logs generated by the shell command as part of the FAIL message when the task fails? Maybe we can add a parameter that keeps the current behaviour as a default to avoid breaking changes. I would find it very useful for return useful details when a DbtShellTask fails… What do you think?
    j
    m
    • 3
    • 11
  • j

    Jacopo Tagliabue

    04/05/2021, 10:41 AM
    hi folks, I"m Jacopo - I built a very small POC on using the ShellTask as a base class to run Metaflow flows as part of a Prefect DAG (
    <https://prefect-community.slack.com/archives/CL09KTZPX/p1617381793056800>
    ). If there is any interest in having MF tasks as part of Prefect task library, I'd be happy to give it some actual love and fix a bunch of things here and there 😉 Let me know what you think!
    d
    • 2
    • 7
  • s

    Sean Talia

    04/06/2021, 2:27 PM
    hey all, this morning some changes that I made to the
    ShellTask
    got released, but it looks like my docstring edits broke the way the documentation renders, as you can see here – is the issue that I needed to put the
    Raises
    block above the
    Example
    block in the docstring? thanks for your help! apologies if this is causing anyone any difficulty
    👀 1
    z
    • 2
    • 6
  • m

    Michael Adkins

    04/07/2021, 3:10 PM
    Hi community 🙂 I improved the return value of the
    StartFlowRun
    Task at https://github.com/PrefectHQ/prefect/pull/4245 and have some minor concerns about it being a breaking change. I expect that nobody was using the return value because it was undocumented but I'm curious if anyone here was? Let me know what you think!
    👏 2
    b
    • 2
    • 1
  • m

    Michal Zawadzki

    04/13/2021, 9:46 AM
    Hi guys 🙂 I need connector/extractor tasks (to load data into a data lake) for several API services, starting with GoogleAds task group for interacting with the Google Ads API (connecting, pulling data). Thinking also about integrating Hubspot, Microsoft Ads, Facebook Ads, DV360, Supermetrics, and other marketing data sources. I'm writing it here so maybe I can get some guidance or if there's already some effort which I could join?
    k
    • 2
    • 9
  • s

    Sean Talia

    04/26/2021, 1:29 PM
    hi all – if I'm digging through some source code and notice that on an already-merged PR something was omitted that maybe should have been included in it, is there a preferred way to bring that to the community's attention? I'm thinking either to bring it up in this channel, the #prefect-community channel, or to actually comment on the since-merged PR. Is there a standard for this kind of thing?
    k
    • 2
    • 9
  • e

    Enda Peng

    04/28/2021, 9:07 PM
    Report a typo in error log
    AttributeError: 'FunctionTask' object has no attribute strftime. Did you call this object within a function that should have beendecorated with @prefect.task?
    been decorated
    j
    j
    • 3
    • 4
  • a

    ale

    04/29/2021, 3:30 PM
    Hey folks 🙂 Are there any plans to add data lineage to Prefect? Maybe an integration with Marquez/Datakin?
    m
    j
    • 3
    • 6
  • m

    Matt Drago

    05/01/2021, 10:25 PM
    Hey Folks,  I'd like to start contributing some tasks that I have been working on (or enhancing) but I'm very new to the python development scene.  I've followed the first few instructions that you have for contributing on https://docs.prefect.io/core/development/overview.html and get a clean check out of a fork of the repository (I'm on the master branch).  I've encountered a few problems: 1. when running `pip install -e ".[dev]" I get a few bdist_wheel command not found. 2. running
    pytest
    results in 15 Failed tests I figure that I have missed some steps. I'll include what I have done in the thread.
    :marvin: 1
    k
    m
    • 3
    • 10
  • j

    Jason Prado

    05/05/2021, 5:28 PM
    I have a few Airtable tasks I’ve written and reused a few times. They snarf a whole Airtable table into a DataFrame, apply updates, and insert new records. The existing Airtable tasks in prefect only insert/read records a row at a time. I’d be interested in contributing them. Here’s the source: https://gist.github.com/jasonprado/0c71196bf21398d3b79123b9172987dd I would guess that most people want to read records from Airtable into python dicts/lists rather than DataFrames, right? If I wrote these classes that way but still wanted to use DFs in my project, would I then write my own tasks that turn the dicts/lists into DataFrames? Or should the tasks take a parameter to specify what kind of output they want (dicts vs DFs)? Just curious what the most idiomatic way to publish these tasks would be.
    👀 1
    k
    • 2
    • 1
  • i

    itay livni

    05/24/2021, 12:31 AM
    Hi - Just a heads up the API Reference displayed on the documentation website is displaying
    0.14.18
    not
    0.14.19
    .
    k
    • 2
    • 1
  • m

    Michael Wheeler

    05/25/2021, 12:35 PM
    Hi everyone, I'm looking to see how people feel about handling this issue: https://github.com/PrefectHQ/server/issues/254 TLDR: there are quite a few places in the Prefect Server code where it's hard-coded to use http, which means that deployments will inexplicably break in an environment where https is the default. I feel like it would be an easy fix to simply add an additional configuration value, but @Kevin Kho encouraged me to solicit feedback here before I start work on a PR.
    m
    • 2
    • 7
  • m

    Michael Wheeler

    05/25/2021, 12:50 PM
    I'd also like feedback on another issue I ran into, detailed in this comment: https://github.com/PrefectHQ/prefect/issues/4049#issuecomment-847407701 TLDR: more non-configurable behavior for different deployment styles of Prefect Server. This time it's preventing the initialization of a tenant at a Prefect API endpoint other than
    localhost
    — specifically because the Prefect Client is only written to use the Cloud configuration, even when
    prefect backend server
    is specified. A potential fix would have the Client read the backend configuration and adjust its initialization accordingly.
    m
    • 2
    • 1
  • o

    Octopus

    06/01/2021, 8:03 PM
    Hi i would like to contribute on the prefect project, I would like to know what is the best way in order to try its code; I want to try to had a new command in the CLI so i’ve created a new python file but how to compile the code and thus test this new command ? How do you work in a dev mode ? thx
    k
    m
    • 3
    • 4
  • z

    Zach Schumacher

    06/14/2021, 3:07 PM
    hi all, I wrote a little func for testing flow imports. I found this helpful and had written something similar in our airflow app (RIP) for testing DAG imports. We do some dynamic code generation w/ jinja, so we use this just to make sure none of the files are malformed. Wanted to drop it here in case anyone else finds it useful, and also if anyone thinks this would be worth PRing into a testing module or something.
    def flow_imports_tester(root_path: Path, module_dot_path: str):  # pragma: no cover
        """
        Helper for testing all flows in a project root_path at module_dot_path are importable
        Example:
            >>> flow_imports_tester(Path("~/my_project").expanduser(), "my_project.my_directory.flows")
        """
        split_dot_path = module_dot_path.split(".")
        package_directory = Path(root_path, *split_dot_path)
    
        flows = list(package_directory.iterdir())
        if not flows:
            raise RuntimeError("No flows found, did you specify the correct path?")
    
        for flow in flows:
            if "." in flow.name:
                filename, file_extension = flow.name.split(".")
            else:
                filename, file_extension = flow.name, ""
    
            if filename == "__init__" or file_extension != "py":
                continue
    
            import_module(".".join([module_dot_path, filename]))
    :marvin: 3
  • s

    sean williams

    06/19/2021, 12:13 AM
    Apologies in advance for etiquette because I’m totally new here. There’s a bug in the tutorial docs where a link leads to a 404. I looked for the issue in existing Issues and didn’t see it. I went to report it and it gave me the option of “chat in slack” along with “report a bug”. So, my question is would it be more appropriate to create an issue in GitHub, or describe it here?
    k
    • 2
    • 10
  • f

    Fei Meng

    08/17/2021, 10:47 PM
    Hi there, we are an autonomous driving startup and would like to explore using Prefect for some of the workflows we are managing. Where can I find someone from the sales/engineering team? Thank you!
    k
    k
    • 3
    • 5
  • d

    Dmitry Kuleshov

    08/20/2021, 1:10 PM
    Hi! We have some question not covered by the documentation (or we can't find it). And general questions on how we can use prefect. Where we should ask those questions (where we'll get answers 🙂), or may be we can have a zoom session with someone who can help?
    k
    • 2
    • 3
  • n

    Nitay J

    08/23/2021, 1:50 PM
    hi not sure if this is right channel - my use case requires a clock/schedule that supports RRules (iCal style recurrences) which the current clock impls don’t support. I’ve implemented an
    RRRuleClock
    and
    RRuleSchedule
    (which is really just a function similar to other Prefect
    Schedule
    creators). however the flow won’t run b/c it doesn’t know how to serialize my
    RRuleClock
    . I assume I need to write a
    RRuleClockSchema
    so that Marshmallow will know how to serialize it. but, where I’m unclear is, it seems like I need to then add it to the
    type_schemas
    here https://github.com/PrefectHQ/prefect/blob/a9270d965aee1e2182843e7fa75d18e1054230d0/src/prefect/serialization/schedule.py#L92. I can add to the map directly via the class, but that would obv only work locally (if at all) not on all the distributed agents. is there another way to register the
    RRuleClockSchema
    so it can get serialized? if not it seems like I need to fork prefect code and run agents with my
    RRuleClock
    codebase, while contributing my clock impl PR back into main branch. lmk if I’m missing something here and there’s a better way to go about this
    k
    m
    c
    • 4
    • 29
  • m

    Marvin

    08/24/2021, 2:44 PM
    Looking to contribute? Here are some issues to look at • Prefect describe flows CLI Command Lacks RunConfig Info . Created on 2021-04-28 • Azure Storage should have the option to use a secret name for the connection string. Created on 2021-04-27 • More Detailed Warning Messaged Around Using Local Storage. Created on 2021-04-09 • SQL Server SqlServerExecute Task does not work. Created on 2021-04-04 • Enable custom texts for notifications. Created on 2020-10-19
  • k

    Kathryn Klarich

    08/24/2021, 10:55 PM
    Hi all, I just saw that my PR was merged (thanks @Michael Adkins!), is there a way I can figure out what release my change will be included in?
    m
    • 2
    • 3
  • m

    Marvin

    08/30/2021, 9:00 AM
    Looking to contribute? Here are some issues to look at • Enable custom texts for notifications. Created on 2020-10-19 • Provide Kubernetes Tasks Examples. Created on 2020-08-21 • Client raises hard-to-understand error on requests to Prefect Cloud if tenant is not set. Created on 2020-06-27 • Mssql Tasks. Created on 2020-02-13 • Docs unit tests fail if an input argument has newlines. Created on 2019-01-07
    🤩 3
  • m

    Marvin

    09/06/2021, 9:00 AM
    Looking to contribute? Here are some issues to look at • Docs - Task Library Not Found. Created on 2021-06-19 • Typo in agent config flag in CLI. Created on 2021-05-27 • More Detailed Warning Messaged Around Using Local Storage. Created on 2021-04-09 • Enable custom texts for notifications. Created on 2020-10-19 • Having trouble using DatesClock.... Created on 2020-03-19
    :upvote: 1
Powered by Linen
Title
m

Marvin

09/06/2021, 9:00 AM
Looking to contribute? Here are some issues to look at • Docs - Task Library Not Found. Created on 2021-06-19 • Typo in agent config flag in CLI. Created on 2021-05-27 • More Detailed Warning Messaged Around Using Local Storage. Created on 2021-04-09 • Enable custom texts for notifications. Created on 2020-10-19 • Having trouble using DatesClock.... Created on 2020-03-19
:upvote: 1
View count: 2