It was nice to see everyone again!! :)

Not sure if this is the right channel for submitting PRs. But I have one to add chunking to GCS to enable handling of larger files or slower internet connections. Not sure how to best add tests for this in CI https://github.com/PrefectHQ/prefect/pull/3968

Hey everyone! I just opened a pull request and could use some help with updating a few CLI tests -- if anyone is able to help out, I would greatly appreciate it! https://github.com/PrefectHQ/prefect/pull/4146

Hey, I'm writing a set of new tasks to interact with MongoDB. looking at other database provider I can see they are no consistency and I want to have the community input 1. Password : Some task like postgresql or mysql are passing the password as a param to the task, whereas Redis for example is getting the password out of the Prefect secret. What is the recommended way of doing it ? 2. db connection string vs individual params : With Mongo I use usually only a single parameter connection string which looking like mongo://username:passwordf@db0.example.com:27017,db1.example.com:27017,db2.example.com:27017/?replicaSet=myRepl for example. However all the db tasks are using distinct params. Any recommendation ? Anything else I need to be aware about ?

Hey folks 🙂 Recently I've integrated SodaSQL with Prefect using a custom Task. Here you can find the docs https://docs.soda.io/soda-sql/documentation/orchestrate_scans.html This is just a basic implementation though. I'm thinking about to implement Soda tasks into Prefect core tasks and I'd like to understand if you have any idea or use case that you want like to address with Prefect regarding Data Quality monitoring

Hey folks 😒imple_smile: I’ve opened this PR https://github.com/PrefectHQ/prefect/pull/4308 Tests are failing due to missing header files for

pyodbc

Can someone help me fix it? Thanks!

I am doing some final testing of an extension to the Bitbucket storage class which adds support for Bitbucket cloud. When I register a flow using my new storage class a new

workspace

variable is not showing in the UI. I have also shown that when the flow runs, in an environment with the same version of the storage class, the

workspace

variable is None. I assumed my storage class would be pickled and stored with the flow and I wouldn’t need to make any code changes outside of this file. What am I missing? https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/bitbucket.py

Hey folks 😒imple_smile: Regarding ShellTask, wouldn’t it be useful to return the logs generated by the shell command as part of the FAIL message when the task fails? Maybe we can add a parameter that keeps the current behaviour as a default to avoid breaking changes. I would find it very useful for return useful details when a DbtShellTask fails… What do you think?

hi folks, I"m Jacopo - I built a very small POC on using the ShellTask as a base class to run Metaflow flows as part of a Prefect DAG (

<https://prefect-community.slack.com/archives/CL09KTZPX/p1617381793056800>

). If there is any interest in having MF tasks as part of Prefect task library, I'd be happy to give it some actual love and fix a bunch of things here and there 😉 Let me know what you think!

hey all, this morning some changes that I made to the

ShellTask

got released, but it looks like my docstring edits broke the way the documentation renders, as you can see here – is the issue that I needed to put the

Raises

block above the

Example

block in the docstring? thanks for your help! apologies if this is causing anyone any difficulty

Hi community 🙂 I improved the return value of the

StartFlowRun

Task at https://github.com/PrefectHQ/prefect/pull/4245 and have some minor concerns about it being a breaking change. I expect that nobody was using the return value because it was undocumented but I'm curious if anyone here was? Let me know what you think!

Hi guys 🙂 I need connector/extractor tasks (to load data into a data lake) for several API services, starting with GoogleAds task group for interacting with the Google Ads API (connecting, pulling data). Thinking also about integrating Hubspot, Microsoft Ads, Facebook Ads, DV360, Supermetrics, and other marketing data sources. I'm writing it here so maybe I can get some guidance or if there's already some effort which I could join?

hi all – if I'm digging through some source code and notice that on an already-merged PR something was omitted that maybe should have been included in it, is there a preferred way to bring that to the community's attention? I'm thinking either to bring it up in this channel, the #prefect-community channel, or to actually comment on the since-merged PR. Is there a standard for this kind of thing?

Report a typo in error log

AttributeError: 'FunctionTask' object has no attribute strftime. Did you call this object within a function that should have beendecorated with @prefect.task?

been decorated

Hey folks 🙂 Are there any plans to add data lineage to Prefect? Maybe an integration with Marquez/Datakin?

Hey Folks,  I'd like to start contributing some tasks that I have been working on (or enhancing) but I'm very new to the python development scene.  I've followed the first few instructions that you have for contributing on https://docs.prefect.io/core/development/overview.html and get a clean check out of a fork of the repository (I'm on the master branch).  I've encountered a few problems: 1. when running `pip install -e ".[dev]" I get a few bdist_wheel command not found. 2. running

pytest

results in 15 Failed tests I figure that I have missed some steps. I'll include what I have done in the thread.

I have a few Airtable tasks I’ve written and reused a few times. They snarf a whole Airtable table into a DataFrame, apply updates, and insert new records. The existing Airtable tasks in prefect only insert/read records a row at a time. I’d be interested in contributing them. Here’s the source: https://gist.github.com/jasonprado/0c71196bf21398d3b79123b9172987dd I would guess that most people want to read records from Airtable into python dicts/lists rather than DataFrames, right? If I wrote these classes that way but still wanted to use DFs in my project, would I then write my own tasks that turn the dicts/lists into DataFrames? Or should the tasks take a parameter to specify what kind of output they want (dicts vs DFs)? Just curious what the most idiomatic way to publish these tasks would be.

Hi - Just a heads up the API Reference displayed on the documentation website is displaying

0.14.18

not

0.14.19

.

Hi everyone, I'm looking to see how people feel about handling this issue: https://github.com/PrefectHQ/server/issues/254 TLDR: there are quite a few places in the Prefect Server code where it's hard-coded to use http, which means that deployments will inexplicably break in an environment where https is the default. I feel like it would be an easy fix to simply add an additional configuration value, but @Kevin Kho encouraged me to solicit feedback here before I start work on a PR.

I'd also like feedback on another issue I ran into, detailed in this comment: https://github.com/PrefectHQ/prefect/issues/4049#issuecomment-847407701 TLDR: more non-configurable behavior for different deployment styles of Prefect Server. This time it's preventing the initialization of a tenant at a Prefect API endpoint other than

localhost

— specifically because the Prefect Client is only written to use the Cloud configuration, even when

prefect backend server

is specified. A potential fix would have the Client read the backend configuration and adjust its initialization accordingly.

Hi i would like to contribute on the prefect project, I would like to know what is the best way in order to try its code; I want to try to had a new command in the CLI so i’ve created a new python file but how to compile the code and thus test this new command ? How do you work in a dev mode ? thx

hi all, I wrote a little func for testing flow imports. I found this helpful and had written something similar in our airflow app (RIP) for testing DAG imports. We do some dynamic code generation w/ jinja, so we use this just to make sure none of the files are malformed. Wanted to drop it here in case anyone else finds it useful, and also if anyone thinks this would be worth PRing into a testing module or something.

def flow_imports_tester(root_path: Path, module_dot_path: str):  # pragma: no cover
    """
    Helper for testing all flows in a project root_path at module_dot_path are importable
    Example:
        >>> flow_imports_tester(Path("~/my_project").expanduser(), "my_project.my_directory.flows")
    """
    split_dot_path = module_dot_path.split(".")
    package_directory = Path(root_path, *split_dot_path)

    flows = list(package_directory.iterdir())
    if not flows:
        raise RuntimeError("No flows found, did you specify the correct path?")

    for flow in flows:
        if "." in flow.name:
            filename, file_extension = flow.name.split(".")
        else:
            filename, file_extension = flow.name, ""

        if filename == "__init__" or file_extension != "py":
            continue

        import_module(".".join([module_dot_path, filename]))

Apologies in advance for etiquette because I’m totally new here. There’s a bug in the tutorial docs where a link leads to a 404. I looked for the issue in existing Issues and didn’t see it. I went to report it and it gave me the option of “chat in slack” along with “report a bug”. So, my question is would it be more appropriate to create an issue in GitHub, or describe it here?

Hi there, we are an autonomous driving startup and would like to explore using Prefect for some of the workflows we are managing. Where can I find someone from the sales/engineering team? Thank you!

Hi! We have some question not covered by the documentation (or we can't find it). And general questions on how we can use prefect. Where we should ask those questions (where we'll get answers 🙂), or may be we can have a zoom session with someone who can help?

hi not sure if this is right channel - my use case requires a clock/schedule that supports RRules (iCal style recurrences) which the current clock impls don’t support. I’ve implemented an

RRRuleClock

and

RRuleSchedule

(which is really just a function similar to other Prefect

Schedule

creators). however the flow won’t run b/c it doesn’t know how to serialize my

RRuleClock

. I assume I need to write a

RRuleClockSchema

so that Marshmallow will know how to serialize it. but, where I’m unclear is, it seems like I need to then add it to the

type_schemas

here https://github.com/PrefectHQ/prefect/blob/a9270d965aee1e2182843e7fa75d18e1054230d0/src/prefect/serialization/schedule.py#L92. I can add to the map directly via the class, but that would obv only work locally (if at all) not on all the distributed agents. is there another way to register the

RRuleClockSchema

so it can get serialized? if not it seems like I need to fork prefect code and run agents with my

RRuleClock

codebase, while contributing my clock impl PR back into main branch. lmk if I’m missing something here and there’s a better way to go about this

Looking to contribute? Here are some issues to look at • Prefect describe flows CLI Command Lacks RunConfig Info . Created on 2021-04-28 • Azure Storage should have the option to use a secret name for the connection string. Created on 2021-04-27 • More Detailed Warning Messaged Around Using Local Storage. Created on 2021-04-09 • SQL Server SqlServerExecute Task does not work. Created on 2021-04-04 • Enable custom texts for notifications. Created on 2020-10-19

Hi all, I just saw that my PR was merged (thanks @Michael Adkins!), is there a way I can figure out what release my change will be included in?

Looking to contribute? Here are some issues to look at • Enable custom texts for notifications. Created on 2020-10-19 • Provide Kubernetes Tasks Examples. Created on 2020-08-21 • Client raises hard-to-understand error on requests to Prefect Cloud if tenant is not set. Created on 2020-06-27 • Mssql Tasks. Created on 2020-02-13 • Docs unit tests fail if an input argument has newlines. Created on 2019-01-07