chicago-joe

    chicago-joe

    11 months ago
    Hey guys, I'm wondering if anyone has had any experience using Prefect Tasks within a class object? Essentially I'm working on a MySQL Class object that will handle authenticate, open/close connections, and queries. The protected "auth" class is using a prefect task to load the config from AWS Secrets Manager. It all works fine until I try to use the class within a flow, then I run into errors. Any help would be greatly appreciated, cheers!https://gist.github.com/chicago-joe/98c165a596c6eb12337a07073f8e9b4d
    Kevin Kho

    Kevin Kho

    11 months ago
    What error do you get?
    chicago-joe

    chicago-joe

    11 months ago
    I import the class and am trying (probably incorrectly) to instantiate it and then use it in the prefect tasks that are defined in the flow:
    with Flow('daily-tiinga-loader') as flow:
        conn = sraController("prod")
    It raises a ValueError:
    ValueError: Secrets should only be retrieved during a Flow run, not while building a Flow.
    Adam Brusselback

    Adam Brusselback

    11 months ago
    Yup, so you'll want to make sure you store your secrets into intermediate variables, and instead of passing them in at init of the mysql task, pass them on run/execute.
    chicago-joe

    chicago-joe

    11 months ago
    what if the MySQL task isn't a prefect task, it's a class that handles the connection, queries, and other common database functions that we use? I can try writing a run() that gets called every time a class module is called, I think that would work for this
    Kevin Kho

    Kevin Kho

    11 months ago
    The error is coming from this line because you can only use PrefectSecret inside a Flow. It’s not about the MySQL task I think
    chicago-joe

    chicago-joe

    11 months ago
    got it, I can try setting the AWS Credentials as docker environment variables when I build the image that this package is compiled in