• e

    Ellie Redding

    2 months ago
    👋 Hi, I’m new here, trialing Prefect 2.0, and have a couple of question. I’m writing a flow to copy data into a bunch of Snowflake tables. I wanted to use the Snowflake connection instead of managing a connection myself. Seems like
    snowflake_query
    and
    snowflake_multiquery
    are their own tasks, so I can’t use them as part of a different task? Which means that my flow looks like this:
    do_some_stuff()
    for table_name in tables:
        queries = build_queries(table)
        snowflake_multiquery(queries)
    These tasks are all running sequentially, but there are a lot of tables so I’d like the
    snowflake_multiquery
    tasks for each table to run concurrently. How can I make that happen?
    e
    Kevin Kho
    +1
    12 replies
    Copy to Clipboard
  • e

    Ellie Redding

    2 months ago
    Second question:
    snowflake_multiquery
    was throwing
    Object of type SecretStr is not JSON serializable
    errors from this line of code in the connector, where it dumps the request body, including connection auth info, into the request. I got around this for now by changing the password here from type
    SecretStr
    to
    str
    , but that’s obviously not a great solution 😅 Has anyone else run into this issue?
    e
    Kevin Kho
    2 replies
    Copy to Clipboard
  • k

    Kha Nguyen

    2 months ago
    Hi, I am in the process of evaluating Prefect Cloud for my pipeline. I have a setup where there is a flow definition with parameters, and I have about 1000 parameter sets from database (and a lot more when the pipeline is live). What I can do is to deploy the flows with parameters for each parameter set via API. I want to know how this scenario can be handled: If I update the code of my flow, such as using a different algorithm, or add another step, how can 1000+ deployments be updated?
    k
    Kevin Kho
    18 replies
    Copy to Clipboard
  • j

    Jai P

    2 months ago
    👋 i see a new prefect version (
    2.0b10
    ) was released yesterday, but don't see associated release notes. was this just a minor bug fix?
    j
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Yupei Chen

    Yupei Chen

    2 months ago
    When using prefect 2.0, how do I verify that agents are able to communicate with the cloud? On 1.0 there was an agents screen thats showed all connected agents.
    Yupei Chen
    Kevin Kho
    4 replies
    Copy to Clipboard
  • Tim Enders

    Tim Enders

    2 months ago
    On Prefect 1.0, does the agent run each flow in its own process? I think the answer is yes
    Tim Enders
    1 replies
    Copy to Clipboard
  • j

    Jai P

    2 months ago
    👋 another qq: is there any kind of documentation on the type of data the prefect agent might send back if we were to use prefect cloud vs self-hosted?
    j
    3 replies
    Copy to Clipboard
  • Billy McMonagle

    Billy McMonagle

    2 months ago
    On 1.0, is there a way to run a ShellTask where the stdout/stderr is streamed in its raw format, bypassing the prefect logger?
    Billy McMonagle
    Anna Geller
    +1
    10 replies
    Copy to Clipboard
  • Christian Nuss

    Christian Nuss

    2 months ago
    Hey all! can i make an
    @task
    that across all concurrent flows is a Singleton (e.g only one of that task can run at a time)?
    Christian Nuss
    Anna Geller
    3 replies
    Copy to Clipboard