@Marvin how do i instruct a custom logger to flush its logs programmatically?

Hi, I’m trying to get a Datadog agent running in k8s to collect application metrics from my Prefect Flows. Just wondering if anybody has this working and if they could share how they did it.

@Marvin Why am I getting this error:

ValueError: Unable to create hash - objects could not be serialized.
  JSON error: Unable to serialize unknown type: <class 'pandas.core.frame.DataFrame'>
  Pickle error: cannot pickle 'weakref.ReferenceType' object

This often occurs when task inputs contain objects that cannot be cached like locks, file handles, or other system resources.

To resolve this, you can:
  1. Exclude these arguments by defining a custom `cache_key_fn`
  2. Disable caching by passing `cache_policy=NONE`

Hey everyone! Does anyone know what are the resource requirements when deploying the prefect-server helm chart? I tried to deploy it on two Raspberry Pi 4s with 8GB RAM that are running k3s, and most of the pods are unable to start, which results in 0 deployments working. All the dead pods are giving the

standard_init_linux.go:228: exec user process caused: exec format error

error in logs though, so not 100% sure the issue's hardware-related. Any suggestions would be appreciated 🙏

Is there a way to use a k8s secret for environment variables in my job variable or prefect deployment? I'd like the pod the prefect-worker spins up to use an environment variable thats loaded from a secret in my namespace. Not sure if this is possible

is there a way programatically to get a list of deployments in prefect? i know we can do

prefect deployment ls

but i want to get all my created deployments...1000 of them and run them all. so im assuming i have to programatically execute prefect deployment run deploymenet_name for each deployment in my deployments

Hello. I'm using

subprocess

to run my dbt project locally (Prefect 1.2.2, Windows). I create my command (

dbt run --vars '{data_processing_start_date: 2022-07-20, data_processing_end_date: 2022-07-20}' --profiles-dir ./

) like this:

command = (
        f"dbt run --vars '{{"
        f"data_processing_start_date: {data_processing_start_date}, "
        f"data_processing_end_date: {data_processing_end_date}}}' --profiles-dir ./ "
    )

The command is created correctly, but dbt gives me this error

dbt: error: unrecognized arguments: 2022-07-20, data_processing_end_date: 2022-07-20}'

Seems like dbt interprets

2022-07-20

as an argument instead of the value for

data_processing_start_date

variable. Keep in mind that when I run the same command in my CLI, dbt works fine. But when it's provided to dbt through

subprocess

this occurs. This is my subprocess:

subprocess.run(
        command,
        check=True,
        stderr=True,
        stdout=True,
        shell=True,
        cwd="dbt",
    )

Any idea what might cause this, and how to solve this? Thank you

@Marvin how do I trigger a deployed flow using the prefect SDK using the requests library? give an example that sends flow parameters

I'm trying to use the Scrapy framework in conjunction with Prefect 2.0, and Scrapy automatically sets up its own core Python Loggers. When I run a flow in the terminal, I can see the logs coming from Scrapy's loggers like so:

13:37:49.758 | DEBUG   | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>

However, those logs don't appear in the flow run logs UI. I'm guessing I need to set

PREFECT_LOGGING_EXTRA_LOGGERS

, but I'm not certain what the right way to do that is.

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"

seems to have no effect. I also tried

prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"

using the name of the specific logger I wanted to see, but that didn't work either.

Anyone else's flows being stuck in 'Pending' state? They were running fine earlier. Is it some sort of limit I've run into by any chance on the free tier?