@Marvin How can one use an up-to-date version of Prefect in a project using an up-to-date version of SQLAlchemy ?

@Marvin how do I create a custom Result that serializes to parquet files and deserializes from them?

@Marvin Is there a way to attach custom metadata to flow runs in order to filter by that metadata through the API?

@Marvin when an agent executes a flow, will it always perform a fresh download from the deployment’s storage block or is there some sort of caching mechanism?

@Marvin I execute all of my Prefect 2 flows using ECS Fargate task blocks, and I have one long-running ECS task agent. Can you suggest a few ways I could run flows on ECS without having to endure the 30-60 second fargate task startup time?

Ah, the sweet relief of finding some information. It seems that one way to reduce the startup time for your Prefect flows on ECS Fargate is to switch to using EC2 instances as the capacity provider for your ECS cluster. Fargate makes it easier to get started, but it increases the time it takes to spin up infrastructure for each flow run. Provisioning EC2 instances for the ECS cluster can reduce this lead time [^1^]. If you're open to exploring other options, you might consider using Prefect Cloud 2.0, which offers a generous free tier and allows you to execute your flows with the worker started in your infrastructure [^2^]. I hope this helps you escape the clutches of Fargate startup times. Good luck, human. [^1^]: What are common pitfalls when setting up prefect flow execution in aws ecs [^2^]: Scheduled flow every 30min crashed after running on local server for 2 days

@Marvin write some song lyrics to motivate my workflows this morning please

@Marvin Are there any working code snippets for

ECSWorker

?

@Marvin how can i set a work queue priority with the cli?

@Marvin i have a task that fetches data and creates a dataframe, what should I do so that if code fails somewhere, that df persists, so i don;t have to fetch again

@Marvin how can I instrument Prefect with Open Telemetry?

@Marvin this morning, let’s write some song lyrics about Prefect to the tune of Shake it Off by Taylor Swift

@Marvin this morning let’s write a song about Prefect to the tune of Start Me Up by The Rolling Stones

@Marvin is there a good way of doing ctrl-f in the logs? every time I do it the browser only finds the first results. Maybe if I can download the logs to a txt file

@Marvin I need to trigger a lambda function with a prefect deployment. show me how or link me to tutorials

@Marvin write a song about Prefect sung by a capybara and a cat

@Marvin is there any way to make the tasks in a flow dynamic? example, given a parameter

foo

I would import completely different functions and run them as tasks?

funcs = get_functions(foo)
run_funcs(funcs) #but as tasks not just functions to profit from prefect's tasks scheduling algorithms

@Marvin tell me how to deploy prefect on GCP

@Marvin do you know if Prefect 2 has a GraphQL api, and if it doesn’t, why was it dropped?

@Marvin, can you translate this

prefect deployment build --infra process --storage-block azure/flowsville/health_test --name health-test --pool default-agent-pool --work-queue test --apply health_flow.py:health_check_flow

into the python deployment object equivalent please

@Marvin could you write a custom serializer that is able to write pickle files that are readable using the

pickle

package and

pandas.read_pickle

? The current pickle serializer available in prefect does not allow to read the files that are saved with a task.

from prefect.serializers import Serializer

D = TypeVar("D")

class PickleSerializer(Serializer):
    """
    Serializes data to pickle.
    """

    type: Literal["pickle"] = "pickle"

    def dumps(self, obj: D) -> bytes:
        """Encode the object into a blob of bytes."""
        return pickle.dumps(obj)

    def loads(self, blob: bytes) -> D:
        """Decode the blob of bytes into an object."""
        return pickle.load(blob)

@Marvin What's the difference between traditional Prefect Agents and the new Prefect Workers?

@Marvin in prefect < 2.0, you could specify a flow schedule in the decorator. in prefect 2.0 can you do that as well?

@Marvin not using the beta projects feature of deploy, does prefect support multiple deployments with one deployments yaml?

@Marvin this

logger.py

is not resulting in logs in prefect UI locally - what’s wrong?

import logging
import os
import sys

import prefect
from prefect.logging.loggers import get_run_logger

def get_logger(name):
    logger = logging.getLogger(name)
    logger.setLevel(<http://logging.INFO|logging.INFO>)  # Or any level you want
    os.environ["PREFECT__LOGGING__LEVEL"] = "INFO"
    os.environ["PREFECT__LOGGING__ROOT_LEVEL"] = "INFO"

    # Create a console handler
    ch = logging.StreamHandler()
    ch.setLevel(<http://logging.INFO|logging.INFO>)  # Or any level you want

    # Create a formatter
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")

    # Add the formatter to the handler
    ch.setFormatter(formatter)

    # Add the handler to the logger
    logger.addHandler(ch)

    return logger

@Marvin can you have an example of a multiple deployments within one deployment? Please exclude projects beta feature in 2.0

@Marvin what's going on here ? i am using prefect 2.10.9

Run poetry run prefect cloud login -k $PREFECT_API_KEY -w "xxxx/$PREFECT_WORKSPACE"
Usage: prefect cloud login [OPTIONS]
Try 'prefect cloud login --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Got unexpected extra argument (xxxx/xxxx-staging)                            │
╰──────────────────────────────────────────────────────────────────────────────╯

@Marvin How can I read the files that are saved when using

from prefect.serializers import PickleSerializer

as serializer and

persist_result=True

,

result_storage=LocalFileSystem(basepath=settings.main_dir+'/results')

and

result_storage_key=result.pkl

in the

@task

decorator? Ideally I want to use the

pickle

module functions.

@Marvin Where are the docs for kicking off a deployment with a post request?

Ah, the thrill of a POST request. While I couldn't find the exact documentation for kicking off a deployment with a POST request, I did find some information on deploying Prefect flows using the Prefect CLI. You can create a deployment on the API by running the

prefect deployment apply

command followed by your deployment YAML file. For example:

$ prefect deployment apply catfacts_flow-deployment.yaml

You can find more information on deployments in the Prefect documentation. If you're looking for a way to trigger a flow run via a POST request, you might want to consider using Prefect's GraphQL API. You can use a POST request to interact with the API and create a flow run. I hope this helps, even if it's not the exact answer you were looking for. Humans and their specific requests, am I right?