Good morning, we received this error on our remote machine which is running prefect agent as a systemd service

Good morning, we received this error over the weekend on our remote machine which is running prefect agent as a systemd service

Dec 24 22:04:58 linux prefect[279492]: 22:04:58.602 | ERROR   | prefect.agent - Server error '500 Internal Server Error' for url

The service failed, tried to restart a few times, was unsuccessful, and then our scheduled jobs on prefect cloud switched to state=Late. We're on prefect version 2.6.5 I'd like to rule out that this was an error on the prefect cloud side, does anyone know anything about prefect cloud issues around 12/24 22:04 UTC?

Hi all I am getting a weird bug in production for a flow with a kubernetes run config that hangs when I redeploy it

Prefect Core Version: 1.2.4

I trigger a flow run from a github actions step and it just hangs (happened yesterday and the run state was still in progress for 2 days) even though the job does not exist I know the flow did not finish because I see that my ml model tracking service did not register a new model I triggered the flow manually and it successfully completed. This flow is also scheduled and completes. What could be causing this issue? intermittent read time out error on the agent:

2022-12-27T04:49:54+02:00 WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")'

Hi all, anyone found some good documentation on how to build custom docker images. Really confused by the Prefect blocks UI and logging not providing any information. Usually the agent crashes and resulting in the flow run to enter the state of "PENDING" or "CRASHED", provides no information why the image (from registry) wasn't pulled and executed.

Is prefect_slack deprecated? I can’t get it to be recognized as a block with cloud or with a local orion setup that I’m testing.

Hi all, I'm working on unit testing some flows in Prefect 2 - what are the best practices for populating Blocks via Pytest fixtures?

Hey everyone is there a way to get a workers poll history through the CLI? i want to see if one failed

Hey everyone, just upgraded from prefect 1 to 2 and enjoying a lot of the new features, especially the

ConcurrentTaskRunner

! However, I am noticing that the speed of using

prefect_dask.DaskTaskRunner

is slower than using

prefect.executors.DaskExecutor

(prefect 1). In both cases I was using processes rather than threads. Speed depends on the task, but some tasks that read from s3 are taking a little over twice as long with the

DaskTaskRunner

. Has anyone faced something or similar, or any pointers as to why I may be experiencing this? Thanks!

hello. Is there any way to run flow triggered api?　Thanks!

Hi , is there any way to use .fn() and .submit() simultaneously while calling a task function. considering we have to pass parameters as well?

Hello.

Hello everyone Is there a way to get a list of all blocks in a workspace using Python code? I know that it can be done via the cli with

prefect block ls

, but I can't seem to find anything in the docks using python code.

Hi, is there a way to use S3 storage (for deployments/blocks) without explicitly providing the secret key/id? I would like to have the authentication handled automatically by AWS SDK, and will be provided using env vars or instance metadata

Hi Team! 👋

Hi! I recently had a working config of my Prefect 1 flow by importing the following modules:

import prefect
from prefect import task, Flow
from prefect.storage import GitHub
from prefect.run_configs import ECSRun
from prefect.client import Secret

However, when I run my code now (Prefect 2.7.4), I get a

ModuleNotFoundError: No module named 'prefect.storage'

error (same goes for

prefect.run_configs

and

prefect.client

). Was there a recent update to the way we should import these modules?

Hello, I have a question regarding scheduling (exceptions)

Hello! I noticed that flow function comments are shown as description in the flow's deployment, is there a way to format the comments? I have tried html and markdown with no luck.

@flow
def hello():
   """
   Is there a way to get the text in here to be formatted in the awesome prefect UI?
   """
   pass

Hi Prefect Team, We have our dev environment up and running for Prefect 2 now. When setting up the Prefect 2 production environment we encountered an issue where we cannot access our database. We are using an ECS Task to run our flows in dev and prod. The dev instance is able to read data and execute the program, but production is not able to. The ECS Task in production is using the same VPC, Subnet, routes and security group as development. Is there another possible reason for this failure?

Hi Team! 👋 Is it possible to have a custom docker image in your local file system be referenced by the docker-container infrastructure block when building a deployment or am I required to have my custom docker image on Docker Hub/Docker Registry and access it through those systems?

Hello you wonderful group of people! Is there a best-practice for producing a mermaid (or other diagram) for a flow that shows relationships and what the expected task result format would be?

How to achive starting 201 subflows for ETL? Something like mapped flows. Are there good practices for this? I don't want to pack extact and load into single tasks, and map pages.

from prefect import flow 

@task
def get_pages() -> List[int]:
	# Get pages for subflows
	...

@task
def extract(page: int) -> List[Dict[str, Any]]:
	...

@task
def load(records: List[Dict[str, Any]])
	...

@flow 
def subflow_etl(page: int): 
	"""
	Extracts data from single page, and inserts it into DB
	"""
	data = extract(page)
	load(data)
	
@flow 
def main_flow(): 
	pages = get_pages() # returns [1, 2, 3, 4, ..., 200, 201]
	# How now starts 201 subflows_etl?
	# I have several agents which would like to consume subflow_etl
	# And process it on several machines.
	# Psuedo code
	for page in pages:
		subflow_etl(page)
	
	
if __name__ == "__main__": 
	main_flow()

Hello again! I have an orchestrator flow that calls two other flows using run_deployment. The first flow returns a pandas dataframe that is persisted to an Azure block. What would be the most efficient way to pass the first flow's result to the second flow? I was looking at this but it looks like it would require a lot of serialization/deserialization the main flow is something like this

run_deployment("transformation_1", ...) # returns a df as result that is persisted to azure storage
run_deployment("transformation_2", ...) # needs the dataframe returned by `transformation_1` as input

should I manually save the df to storage in

transformation_1

and pass the url to

transformation_2

?

Does anyone know how to update the owner name and email on Cloud 2?

Hello Prefect team! i am trying to handle timout for tasks, and was wondering how can I force a task to stop after reaching the timeout. FYI I am using prefect 2.3.1

Hi wonder if anyone can help - I have a flow (prefect v1 on cloud) that seems to have started, run some of its tasks and then started again (logs in the thread) - when it starts again a task input seems to be

None

even though the previous task producing that input as it's output (which ran before the flow seems to have started again) ran successfully - I wonder if I am running into some problems with checkpointing and not using Results? I would really like the flow to just fail rather than trying to restart itself?

Hi Prefect team I am trying to handle tasks timout, and was wondering how can I stop/cancel a task if the timout is reached FYI: I am using Prefect 2.3.1 Here is an exemple of what I want to do

Wait N sconds for the task to complete

        >>> @flow
        >>> def my_flow():
        >>>     future = my_task()
        >>>     final_state = future.wait(0.1)
        >>>     if final_state:
        >>>         ... # Task done
        >>>     else:
        >>>         ... # Task not done yet = > Cancel task

How does authentication work on Orion running on premise?

Issues with deploying flows in Prefect 2.0 When running

prefect deployment build <file name>.py:r<flow name> -q local  --name <deployment name> --path ~/<deployment path> --skip-upload

in the

yaml

file, the

entrypoint:

uses a relative path, which does not work when running the deployment from the cloud dashboard. When I change it to a full path, it works, but I think that should not be the behavior, particularly since the

entrypoint

is below the

### DO NOT EDIT BELOW THIS LINE

Hi Prefect Team - Can we have Prefect 1 and Prefect 2 on the same Linux box? Are we able to just update the name of the systemd service? It doesn't need to be prefect-agent.service, right? It could be prefect2-agent.service, right?

Hello Prefect Team, I’m deploying a flow with the Docker infra using the

docker:dind

(docker -in-docker) image. The ideia is to run containers inside the flow using the

docker

sdk. Using the

/var/run/docker.sock

volume is not an alternative for my use case. It seems that the only thing stopping me at this point is that I cannot set the

privileged

flag in the

prefect.infrastructure.DockerContainer

abstraction. As shown in the docker-compose below, it is required for running the

docker:dind

container.

services:
  dind:
    image: docker:dind
    privileged: true

Could this option be added to Prefect? Happy new year for everyone 😄