David Cheedie
12/27/2022, 2:54 PMDavid Cheedie
12/27/2022, 3:03 PMDec 24 22:04:58 linux prefect[279492]: 22:04:58.602 | ERROR | prefect.agent - Server error '500 Internal Server Error' for url
The service failed, tried to restart a few times, was unsuccessful, and then our scheduled jobs on prefect cloud switched to state=Late. We're on prefect version 2.6.5
I'd like to rule out that this was an error on the prefect cloud side, does anyone know anything about prefect cloud issues around 12/24 22:04 UTC?Michael Levenson
12/27/2022, 4:41 PMPrefect Core Version: 1.2.4
I trigger a flow run from a github actions step and it just hangs (happened yesterday and the run state was still in progress for 2 days) even though the job does not exist
I know the flow did not finish because I see that my ml model tracking service did not register a new model
I triggered the flow manually and it successfully completed.
This flow is also scheduled and completes.
What could be causing this issue?
intermittent read time out error on the agent:
2022-12-27T04:49:54+02:00 WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")'
PKay
12/27/2022, 5:14 PMChris Whatley
12/27/2022, 5:54 PMJohn-Craig Borman
12/27/2022, 7:35 PMSebastian Steele
12/27/2022, 8:57 PMMatt Morris
12/27/2022, 9:06 PMConcurrentTaskRunner
! However, I am noticing that the speed of using prefect_dask.DaskTaskRunner
is slower than using prefect.executors.DaskExecutor
(prefect 1). In both cases I was using processes rather than threads. Speed depends on the task, but some tasks that read from s3 are taking a little over twice as long with the DaskTaskRunner
. Has anyone faced something or similar, or any pointers as to why I may be experiencing this? Thanks!Rikimaru Yamaguchi
12/28/2022, 5:30 AMMohit Singhal
12/28/2022, 11:45 AMAbosede Oladipo
12/28/2022, 12:59 PMAbosede Oladipo
12/28/2022, 1:47 PMprefect block ls
, but I can't seem to find anything in the docks using python code.Tadej Svetina
12/28/2022, 6:00 PMDalin Stone
12/28/2022, 7:25 PMKen Nguyen
12/28/2022, 7:53 PMimport prefect
from prefect import task, Flow
from prefect.storage import GitHub
from prefect.run_configs import ECSRun
from prefect.client import Secret
However, when I run my code now (Prefect 2.7.4), I get a ModuleNotFoundError: No module named 'prefect.storage'
error (same goes for prefect.run_configs
and prefect.client
). Was there a recent update to the way we should import these modules?CA Lee
12/29/2022, 8:39 AMPaco Ibañez
12/29/2022, 3:03 PM@flow
def hello():
"""
Is there a way to get the text in here to be formatted in the awesome prefect UI?
"""
pass
Chris Gunderson
12/29/2022, 3:57 PMDalin Stone
12/29/2022, 4:20 PMJoshua Grant
12/29/2022, 5:18 PMMichał
12/29/2022, 7:20 PMfrom prefect import flow
@task
def get_pages() -> List[int]:
# Get pages for subflows
...
@task
def extract(page: int) -> List[Dict[str, Any]]:
...
@task
def load(records: List[Dict[str, Any]])
...
@flow
def subflow_etl(page: int):
"""
Extracts data from single page, and inserts it into DB
"""
data = extract(page)
load(data)
@flow
def main_flow():
pages = get_pages() # returns [1, 2, 3, 4, ..., 200, 201]
# How now starts 201 subflows_etl?
# I have several agents which would like to consume subflow_etl
# And process it on several machines.
# Psuedo code
for page in pages:
subflow_etl(page)
if __name__ == "__main__":
main_flow()
Paco Ibañez
12/29/2022, 8:18 PMrun_deployment("transformation_1", ...) # returns a df as result that is persisted to azure storage
run_deployment("transformation_2", ...) # needs the dataframe returned by `transformation_1` as input
should I manually save the df to storage in transformation_1
and pass the url to transformation_2
?kiran
12/30/2022, 12:20 AMNaila Chennit
12/30/2022, 9:38 AMJoshua Greenhalgh
12/30/2022, 9:59 AMNone
even though the previous task producing that input as it's output (which ran before the flow seems to have started again) ran successfully - I wonder if I am running into some problems with checkpointing and not using Results? I would really like the flow to just fail rather than trying to restart itself?Naila Chennit
12/30/2022, 10:01 AMWait N sconds for the task to complete
>>> @flow
>>> def my_flow():
>>> future = my_task()
>>> final_state = future.wait(0.1)
>>> if final_state:
>>> ... # Task done
>>> else:
>>> ... # Task not done yet = > Cancel task
Edmondo Porcu
12/30/2022, 4:36 PMYD
12/30/2022, 4:41 PMprefect deployment build <file name>.py:r<flow name> -q local --name <deployment name> --path ~/<deployment path> --skip-upload
in the yaml
file, the entrypoint:
uses a relative path, which does not work when running the deployment from the cloud dashboard.
When I change it to a full path, it works, but I think that should not be the behavior, particularly since the entrypoint
is below the ### DO NOT EDIT BELOW THIS LINE
Chris Gunderson
12/30/2022, 7:55 PMDavid Steiner Sand
12/30/2022, 8:35 PMdocker:dind
(docker -in-docker) image. The ideia is to run containers inside the flow using the docker
sdk. Using the /var/run/docker.sock
volume is not an alternative for my use case.
It seems that the only thing stopping me at this point is that I cannot set the privileged
flag in the prefect.infrastructure.DockerContainer
abstraction. As shown in the docker-compose below, it is required for running the docker:dind
container.
services:
dind:
image: docker:dind
privileged: true
Could this option be added to Prefect?
Happy new year for everyone 😄