Yaron Levi
09/03/2022, 6:18 PMYaron Levi
09/03/2022, 6:18 PMYaron Levi
09/03/2022, 6:18 PMYaron Levi
09/03/2022, 6:19 PMvk
09/03/2022, 6:49 PM@flow
def print_hi():
print(f'Hi, 123') # Press ⌘F8 to toggle the breakpoint.
deployment = Deployment.build_from_flow(
name="print_hi",
flow=print_hi,
skip_upload=True,
infrastructure=KubernetesJob(
image='my-image',
)
)
deployment.apply()
This created a deployment, but why does it say This deployment is deprecated
With the General Availability release of Prefect 2.0, we modified the approach to creating deployments.
? This is the code from latest test cases, prefect 2.3.1, how can it be deprecated? And why is "Run" button inactive?
my-image
is based on prefect-onion:latest
and has my flow code in /opt/prefect/flows
. By the way, how does it know which flow to run if there is more that one flow in that folder?Yaron Levi
09/03/2022, 9:24 PMJari Rosti
09/04/2022, 9:24 AMMichael Levenson
09/04/2022, 5:25 PMGeorgi Yanev
09/04/2022, 9:11 PMYousef Hosny
09/05/2022, 2:25 AMprefect-airbyte
to monitor & schedule my airbyte pipelines but unfortunately I am only getting these logsSaurabh Indoria
09/05/2022, 3:55 AMTan Li
09/05/2022, 5:15 AMprefect deployment build --cron "00 1 * * *"
But how can I specify timezone in cli like how it can be done through the python object interface (I am having a weird bug, i.e. some folder under /tmp/* could not be found if a python deployment get execute in the second time)? I searched a bit on Github but only found some python code, and I also tried to looked at the code implementation (on 2.3.0), but it’s not very clear to me where that TimeZone flag got specified (already got lost in the *args maze) Any help would be appreciated! 🙏🙏🙏Jari Rosti
09/05/2022, 8:01 AMcurl -X 'POST' \
'<http://localhost:4200/api/flow_runs/>' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"name": "my-flow-run",
"flow_id": "adc10e6b-62b1-4a5a-ba75-a9d03107c75c",
"deployment_id": "0a26b313-0134-43ac-b4ab-67bcbeeaebf9",
"parameters": { "flow_run_id": "d3f49714-3b14-4bbf-a42c-7081149b9fb3" }
}'
and get a response:
{
"id": "1277b255-4250-45f7-b0bf-c06630dcac86",
"created": "2022-09-05T07:51:42.448486+00:00",
"updated": "2022-09-05T07:51:42.450190+00:00",
"name": "my-flow-run",
"flow_id": "adc10e6b-62b1-4a5a-ba75-a9d03107c75c",
"state_id": "aae8e698-9df6-4efa-8692-2d9e97e01112",
"deployment_id": "0a26b313-0134-43ac-b4ab-67bcbeeaebf9",
"work_queue_name": null,
...
Mohamed Alaa
09/05/2022, 8:18 AMMatt Fysh
09/05/2022, 8:41 AMNuno Silva
09/05/2022, 9:50 AMBen Muller
09/05/2022, 10:48 AMMalavika S Menon
09/05/2022, 12:02 PMYoussef Ben Farhat
09/05/2022, 12:45 PMNiels Prins
09/05/2022, 1:58 PMAndreas Nord
09/05/2022, 3:13 PMRhys Mansal
09/05/2022, 3:29 PMTim Helfensdörfer
09/05/2022, 3:40 PMwork_queue_name
and tags
. Both work queues are running at the moment, e.g. with prefect agent start --work-queue "Production"
. But only the Staging work queue picks up jobs. The production work queue does not.
It is correctly assigned in the cloud ui, i.e. I can see the late jobs in the work queue tab, but the queue does not get any data: (moved to thread)F. Nikita Thomas
09/05/2022, 4:54 PMprefect[viz]
was installed, but with the current version ,<2.3.1> , I'm having difficulties get some basic code to work - Could someone please assist? Thanks!
import requests
from pprint import pprint as pp
from prefect import flow, task, Flow
import pandas as pd
import json
## extract
@task(name="extract")
def get_data():
r = requests.get("<https://rickandmortyapi.com/api/character/>")
return r.json()
@task
def transform(r: dict):
df = pd.json_normalize(r["results"])
print(df)
@task(name="load")
def write_frame(df: pd.DataFrame):
pass
"""
## This is the way to visualize flow before current version
with Flow("ETL") as flow:
e = get_data()
t = transform(e)
l = write_frame(t)
flow.visualize()
"""
@flow
def flow_test():
e = get_data()
t = transform(e)
l = write_frame(t)
flow_test().visualize() # This doesn't work...
Arun Giridharan
09/05/2022, 4:58 PMSomething went wrong. Please wait a few moments and try again
. Any idea how I can debug this?Tom Kaszemacher
09/05/2022, 5:20 PMEsdras Lopes Nani
09/05/2022, 8:21 PMIan Andres Etnyre Mercader
09/06/2022, 12:33 AMorion_agent-orion_server-1 | 2022-09-06 00:21:29,528 - distributed.worker - WARNING - Compute Failed
orion_agent-orion_server-1 | Key: 1843a132-66f7-45c1-9c76-9736ca31cd4a
orion_agent-orion_server-1 | Function: begin_task_run
orion_agent-orion_server-1 | args: ()
orion_agent-orion_server-1 | kwargs: {'task': <prefect.tasks.Task object at 0x7f28a2f18a60>, 'task_run': TaskRun(id=UUID('9ac621e9-cac9-4ff7-813e-da0ce5d2bda3'), created=DateTime(2022, 9, 6, 0, 15, 11, 280979, tzinfo=Timezone('+00:00')), updated=DateTime(2022, 9, 6, 0, 15, 12, 995201, tzinfo=Timezone('+00:00')), name='download_xml-39058d84-636', flow_run_id=UUID('258498db-6397-412e-a79a-39f252b8316a'), task_key='lib.dask_context.dask_context.<locals>._dask_task', dynamic_key='636', cache_key=None, cache_expiration=None, task_version=None, empirical_policy=TaskRunPolicy(max_retries=0, retry_delay_seconds=0.0, retries=15, retry_delay=2), tags=['map_process'], state_id=UUID('79733d18-24d6-43b5-9f5f-1b04f3022edd'), task_inputs={'args': [TaskRunResult(input_type='task_run', id=UUID('fb87e646-0e3b-4647-abe5-8dea70156bbd'))], 'kwargs': []}, state_type=StateType.PENDING, state_name='Pending', run_count=0, expected_start_time=DateTime(2022, 9, 6, 0, 15, 11, 280780, tzinfo=Timezone('+00:00')), next_scheduled_start_time=None, start_
orion_agent-orion_server-1 | Exception: "RuntimeError('The connection pool was closed while 1 HTTP requests/responses were still in-flight.')"
are they the attempts of the task that failed and are going to retry? or are they something else?Faheem Khan
09/06/2022, 1:09 AM