Chu Lục Ninh
03/27/2022, 11:31 AMhimanshu pandey
03/27/2022, 2:06 PMLI LIU
03/28/2022, 1:02 AMJeff Kehler
03/28/2022, 8:31 AMParameter
into the class constructor __init__
of a task that has been subclassed from prefect.Task
I am receiving a prefect.Task
object instead of the value from the Parameter itself. I can't seem to figure out what I'm doing wrong.Michael Smith
03/28/2022, 8:36 AMJons Cyriac
03/28/2022, 9:14 AMBennett Lambert
03/28/2022, 10:55 AMrequests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))
Is there a way to provide proxy credentials when starting the agent? I've tried using the --env flag to supply proxy information. Or is there something similar to a --verify=false that I can set?Rahul Kadam
03/28/2022, 12:42 PMFlorian Guily
03/28/2022, 1:05 PMBennett Lambert
03/28/2022, 2:13 PMDekel R
03/28/2022, 2:44 PMwith Flow('html_data_extraction__dev',
storage=Docker(registry_url="us-central1-docker.pkg.dev/***/",
dockerfile="./Dockerfile"),
schedule=daily_schedule, executor=LocalDaskExecutor(scheduler="processes")) as flow:
mode = Parameter(name='mode', default=None)
with case(mode, None):
data_a=task_a
with case(mode, 'onboard'):
data_b=task_b
data_c=merge(data_a, data_b)
task_c
task_d
with case(mode, None):
task_x
with case(mode, 'onboard'):
task_y
Tasks a and b retrieve some data (each one is relevant for a different data source), task c and d are common (not in a “case”) - doing X on the data (the data looks the same at this point) and then again tasks x and y are different - each one is relevant for a different case.
When running locally (mac, flow,run…) it all works as expected.
When running on Prefect cloud - all of the tasks gets skipped (exactly the same code and credentials)
Any idea on what I’m missing here?
I’m using “upstream_tasks” in order to run the tasks in a specific order when necessary.
Thankskevin
03/28/2022, 3:54 PM{'_schema': 'Invalid data type: None'}
after it is scheduled but before the first task gets executed. When I try to run this flow in a local environment it executes as expected. Any idea what could be causing this issue?Michael Smith
03/28/2022, 4:46 PMMyles Steinhauser
03/28/2022, 4:48 PMKen Nguyen
03/28/2022, 5:17 PM{'errors': [{'path': ['flow_run'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}], 'data': None}
When re-running a flow at a later time, it runs successfully. Are there any docs that can provide info on API limits?Harry Baker
03/28/2022, 7:05 PMMichael Smith
03/28/2022, 7:44 PMMyles Steinhauser
03/28/2022, 8:36 PMFlows
? Specifically, I’m trying to workaround some delayed scaling issues with ECS using EC2 instances (not ECS with Fargate tasks)
Often, this failure is reported back to Prefect like the following error until Capacity Provider scaling has caught up again:
FAIL signal raised: FAIL('a4f09101-0577-41ce-b8b0-31b84f26d855 finished in state <Failed: "Failed to start task for flow run a4f09101-0577-41ce-b8b0-31b84f26d855. Failures: [{\'arn\': \'arn:aws:ecs:us-east-1:<redacted>:container-instance/a8bc98b7c6864874bc6d1138f758e8ea\', \'reason\': \'RESOURCE:CPU\'}]">')
I’m using the following calls to launch the sub-flows (as part of a larger script):
flow_a = create_flow_run(flow_name="A", project_name="myles")
wait_for_flow_a = wait_for_flow_run(flow_a, raise_final_state=True, stream_logs=True)
Alex Prokop
03/28/2022, 8:39 PMEric Mauser
03/28/2022, 8:50 PMconnections = ['conn1', 'conn2',
'conn3']
with Flow("flow_name", run_config=RUN_CONFIG, storage=STORAGE, schedule=SCHEDULE) as flow:
for conn_id in connections:
flow.add_task( AirbyteConnectionTask(
airbyte_server_host=<Airbyte host>
airbyte_server_port=<airbyte port>,
airbyte_api_version="v1",
connection_id=conn_id
)
)
flow.run(executor=LocalDaskExecutor)
Leo Kacenjar
03/28/2022, 9:47 PMCMD
in my dockerfile and it seems to not be executing. That makes me think it is being overwritten. Maybe I have to provide an ENTRYPOINT
intead?Sacha Ventura
03/29/2022, 5:36 AMBennett Lambert
03/29/2022, 7:21 AMprefect.exceptions.ClientError: Malformed response received from API.
BBPP
03/29/2022, 7:58 AMАндрій Демиденко
03/29/2022, 10:32 AMMichael Smith
03/29/2022, 11:15 AMChu Lục Ninh
03/29/2022, 11:25 AMprefect.tasks.kubernetes.job.RunNamespacedJob
doesn't report properly to prefect server. I didn't receive any log from spawned kubernetes job at all. Here are my screenshotsCamilo Fernandez
03/29/2022, 12:18 PMprefect run -i "efc4568a-1fbd-4a14-8ef1-f137bdb45fcf" --watch
I receive the following warning:
WARNING | It has been 270 seconds and your flow run has not been submitted by an agent. Agent 122a102c-f083-48a8-b6f4-efd38e1b14f6 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
Mia
03/29/2022, 12:43 PMhelm upgrade
ran successfully, I get commands such as
1 Run the following command to get the UI URL:
UI_HOST=$( \
kubectl get svc \
--namespace prefect \
--template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}" \
prefect-server-ui \
) \
&& echo "UI available at: http://$UI_HOST:8080"...
when I run the above command, I get an error saying
error: error executing template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}": template: output:1:10: executing "output" at <index .status.loadBalancer.ingress 0>: error calling index: index of untyped nil
Then I have to run kubectl port-forward <name of the service> <port>:<port>
to get ui and apollo servers running. Is there a way to run this without port forwarding?
2. I’m trying to run a hello world example flow on prefect server on my EKS cluster. I tried to trigger it from my command line by creating a project and running prefect register --project test-project -p hello/
. The registering seem to run successfully but I don’t see the flow on the ui. I’m also not able to manually create flow project using ui. Am I interacting with two different prefect engine here? How do I register prefect flow so that it is running on my prefect server on EKS cluster?Zhiyuan Ma
03/29/2022, 12:53 PM