Bennett Lambert
03/29/2022, 7:21 AMprefect.exceptions.ClientError: Malformed response received from API.
PB
03/29/2022, 7:58 AMАндрій Демиденко
03/29/2022, 10:32 AMMichael Smith
03/29/2022, 11:15 AMChu Lục Ninh
03/29/2022, 11:25 AMprefect.tasks.kubernetes.job.RunNamespacedJob
doesn't report properly to prefect server. I didn't receive any log from spawned kubernetes job at all. Here are my screenshotsCamilo Fernandez
03/29/2022, 12:18 PMprefect run -i "efc4568a-1fbd-4a14-8ef1-f137bdb45fcf" --watch
I receive the following warning:
WARNING | It has been 270 seconds and your flow run has not been submitted by an agent. Agent 122a102c-f083-48a8-b6f4-efd38e1b14f6 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
Maria
03/29/2022, 12:43 PMhelm upgrade
ran successfully, I get commands such as
1 Run the following command to get the UI URL:
UI_HOST=$( \
kubectl get svc \
--namespace prefect \
--template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}" \
prefect-server-ui \
) \
&& echo "UI available at: http://$UI_HOST:8080"...
when I run the above command, I get an error saying
error: error executing template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}": template: output:1:10: executing "output" at <index .status.loadBalancer.ingress 0>: error calling index: index of untyped nil
Then I have to run kubectl port-forward <name of the service> <port>:<port>
to get ui and apollo servers running. Is there a way to run this without port forwarding?
2. I’m trying to run a hello world example flow on prefect server on my EKS cluster. I tried to trigger it from my command line by creating a project and running prefect register --project test-project -p hello/
. The registering seem to run successfully but I don’t see the flow on the ui. I’m also not able to manually create flow project using ui. Am I interacting with two different prefect engine here? How do I register prefect flow so that it is running on my prefect server on EKS cluster?Zhiyuan Ma
03/29/2022, 12:53 PMDaniel Komisar
03/29/2022, 1:19 PMAlex Prokop
03/29/2022, 3:14 PMWieger Opmeer
03/29/2022, 3:27 PMKevin Kho
03/29/2022, 4:02 PMRaimundo Pereira De Souza Neto
03/29/2022, 5:35 PMprefect = "2.0a13"
, I did read this link, but I'm using async flows.
More infos in the thread.Ben Welsh
03/29/2022, 6:33 PMBen Welsh
03/29/2022, 6:34 PMScott Aefsky
03/29/2022, 7:38 PMtask_definition_path
, and running into an issue. My flows run on different ECR images, so I don't want to specify the image in the Task Definition file. However, when I leave it out, and specify it in the constructor:
run_config = ECS_RUN(image=ECR_IMAGE, task_definition_path='s3://<path>'
I get an error that container.image should not be empty. My task definition file that I'd like to use is:
networkMode: awsvpc
cpu: 1024
memory: 2048
containerDefinitions:
- name: flow
logConfiguration:
logDriver: awslogs
options:
awslogs-group: test_prefect_log_group
awslogs-region: us-east-1
awslogs-stream-prefix: ecs-prefect
awslogs-create-group: "True"
Is there any way to use a Task Definition file without specifying an image? Or is there another/better way to make sure my ECS tasks get logged? Thanks for any help.Wei Mei
03/29/2022, 8:51 PMdate
and today
context in a task, but getting None.
date = prefect.context.get("date")
today = prefect.context.get("today")
def get_data():
print(f"{date}")
print(today)
Dominick Olivito
03/30/2022, 12:27 AMprint("hello world 1")
if not skip_task2:
print("hello world 2")
if not skip_task3:
print("hello world 3")
print("hello world 4")
here's an attempt in Prefect:
@task()
def print_task(value: str) -> str:
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(value)
return value
with Flow("case_skip_flow") as flow:
skip_task2 = Parameter("skip_task2", True)
skip_task3 = Parameter("skip_task3", True)
task1 = print_task("hello world 1")
with case(skip_task2, False):
task2 = print_task("hello world 2", upstream_tasks=[task1])
with case(skip_task3, False):
task3 = print_task(
"hello world 3", upstream_tasks=[task1, task2], task_args={"skip_on_upstream_skip": False}
)
task4 = print_task("hello world 4", upstream_tasks=[task3], task_args={"skip_on_upstream_skip": False})
i'm getting into trouble with the upstream skip status for task3
.
• if I set skip_on_upstream_skip
to True
, then when task2
is skipped, so is task3
regardless of the value for skip_task3
• if I set skip_on_upstream_skip
to False
, then task3
runs regardless of the value for skip_task3
how can i implement a pattern like this?Maria
03/30/2022, 6:35 AMflow.run_config = DockerRun(image="my_image", host_config={"binds":["/var/run/docker.sock:/var/run/docker.sock"]})
But I also need to pull an image when its not available and this step fails since I am not authenticated from inside my flow container.
I can probably create a shell task that does docker login for me before image pull, but I am wondering if there are better options?Michael Smith
03/30/2022, 8:00 AMRamonV
03/30/2022, 8:03 AMYeachan Park
03/30/2022, 8:12 AMep
03/30/2022, 9:00 AMAndreas Ntonas
03/30/2022, 9:17 AMJacob Blanco
03/30/2022, 9:18 AMMatthias
03/30/2022, 11:05 AMEddie Atkinson
03/30/2022, 11:13 AMprefect/task/{ECS-task-id}
. My aim is to display those logs in our webapp to show admin users the status of the flows they’ve started
Ideally I’d be able to use the ID of a flow run and use that to query the ECS task it ran on and then use that to query the logs of the flowBennett Lambert
03/30/2022, 11:13 AMAndreas Nord
03/30/2022, 11:16 AMJose Daniel Posada Montoya
03/30/2022, 1:27 PMJose Daniel Posada Montoya
03/30/2022, 1:27 PMAnna Geller
03/30/2022, 1:36 PMMarvin
03/30/2022, 1:38 PMJose Daniel Posada Montoya
03/30/2022, 1:54 PMMichael Smith
03/30/2022, 3:58 PMJose Daniel Posada Montoya
03/30/2022, 4:07 PMcreate_flow_run_from_deployment