Miguel
03/23/2021, 5:25 PM[2021-03-23 17:16:53+0000] ERROR - prefect.my-flow | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'prefect'")
This is what I'm doing:
executor = DaskExecutor("my-dask-cluster-ip")
flow.run(executor=executor)
If I use other executors (LocalExecutor or LocalDaskExecutor) it runs fine.
Any idea on what I can do to address this issue?Charles Leung
03/23/2021, 9:34 PMINTERNAL_SERVER_ERROR: Variable "$input" got invalid value null at
"input.states[0].task_run_id"; Expected non-nullable type UUID! not to be null.
The GraphQL query was:
mutation($input: set_task_run_states_input!) {
set_task_run_states(input: $input) {
states {
message
status
id
}
}
}
The passed variables were:
{"input": {"states": [{"state": {"message": "Trigger was \"all_successful\" but some of the upstream tasks failed.", "_result": {"location": null, "__version__": "0.14.11", "type": "Result"}, "cached_inputs": {}, "context": {"tags": []}, "__version__": "0.14.11", "type": "TriggerFailed"}, "task_run_id": null, "version": null}]}}
what is the issue? I'm not sure how to debug this
Michael Hadorn
03/24/2021, 7:43 AMClemens
03/24/2021, 9:55 AMFinished task run for task with final state: 'Running'
after that the flows state is completely off and it get’s hard to understand what happens. But already that log seems wrong. Any clue about why and when a task might finish with a running state?Michael Hadorn
03/24/2021, 11:10 AMJosh Greenhalgh
03/24/2021, 1:44 PMpostgresql:
postgresqlPassword: ***
nodeSelector:
<http://cloud.google.com/gke-nodepool|cloud.google.com/gke-nodepool>: some-node-pool
Josh Greenhalgh
03/24/2021, 2:37 PMcore-0.14.12
and I am getting this error;
{'type': ['Unsupported value: Module']}
Has module storage been removed?Charles Leung
03/24/2021, 9:54 PMMilton
03/24/2021, 10:30 PMDominique
03/25/2021, 2:51 AMJillian Kozyra
03/25/2021, 7:32 AMModuleNotFoundError: No module named 'data_flows'
when I try to register my flow locally using docker storage (specifically it’s failing the cloudpickle_deserialization_check
healthcheck). we’re using fully qualified imports in our flows (e.g. from data_flows.utils.data_extraction tasks import thing
) to satisfy mypy, so i’m hoping there’s a way to get this working as well. i think the config matches the tutorial, and i’ve tried different values for the python path without any luck. Here’s our Docker setup:
Docker(
registry_url=ecr_registry_url,
python_dependencies=python_dependencies,
files={
"/Users/jilliankozyra/projects/projectname/data_flows/utils/__init__.py": "/data_flows/utils/__init__.py",
"/Users/jilliankozyra/projects/projectname/data_flows/utils/install_dependencies.py": "/data_flows/utils/install_dependencies.py",
"/Users/jilliankozyra/projects/projectname/data_flows/utils/data_extraction_tasks.py": "/data_flows/utils/data_extraction_tasks.py"
},
env_vars={
"PYTHONPATH": "$PYTHONPATH:data_flows/"
},
)
Milton
03/25/2021, 3:24 PMCharles Leung
03/25/2021, 4:37 PMAdam Lewis
03/26/2021, 7:29 PMCharles Leung
03/26/2021, 7:43 PMDana Merrick
03/26/2021, 9:18 PMRunNamespacedJob
task?Michael Hadorn
03/29/2021, 12:57 PMGopinath Jaganmohan
03/30/2021, 2:28 AMNathan Walker
03/30/2021, 1:00 PMDiego Alonso Roque Montoya
03/30/2021, 3:38 PMDiego Alonso Roque Montoya
03/30/2021, 6:26 PMRoss
03/31/2021, 9:16 AMprefect.engine.flow_runner.FlowRunner
or prefect.engine.cloud.flow_runner.CloudFlowRunner
with a prefect server backend? I am wanting to run a flow and get task results so wanting to use the arg return_tasks=[task1,task2]
. But I don't think I can do this with prefect.Client().create_flow_run(version_group_id="redacted")
?Tomás Emilio Silva Ebensperger
03/31/2021, 1:38 PMJonathan Wright
03/31/2021, 5:26 PMprefect server create-tenant -n data-eng
Tomás Emilio Silva Ebensperger
04/02/2021, 12:51 PMSidd
04/02/2021, 5:06 PMeric lumsden
04/05/2021, 12:11 AMpip install prefect[aws]
zsh: no matches found: prefect[aws]
Luca Sorgiacomo
04/05/2021, 7:32 AMasync def
 and await
in defining and within tasks?Luca Sorgiacomo
04/05/2021, 7:33 AMeric lumsden
04/05/2021, 5:30 PMValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
eric lumsden
04/05/2021, 5:30 PMValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
Kevin Kho
04/05/2021, 5:35 PMeric lumsden
04/05/2021, 5:43 PMKevin Kho
04/05/2021, 5:45 PMeric lumsden
04/05/2021, 5:47 PMKevin Kho
04/05/2021, 5:50 PMeric lumsden
04/05/2021, 5:53 PMKevin Kho
04/05/2021, 5:54 PMeric lumsden
04/05/2021, 5:56 PMKevin Kho
04/05/2021, 5:57 PMeric lumsden
04/05/2021, 6:01 PMKevin Kho
04/05/2021, 6:02 PMaws ecs create-cluster
try this?eric lumsden
04/05/2021, 6:11 PMKevin Kho
04/05/2021, 6:15 PMeric lumsden
04/05/2021, 6:15 PMKevin Kho
04/05/2021, 6:17 PMeric lumsden
04/05/2021, 6:25 PMKevin Kho
04/05/2021, 6:27 PMRUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
run_task_kwargs={
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
"securityGroups": ["sg-123456"],
}
}
},
)
run_config=RUN_CONFIG
. get the VPC configuration from the default subnetseric lumsden
04/05/2021, 6:30 PMKevin Kho
04/05/2021, 6:44 PMFlow()
call.
RUN_CONFIG = ECSRun(execution_role_arn="arn:aws:iam::1234:role/ecsTaskExecutionRole",
run_task_kwargs={
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"subnets": ['subnet-123456', 'subnet-123456', 'subnet-123456'],
"securityGroups": ["sg-123456"],
}
}
},
)
with Flow(name="ecs-test", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
print_log()
flow.run()
eric lumsden
04/05/2021, 7:42 PMKevin Kho
04/05/2021, 7:50 PMflow.py
?prefect
inside?print_log
with print("1")
to test this block. You don’t have the STORAGE and EXECUTOR defined though so this will error. Do you have the original snippet that caused the error? Maybe we can take a look at thateric lumsden
04/05/2021, 8:02 PMKevin Kho
04/05/2021, 8:04 PMSTORAGE
is to point to where your code lives. For ECS it might point to the Docker image location. Something like:
STORAGE = Docker(
registry_url="<http://1234.dkr.ecr.us-east-2.amazonaws.com|1234.dkr.ecr.us-east-2.amazonaws.com>",
image_name="ecs-test",
)
eric lumsden
04/05/2021, 8:04 PMKevin Kho
04/05/2021, 8:05 PMeric lumsden
04/05/2021, 8:06 PMKevin Kho
04/05/2021, 8:06 PMeric lumsden
04/06/2021, 3:38 PM--run-task-kwargs
I'm guessing I need to set the networkMode to an existing vpc. Unfortunately my environment shifted it's hosting region overnight so I am pointed somewhere with no VPCs. How do I point to the new region and select a VPC?Kevin Kho
04/06/2021, 3:41 PMnetworkMode: awsvpc
cpu: 256
memory: 512
containerDefinitions:
- name: flow
requiresCompatibilities:
- FARGATE
executionRoleArn: arn:aws:iam::xxxxxxxxxx:role/tf-iam-role-prefect-ecs-tasks <--- change to whatever you need
eric lumsden
04/06/2021, 3:58 PMKevin Kho
04/06/2021, 4:00 PMeric lumsden
04/06/2021, 4:00 PMKevin Kho
04/06/2021, 4:21 PMrun_task.yaml
 with this:
networkConfiguration:
awsvpcConfiguration:
subnets:
- 'subnet-1234'
securityGroups:
- 'sg-1234'
assignPublicIp: 'ENABLED'
run prefect agent ecs start --run-task-kwargs /ecs/run_task.yaml
eric lumsden
04/06/2021, 4:37 PMKevin Kho
04/06/2021, 4:59 PMeric lumsden
04/06/2021, 5:12 PMprefect agent ecs start --run-task-kwargs <s3://bucketname>
prefect.utilities.exceptions.AuthorizationError: No agent API token provided.
Kevin Kho
04/06/2021, 5:21 PMprefect agent ecs start
on your local machine and you want the local machine to kick off ECS jobs right?eric lumsden
04/06/2021, 5:53 PMKevin Kho
04/06/2021, 5:55 PMeric lumsden
04/06/2021, 6:02 PMKevin Kho
04/06/2021, 6:08 PMeric lumsden
04/06/2021, 6:30 PMKevin Kho
04/06/2021, 6:36 PM