Dev Dasgupta
01/31/2022, 5:13 PMprefect register --project my_project -p bar
)
foo_root_folder
├── README.md
├── bar
│ └── flow.py
├── pyproject.toml
├── requirements.txt
├── setup.cfg
Since the file flow.py
was cluttered with all utils functions required for the flow execution I refactored it into a separate python package and made calls to those functions in flow.py
from bar.utils import get_some_useful_function
The resultant folder looked like the following:
foo_root_folder
├── README.md
├── bar
│ ├── __init__.py
│ ├── utils.py
│ └── flow.py
├── pyproject.toml
├── requirements.txt
├── setup.cfg
When I try to register now I am getting error as ModuleNotFoundError: No module named 'bar'
Can anyone please let me know if anyone faced similar issues before and how they resolved it?
ThanksAndrew Lawlor
01/31/2022, 5:38 PMKeith Veleba
01/31/2022, 5:53 PMDavid Yang
01/31/2022, 6:46 PMKeith Veleba
01/31/2022, 7:16 PMChris Hemphill
01/31/2022, 8:05 PMDave
01/31/2022, 8:21 PMCouldn't connect to *Prefect Server* at *<http://34.82.119.23:4200/graphql>
,* can anyone help?Leon Kozlowski
01/31/2022, 9:10 PMDaniel Kornhauser
01/31/2022, 10:26 PMYorrick Mendes
01/31/2022, 10:34 PMSuresh R
02/01/2022, 7:40 AMINFO
CloudTaskRunner
Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
INFO
CloudTaskRunner
Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
Vinoth Kumar
02/01/2022, 8:03 AMTony Waddle
02/01/2022, 12:03 PMMuddassir Shaikh
02/01/2022, 12:31 PMhandler = slack_notifier(only_states=[Failed])
do not contain details of the flow name and the taskname, how can we add that?Konstantin
02/01/2022, 12:41 PMHello everyone, I can't figure out how to use native libraries in Prefect, for example "prefect.tasks.gsheets"
Give an example of the code for using this class
Muddassir Shaikh
02/01/2022, 12:54 PMMadhup Sukoon
02/01/2022, 1:57 PMflow.py
file and then run it via a Docker Agent. I also have some external (pip-installable) dependencies that would need to be present for the flow to run.
Is it possible to do this? Does someone have an example Flow that uses Github Storage and Docker Run?Vladimir Bolshakov
02/01/2022, 2:09 PMlocalhost:4200
. I think the reason is here: https://github.com/PrefectHQ/prefect/blob/orion/orion-ui/src/plugins/api.ts#L198Madhup Sukoon
02/01/2022, 5:24 PMenv
or host_config
) to have the Docker Agent install packages in the docker container at runtime?
(Looking for something similar to Docker Storage's python_dependencies
, but for runtime)Constantino Schillebeeckx
02/01/2022, 5:44 PMMadhup Sukoon
02/01/2022, 6:16 PMScheduled
> Submitted
and then nothing happens. The only entry in logs is:
Submitted for execution: Container ID: 143cabe06c721.....
My flow uses GitHub Storage and DockerRun, and uses some secrets. Any way I could get more verbose logs on what might be happening?
This is what my flow.py
looks like:
from main import planner, creator, publisher
planner = task(planner)
creator = task(creator)
publisher = task(publisher)
foo = Secret('foo')
bar = Secret('bar')
with Flow(
"myflow",
run_config = DockerRun(
env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
) as flow:
plan = planner()
media = creator(foo, bar, plan)
post = publisher(foo, bar, media)
main
is an adjacent python file (main.py
)Davoud Hejazi
02/01/2022, 9:22 PMfrom prefect.utilities.notifications import slack_notifier
from prefect.engine.state import Failed
SLACK_WEBHOOK_URL = "the slack secret webhook url"
# os.environ["PREFECT__CONTEXT__SECRETS__SLACK_WEBHOOK_URL"] = "<https://hooks.slack.com/services/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>"
handler = slack_notifier(only_states=[Failed])
prefect.context.setdefault("secrets", {})
prefect.context.secrets["SLACK_WEBHOOK_URL"] = "the slack secret webhook url"
I have even created config.toml file in .predict directory with the content below:
[context.secrets]
SLACK_WEBHOOK_URL = "the slack secret webhook url"
My task stats with:
@task(nout=2, max_retries=1, retry_delay=timedelta(seconds=1), checkpoint=False, state_handlers=[handler])
I also tried to add prefect app to slack. But when running my flows I am seeing this error (part of the error): see thread
To make it clear, when I remove the state_handlers=[handler]
from my task definition the flow runs fine. When I visit the prefect app in slack I see this too, which I am not sure if it connected to my problem or not. Any comments on how I can solve the slack notification problem?Kevin Mullins
02/01/2022, 10:33 PMShaoyi Zhang
02/01/2022, 11:49 PMHoratiu Bota
02/02/2022, 8:37 AMflow.run(parameters=json.loads(open(file).read()))
-- the parameters file looks something like this:
{
"other_params": {},
"company_params": {
"company_id": "ID1",
"company_name": "Company",
"datasets": [
{
"dataset_label": "label",
"dataset_url": "URL"
},
]
}
}
is there any way to access nested values in the actual flow? something like:
with Flow("flow") as flow:
company_params = Parameter("company_params")
datasets = company_params.get("datasets")
Suresh R
02/02/2022, 10:45 AMChris Arderne
02/02/2022, 12:09 PM@task
decorator has log_stdout=True
but is there any way to add stderr
to logging?shijas km
02/02/2022, 2:50 PMConstantino Schillebeeckx
02/02/2022, 3:36 PMDavid Wang
02/02/2022, 4:02 PMessential container in task exited
. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of ResourceInitializationError: failed to validate logger args: : signal: killed
. I’ve double checked with devOps that the IAM roles and network configurations should be correct too.
Any ideas on how to debug this or why this is happening?David Wang
02/02/2022, 4:02 PMessential container in task exited
. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of ResourceInitializationError: failed to validate logger args: : signal: killed
. I’ve double checked with devOps that the IAM roles and network configurations should be correct too.
Any ideas on how to debug this or why this is happening?Kevin Kho
02/02/2022, 4:05 PMResourceInitializationError
myself. Are you using the base Prefect image or your own? Are you just following the docs or do you have any added configuration?David Wang
02/02/2022, 4:06 PMKevin Kho
02/02/2022, 4:15 PMDavid Wang
02/02/2022, 4:29 PMKevin Kho
02/02/2022, 4:31 PMDavid Wang
02/02/2022, 4:32 PM"image": "prefecthq/prefect:latest-python3.8",
Kevin Kho
02/02/2022, 4:34 PMDavid Wang
02/02/2022, 10:20 PM"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$ECS_LOG_GROUP_NAME",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "true"
}
}
Kevin Kho
02/02/2022, 10:23 PMtask exited
can also be due to a lack of IAM permissionsDavid Wang
02/07/2022, 4:21 PMCannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "<http://docker.io/prefecthq/prefect:latest-python3.8|docker.io/prefecthq/prefect:latest-python3.8>": failed to do request: Head <https://registry-1.docker.io/v2/prefecthq/prefect/manifests/latest-python3.8>: dial tcp ...
error while I was messing around with the security groupsAnna Geller
02/07/2022, 5:08 PMassignPublicIp=ENABLED
You can set up as part of the network configuration:
aws ecs create-service \
--service-name $ECS_SERVICE_NAME\
--task-definition $ECS_SERVICE_NAME:1 \
--desired-count 1 \
--launch-type FARGATE \
--platform-version LATEST \
--cluster $ECS_CLUSTER_NAME \
--network-configuration awsvpcConfiguration="{subnets=[$SUBNET1, $SUBNET2, $SUBNET3],assignPublicIp=ENABLED}" --region $AWS_REGION
To explain it a bit more - my intuition (not 100% sure) is that your flow run container doesn't have access to the Internet in order to pull the container image from DockerhubDavid Wang
02/07/2022, 6:50 PMAnna Geller
02/07/2022, 6:57 PMDavid Wang
02/07/2022, 6:59 PMAnna Geller
02/07/2022, 7:04 PMhttps://www.youtube.com/watch?v=o_qSS4S1g34▾
https://www.youtube.com/watch?v=eq4wL2MiNqo▾
David Wang
02/07/2022, 9:06 PMAnna Geller
02/08/2022, 2:43 PMDavid Wang
02/08/2022, 4:08 PMResourceInitializationError: failed to validate logger args: : signal: killed
error again. And even if I remove the log configuration part to see if it will run it will give the essential container in task exited
Kevin Kho
02/08/2022, 4:25 PMDavid Wang
02/08/2022, 10:33 PMParameter validation failed: Missing required parameter in networkConfiguration.awsvpcConfiguration: "subnets" Unknown parameter in networkConfiguration.awsvpcConfiguration: "Subnets", must be one of: subnets, securityGroups, assignPublicIp
. Do I need to specify network configurations again somewhere in the run config?Kevin Kho
02/08/2022, 11:09 PMDavid Wang
02/09/2022, 4:19 PMKevin Kho
02/09/2022, 4:24 PMDavid Wang
02/09/2022, 4:24 PMKevin Kho
02/09/2022, 4:31 PMtask_definition_path
. That links to here , and then you can specify those subnet and security group there.
The ECS agent also takes a definition upon starting . You can pass --task_definition_path
. You just need to make sure that these live somewhere the agent can pull during runtime (like an S3 bucket it has access to).
The agent task_definition serves as a default for the Flows that it runs, but the RunConfig can override it. The default one the agent uses can be found hereDavid Wang
02/09/2022, 8:10 PMnetworkMode: awsvpc
cpu: 512
memory: 1024
containerDefinitions:
- name: prefectEcsAgent
networkConfiguration:
awsvpcConfiguration:
Subnets:
- subnet-xxx
securityGroups:
- sg-xxx
assignPublicIp: DISABLED
Kevin Kho
02/09/2022, 8:21 PMrun_task_kwargs
or on the agent
If you specify the awsvpc network mode, the task is allocated an elastic network interface, and you must specify a NetworkConfiguration when you create a service or run a task with the task definition.
This is so painful I dunno why you can’t do it as part of the task definitionDavid Wang
02/09/2022, 10:17 PMKevin Kho
02/09/2022, 10:22 PMDavid Wang
02/10/2022, 3:57 PM{
"family": "$ECS_SERVICE_NAME",
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "512",
"memory": "1024",
"taskRoleArn": "arn:aws:iam::xxx:role/prefectTaskRole",
"executionRoleArn": "arn:aws:iam::xxx:role/prefectECSAgentTaskExecutionRole",
"containerDefinitions": [
{
"name": "$ECS_SERVICE_NAME",
"image": "prefecthq/prefect",
"essential": true,
"command": [
"prefect",
"agent",
"ecs",
"start",
"--run-task-kwargs",
"<s3://xxx-test-bucket/david/ecs-config.yaml>"
],
"environment": [
{
"name": "PREFECT__CLOUD__API_KEY",
"value": "xxx"
},
{
"name": "PREFECT__CLOUD__AGENT__LABELS",
"value": "['dev']"
},
{
"name": "PREFECT__CLOUD__AGENT__LEVEL",
"value": "INFO"
},
{
"name": "PREFECT__CLOUD__API",
"value": "<https://api.prefect.io>"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$ECS_LOG_GROUP_NAME",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "true"
}
}
}
]
}
and inside the yaml file
networkConfiguration:
awsvpcConfiguration:
Subnets:
- subnet-xxx
securityGroups:
- sg-xxx
assignPublicIp: DISABLED
I also have the network configuration setup when creating the service with aws ecs create-service
aws ecs create-service \
--service-name $ECS_SERVICE_NAME\
--task-definition $ECS_SERVICE_NAME:1 \
--desired-count 1 \
--launch-type FARGATE \
--cluster $ECS_CLUSTER_NAME \
--network-configuration awsvpcConfiguration="{subnets=[$SUBNET1],securityGroups=[$SECURITYGROUP]}" --region $AWS_REGION
RUN_CONFIG = ECSRun(
labels=["dev"],
task_role_arn="arn:aws:iam::xxx:role/prefectTaskRole",
run_task_kwargs=dict(cluster="prefectEcsCluster",networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-xxx'
],
'securityGroups': [
'sg-xxx',
],
'assignPublicIp': 'DISABLED'
}
}),
)
Christopher
02/11/2022, 6:53 AMDavid Wang
02/11/2022, 4:28 PM