Leon Kozlowski
01/31/2022, 9:10 PMDaniel Kornhauser
01/31/2022, 10:26 PMYorrick Mendes
01/31/2022, 10:34 PMSuresh R
02/01/2022, 7:40 AMINFO
CloudTaskRunner
Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
INFO
CloudTaskRunner
Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
Vinoth Kumar
02/01/2022, 8:03 AMTony Waddle
02/01/2022, 12:03 PMMuddassir Shaikh
02/01/2022, 12:31 PMhandler = slack_notifier(only_states=[Failed])
do not contain details of the flow name and the taskname, how can we add that?Konstantin
02/01/2022, 12:41 PMHello everyone, I can't figure out how to use native libraries in Prefect, for example "prefect.tasks.gsheets"
Give an example of the code for using this class
Muddassir Shaikh
02/01/2022, 12:54 PMMadhup Sukoon
02/01/2022, 1:57 PMflow.py
file and then run it via a Docker Agent. I also have some external (pip-installable) dependencies that would need to be present for the flow to run.
Is it possible to do this? Does someone have an example Flow that uses Github Storage and Docker Run?Vladimir Bolshakov
02/01/2022, 2:09 PMlocalhost:4200
. I think the reason is here: https://github.com/PrefectHQ/prefect/blob/orion/orion-ui/src/plugins/api.ts#L198Madhup Sukoon
02/01/2022, 5:24 PMenv
or host_config
) to have the Docker Agent install packages in the docker container at runtime?
(Looking for something similar to Docker Storage's python_dependencies
, but for runtime)Constantino Schillebeeckx
02/01/2022, 5:44 PMMadhup Sukoon
02/01/2022, 6:16 PMScheduled
> Submitted
and then nothing happens. The only entry in logs is:
Submitted for execution: Container ID: 143cabe06c721.....
My flow uses GitHub Storage and DockerRun, and uses some secrets. Any way I could get more verbose logs on what might be happening?
This is what my flow.py
looks like:
from main import planner, creator, publisher
planner = task(planner)
creator = task(creator)
publisher = task(publisher)
foo = Secret('foo')
bar = Secret('bar')
with Flow(
"myflow",
run_config = DockerRun(
env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
),
storage = GitHub(
repo="me/myrepo",
path="flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN"
)
) as flow:
plan = planner()
media = creator(foo, bar, plan)
post = publisher(foo, bar, media)
main
is an adjacent python file (main.py
)Davoud Hejazi
02/01/2022, 9:22 PMfrom prefect.utilities.notifications import slack_notifier
from prefect.engine.state import Failed
SLACK_WEBHOOK_URL = "the slack secret webhook url"
# os.environ["PREFECT__CONTEXT__SECRETS__SLACK_WEBHOOK_URL"] = "<https://hooks.slack.com/services/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>"
handler = slack_notifier(only_states=[Failed])
prefect.context.setdefault("secrets", {})
prefect.context.secrets["SLACK_WEBHOOK_URL"] = "the slack secret webhook url"
I have even created config.toml file in .predict directory with the content below:
[context.secrets]
SLACK_WEBHOOK_URL = "the slack secret webhook url"
My task stats with:
@task(nout=2, max_retries=1, retry_delay=timedelta(seconds=1), checkpoint=False, state_handlers=[handler])
I also tried to add prefect app to slack. But when running my flows I am seeing this error (part of the error): see thread
To make it clear, when I remove the state_handlers=[handler]
from my task definition the flow runs fine. When I visit the prefect app in slack I see this too, which I am not sure if it connected to my problem or not. Any comments on how I can solve the slack notification problem?Kevin Mullins
02/01/2022, 10:33 PMShaoyi Zhang
02/01/2022, 11:49 PMHoratiu Bota
02/02/2022, 8:37 AMflow.run(parameters=json.loads(open(file).read()))
-- the parameters file looks something like this:
{
"other_params": {},
"company_params": {
"company_id": "ID1",
"company_name": "Company",
"datasets": [
{
"dataset_label": "label",
"dataset_url": "URL"
},
]
}
}
is there any way to access nested values in the actual flow? something like:
with Flow("flow") as flow:
company_params = Parameter("company_params")
datasets = company_params.get("datasets")
Suresh R
02/02/2022, 10:45 AMChris Arderne
02/02/2022, 12:09 PM@task
decorator has log_stdout=True
but is there any way to add stderr
to logging?shijas km
02/02/2022, 2:50 PMConstantino Schillebeeckx
02/02/2022, 3:36 PMDavid Wang
02/02/2022, 4:02 PMessential container in task exited
. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of ResourceInitializationError: failed to validate logger args: : signal: killed
. I’ve double checked with devOps that the IAM roles and network configurations should be correct too.
Any ideas on how to debug this or why this is happening?E Li
02/02/2022, 4:08 PMJason Motley
02/02/2022, 5:26 PMAn Hoang
02/02/2022, 5:38 PMflow_A
with task1, task2... task10
. Now I want to construct flow_B
with task1, task2.... task 20
and task11_result = task11(task10_result)
. How do I add the tasks of flow_A
as upstream tasks for flow_B
. I know I can do create_flow_run
, but I do not want monitoring for task1...task10
in flow_B
Kind of like:
with Flow("flow_A") as flow:
flow_A_param1 = Parameter("flow_A_param1")
...
flow_A_param_n = Parameter(...)
...
task1 = ....
...
task10_result = task10(...)
with Flow("flow_B") as flow:
flow_A_param1 = Parameter(...)
...
flow_A_param_n = Parameter(...)
flow_A_result = flow_A.run(parameters = {"flow_A_param1": flow_A_param1,...., "flow_A_param_n": flow_A_param_n}) #dont want monitoring for these tasks
task10_result = flow_A_result.result[flow_A.get_task(task10)]
task11_result = task11(task10_result)
...
task20_result = task20(...)
where `flow_B`'s parameters are just `flow_A`'s parameters. But in the example above I always have to repeat the parameters of flow_A
in flow_B
and keep them both up to date with each other. Is there a better way to do this?Chris Reuter
02/02/2022, 6:00 PMDevin Flake
02/02/2022, 7:51 PMNelson Griffiths
02/02/2022, 9:58 PMjspeis
02/03/2022, 1:43 AMupstream_tasks
kwarg (e.g. as in example below) I get an Unexpected keyword argument 'upstream_tasks' in function call (unexpected-keyword-arg)
... is there a way I can get pylint to recognize this is ok? (other than disabling?)
@task
def add_one(val):
return val + 1
with Flow("Add One") as flow:
p = Parameter("param")
first = add_one(p)
second = add_one(p, upstream_tasks=[first])