Sami Serbey
05/12/2021, 5:55 PMPrefect Team
option disabled from Prefect Core's server ?Max Lei
05/12/2021, 7:46 PM[2021-05-12 12:26:05-0700] INFO - prefect.task_name | Waiting for next scheduled run at 2021-05-13T10:00:00+00:00
2021-05-12 12:26:05,221 - prefect.task_name - INFO - Waiting for next scheduled run at 2021-05-13T10:00:00+00:00
But the local agent somehow runs two of the same flow twice.
How should I debug this issue?Andrew Nichol
05/12/2021, 10:02 PMLet's say that you want to always run a flow with parameters that are generated by another flow. Naively, you might manually wait for one flow to finish and then manually trigger the next flow to run with the appropriate parameter values. Prefect makes this pattern easy to automate via the StartFlowRun:
but the example code given represents a flow which is dependent upon a task that returns parameters, not a flow that is dependent upon another flow.
i think the main piece im struggling with is how to return/access some values of the preceding flow from the succeeding flow. (basically getting the input to the parameters
arg of the succeeding flow)Joseph Loss
05/12/2021, 10:09 PMChris McClellan
05/12/2021, 11:08 PMLeandro Mana
05/12/2021, 11:33 PMDaskKubernetesEnvironment
was supported and that worked great, now Im using the run_configs
and executors
and Im having the botocore region issue displayed, not sure Im missing any argument for the DaskExecutor
please check the code bellow:
flow.storage = Docker(registry_url=ecr,
python_dependencies=python_dependencies,
files={f'{getcwd()}/src/dask_flow': '/modules/dask_flow'},
env_vars={"PYTHONPATH": "$PYTHONPATH:modules/"},
image_tag='latest')
flow.run_config=KubernetesRun()
flow.executor=DaskExecutor(
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={'n_workers': 5, 'region_name':aws_region})
Sébastien Arnaud
05/13/2021, 3:40 AMclient
object from Prefect's DaskExecutor? I need to be able to upload a python egg to the Dask cluster before starting executing Tasks (which I would achieve usually with client.upload_file)
Thank you in advance!Matej
05/13/2021, 7:31 AMModuleNotFoundError: No module named 'aircraftlib'
[2021-05-13 09:27:29+0200] ERROR - prefect.etl | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'aircraftlib'")
aircraftlib is a module from the tutorial imported by the flow.
it works when I run it all locally, it does not work if i send it to a remote dask scheduler.
I could place it all in a single file "flow.py" but is there an elegant way to do this?
thanks.Matej
05/13/2021, 9:23 AMhelm install prefect prefecthq/prefect-server -f values.yaml
with the following error:
Error: YAML parse error on prefect-server/templates/hasura/deployment.yaml: error converting YAML to JSON: yaml: line 31: mapping values are not allowed in this context
Attached you can find the values.yaml file that I try to provide for the helm install command.
The only modification is the securityContext:
securityContext:
runAsUser: 1000
runAsGroup: 1000
instead of
securityContext: {}
for hasura service
The errored line 31 of the hasura deployment.yaml looks as follows:
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
Thanks for help.Alessio Civitillo
05/13/2021, 12:22 PMLone Pine Account
05/13/2021, 1:37 PMciaran
05/13/2021, 1:52 PM0.14.17
-> 0.14.19
and I'm getting an error when my Dask Scheduler pod starts up:
Traceback (most recent call last):
File "/usr/local/bin/dask-scheduler", line 5, in <module>
from distributed.cli.dask_scheduler import go
File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 120, in <module>
def main(
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 247, in decorator
_param_memo(f, OptionClass(param_decls, **option_attrs))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2467, in __init__
super().__init__(param_decls, type=type, multiple=multiple, **attrs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2106, in __init__
raise ValueError(
ValueError: 'default' must be a list when 'multiple' is true.
Sean Harkins
05/13/2021, 3:56 PMRavikiran Sharvirala
05/13/2021, 4:13 PMNewskooler
05/13/2021, 4:24 PMCharles Leung
05/13/2021, 7:24 PMNelson Griffiths
05/13/2021, 7:56 PMJoseph Loss
05/13/2021, 8:11 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/root/'")
Braun Reyes
05/13/2021, 9:40 PMJeremy Yeo
05/14/2021, 12:33 AMIntervalSchedule
similar to having multiple clocks in the docs (https://docs.prefect.io/core/concepts/schedules.html#clocks)? What would be the recommended way to have a single flow that can run on 2 schedules (say every 5 minutes and every 30 minutes)? I guess we could probably achieve this with a 1 minute schedule and check within the flow run if now().minute == 30
or now().minute % 5 == 0
or some similar method but just wondering what would "prefecters" do...Kevin Kho
Kha Nguyen
05/14/2021, 12:51 PMfrom prefect.agent.ecs import ECSAgent
agent = ECSAgent(
name='Test Agent',
region_name='eu-west-1',
cluster='my-test-cluster',
launch_type='FARGATE',
run_task_kwargs_path='./config.yaml'
)
agent.start()
This then failed with the following message
[2021-05-14 12:46:54,069] ERROR - Test Agent | Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
Traceback (most recent call last):
File "/Users/kha/Desktop/foo/agent.py", line 3, in <module>
agent = ECSAgent(
File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 234, in __init__
] = self.infer_network_configuration()
File "/opt/homebrew/Caskroom/miniforge/base/envs/zoined/lib/python3.9/site-packages/prefect/agent/ecs/agent.py", line 274, in infer_network_configuration
raise ValueError(msg)
ValueError: Failed to infer default networkConfiguration, please explicitly configure using `--run-task-kwargs`
Kha Nguyen
05/14/2021, 2:30 PMimport prefect
from prefect import task, Flow
from prefect.run_configs import ECSRun
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello World!")
return 0
task_definition = {
'networkMode': 'awsvpc',
'cpu': 1024,
'memory': 2048,
'containerDefinitions': [
{'name': 'flow'}
]
}
run_config = ECSRun(
image='prefecthq/prefect',
task_definition=task_definition
)
flow = Flow("hello-flow",
tasks=[hello_task],
run_config=run_config)
flow.register(project_name='Prefect_Tutorials', labels=['aws', 'ecs', 'fargate'])
The ECS agent can successfully trigger an ECS task. However, the flow run failed with error Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/kha/'")
. Why does it include something specific to my name in the flow?Ben Davison
05/14/2021, 4:13 PMprefect register flow_name --label=staging --no-schedule
We've also messed about with setting a bad label on the schedule itself, which "kind of" works, as in the flow won't run because it can't find an agent, but, the UI still get overwhelmed with flows being scheduled, but not being able to be run.Zach Schumacher
05/14/2021, 4:54 PMLone Pine Account
05/14/2021, 5:14 PMLone Pine Account
05/14/2021, 5:14 PMLone Pine Account
05/14/2021, 5:14 PMfrom prefect import task, Flow
from prefect.tasks.secrets import PrefectSecret
from prefect.tasks.databricks import DatabricksRunNow
import requests
conncfg = {"host":"<https://xxx.cloud.databricks.com/>", "token":"xxx"}
job_json = {
"job_id": 24242,
"notebook_params": {}
}
@task
def run_databricks(job_config):
notebook_run = DatabricksRunNow(json=job_config).run()
run_id = notebook_run(databricks_conn_secret=conncfg)
headers = { "Authorization" : "Bearer xxx" }
r = requests.get('<https://xxx.cloud.databricks.com/api/2.0/jobs/runs/get-output?run_id=%s'%(runid)>, headers=headers)
results = r.json()
output = results['metadata']['notebook_output']
if output.has_key('message'):
raise Exception(output['message'])
with Flow('testflow') as flow:
run_databricks(job_json)
Lone Pine Account
05/14/2021, 5:15 PMLone Pine Account
05/14/2021, 5:15 PMUnexpected error: AssertionError('A databricks connection string must be supplied as a dictionary or through Prefect Secrets')