Zach Schumacher
06/02/2021, 8:06 PMChris Leber
06/02/2021, 8:52 PMJustin Chavez
06/02/2021, 9:46 PMfrom typing import Dict, List
from prefect import task, Flow
@task
def generate_user_status_list() -> List[Dict]:
return [
{
"user": "User 1",
"status": "active"
},
{
"user": "User 2",
"status": "inactive"
}
]
@task(log_stdout=True)
def print_user_status(user: str, status: str):
print(f"Status of {user} is {status}")
with Flow('check user status') as flow:
user_status_list = generate_user_status_list
print_user_status.map(user_status_list) # <<<< Fails here as it doesn't recognize the parameters in the dictionaries
flow.run()
It's the second to last line that I am trying to figure out. I want each member of the generated list in the first task to be the parameters to the second. It's like I am trying to do the parameterized flow run in the StartFlowRun section, except I want to run a task instead of a flow with the parameters: https://docs.prefect.io/core/idioms/flow-to-flow.html#running-a-parametrized-flow
I can get around this by creating separate tasks that create each parameter as a list of str, but was hoping for a simple all in one task that generates the params for some of my more complex downstream tasks.Michael Law
06/03/2021, 7:32 AMStéphan Taljaard
06/03/2021, 8:24 AMRanu Goldan
06/03/2021, 1:22 PMagent:k8s-prod
to 100
But I’ve seen the maximum of concurrent run is only 22-24, the other runs is scheduled
I have 1 kubernetes agent deployment on my k8s cluster
How to make the flow run concurrency more than 24? Anything I miss?
Thanks in advanceJulio Venegas
06/03/2021, 1:45 PMZach Schumacher
06/03/2021, 3:40 PMbral
06/03/2021, 5:46 PMCharles Liu
06/03/2021, 6:53 PMJason Prado
06/03/2021, 7:33 PMPostgresFetch
.
db_host = PrefectSecret('db_host')
fetch = PostgresFetch(
host=db_host,
...,
)
I can’t do this because UserWarning: A Task was passed as an argument to PostgresFetch, you likely want to first initialize PostgresFetch with any static (non-Task)...
. Does this mean I need to either put the DB host in my code or get it from another method like an environment variable? Why is host
part of the static args to PostgresFetch
if I’m likely to extract it from a Secret?Pedro Machado
06/03/2021, 7:49 PM--show-flow-logs
. It looks like the flow is not communicating with Prefect Cloud.
The flow is running an R script. I am using the rocker/r-ver:4.1.0
image as a base and installing python 3.7 via apt-get
. I suspect the issue is related to this. Maybe I am not installing python correctly. See my docker file in the thread. Thanks!Hugo Kitano
06/03/2021, 10:17 PMECSRun
functionality could work, but it is designed for running flows, and this would necessitate flows in flows which could be a little tricky. What would be the best way to reconfigure a Fargate task as a Prefect task?DK
06/04/2021, 1:49 AMWai Kiat Tan
06/04/2021, 6:30 AMash
06/04/2021, 9:04 AMPrabin Mehta
06/04/2021, 9:05 AMrunning flows through rest API.
I want to run a flow passing config with the request call. Can anyone help me on this?Dali
06/04/2021, 10:37 AMMarwan Sarieddine
06/04/2021, 2:12 PMHugo Kitano
06/04/2021, 5:18 PMFina Silva-Santisteban
06/04/2021, 5:34 PMHamza Ahmed
06/04/2021, 5:45 PMimagePullSecrets
value in the prefecthq/prefect-server
helm chart?
Tried multiple ways, but none seem to work (examples in comments)Julio Venegas
06/04/2021, 11:23 PMMichael Law
06/04/2021, 11:58 PMOussama Louati
06/05/2021, 9:49 AMColin
06/05/2021, 3:45 PMCA Lee
06/06/2021, 11:03 AM0.14.21
I am using VS Code, dev container, and running pip install prefect[aws]
in the build step . After the container is done, I am unable to do `prefect auth login -t {token}`as it leads to the error below:
vscode ➜ /workspaces/dbt-bigquery (main ✗) $ prefect auth login -t $PREFECT_AUTH_TOKEN
Traceback (most recent call last):
File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vscode/.prefect/client/https-api.prefect.io-graphql'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/cli/auth.py", line 83, in login
success_login = client.login_to_tenant(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 654, in login_to_tenant
self._save_local_settings(settings)
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 524, in _save_local_settings
self._local_settings_path.parent.mkdir(exist_ok=True, parents=True)
File "/usr/local/lib/python3.8/pathlib.py", line 1291, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/home/vscode/.prefect/client'
Jacob Blanco
06/07/2021, 6:29 AMThomas Hoeck
06/07/2021, 7:34 AMSatheesh K
06/07/2021, 10:14 AMrun_config
while building the flow, but it seems not working.