Patrick Tan
06/06/2022, 8:01 PMcreate_flow_run.map(
flow_name=unmapped("LiveLots-ETL"),
project_name=unmapped(f"{prefect_project}"),
run_name=flowname_list,
run_config=unmapped(LocalRun(labels=[prefect_label])),
idempotency_key=idempotency_keys,
parameters=filelist,
)
Matt Alhonte
06/06/2022, 11:01 PMRandomizedSearchCV
)in dask-ml
? map
?Sandip Viradiya
06/06/2022, 11:43 PMDileep Damodaran
06/07/2022, 3:53 AMStephen Lloyd
06/07/2022, 5:14 AMDaskExecutor
might be an option, but I don’t quite understand all the setup I would need to do.Antti Tupamäki
06/07/2022, 8:57 AMSang Young Noh
06/07/2022, 11:05 AMClient error '401 Unauthorized' for url '<https://api-beta.prefect.io/api/accounts/5c1f7ebd-7031-4f69-bc00-6d21a9b17548/workspaces/9a321454-aa9e-4fb8-8f4c-17e327a77a01/blocks/get_default_storage_block>'
For reference, I was trying to run some deployments on prefect cloud, and when that didnt work, I logged out of prefect cloud and activated a local orion server instead.
The deployment command I am running is:
prefect orion database reset && prefect deployment create manager.py
Any help on this would be much appreciated!Ilya Sapunov
06/07/2022, 11:45 AMwith Flow("import") as flow:
for i in range(5):
res = import_task(
command="echo $CREDENTIALS",
env={
"CREDENTIALS": str(i)
},
)
But in that case all tasks will run in parallel, which will cause the request limits to be exceeded
Can I run those tasks sequentially ?flurven
06/07/2022, 11:59 AMMartin T
06/07/2022, 1:48 PMfrom prefect import Flow, Parameter, case, task
@task
def mytask(p2):
print(p2)
with Flow("case-demo") as flow:
p1 = Parameter("p1", default=False)
p2 = Parameter("p2", default="good")
with case(p1, True):
p2 = "bad"
mytask(p2)
print(flow.tasks)
$ prefect build -p flow.py
UserWarning: Tasks were created but not added to the flow: {<Parameter: p1>, <Parameter: p2>}.
$ prefect run -p flow.py
...bad...
Marcin Grzybowski
06/07/2022, 1:51 PM-v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
when I run orion dashboards from host:
prefect orion start
it works as intended - shows flows run from container.
But when I try to host dashboards from container I get empty dashboards (also mounting oriond.db, so both containers see the same database
-v /home/marcin/.prefect/orion.db:/root/.prefect/orion.db
added -p 4200:4200 to map port fdrom host to container, and exported PREFECT_ORION_API_HOST=0.0.0.0 so I can access dashboards in container from my host.
orion.db is properly mounted in both containers, because I see that it changes in them when i run a flow. Dashboards are accessible, but empty...Florian Guily
06/07/2022, 2:22 PMJohn O'Farrell
06/07/2022, 2:40 PMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty.
even though I know that the image URI that I'm using within containerDefinitions
has content.Dekel R
06/07/2022, 5:57 PM@task()
def task_a():
df = pd.Dataframe()
......
<http://logger.info|logger.info>(df.shape) # 19M and 13 cols
return df
@task()
def task_b(df):
<http://logger.info|logger.info>(type(df)) # Nonetype
with Flow('****',
storage=Docker(registry_url="us-central1-docker.pkg.dev****/",
dockerfile="./Dockerfile"), executor=LocalDaskExecutor(scheduler="processes")) as flow: #
df = task_a()
task_b(df)
Malthe Karbo
06/07/2022, 6:13 PMJason Motley
06/07/2022, 7:54 PMRon Budnar
06/07/2022, 9:31 PM@task
def get_dbt_kwargs(**kwargs):
return kwargs
with Flow(name="dbt_command_flow", run_config=docker_run, storage=STORAGE) as flow:
dbt_repo = Parameter(dbt_repo_url", ...)
dbt_env = get_dbt_kwargs(
DBT_ENV_SECRET__SQL_SERVER=PrefectSecret("DBT_ENV_SECRET__SQL_SERVER"),
DBT_ENV_SECRET__SQL_DATABASE=PrefectSecret("DBT_ENV_SECRET__SQL_DATABASE"),
DBT_ENV_SECRET__SQL_USER=PrefectSecret("DBT_ENV_SECRET__SQL_USER"),
DBT_ENV_SECRET__SQL_PASSWORD=PrefectSecret("DBT_ENV_SECRET__SQL_PASSWORD"),
)
... # a couple of tasks omitted for brevity
dbt_run = dbt(
env=dbt_env,
command=dbt_command,
task_args={"name": "Run DBT Shell command"},
upstream_tasks=[pull_task, dbt_env],
)
My question is - is there a better/preferred way to do the above? Basically, I'm trying to ensure that if credentials/secrets are changed in Prefect Cloud, then the next run picks them up correctly.Jacob Bedard
06/08/2022, 12:00 AMAndreas Nigg
06/08/2022, 6:59 AM◦ If you are using an agent running on Kubernetes, update the Prefect image version toThanks for the update by the way - up until now, working with prefect is just awesome - I enjoy every minute of it.in your Kubernetes manifest and re-apply the deployment.2.0b6
Tarek
06/08/2022, 7:44 AMprefect server create-tenant --name default --slug default
in the dockerfile of my app, it doesn't work.
why do other users have to install prefect locally outside the containers and run the command there.
no way to have the command somewhere in the docker images?Andreas
06/08/2022, 8:51 AMArthur Jacquemart
06/08/2022, 11:26 AMFrederick Thomas
06/08/2022, 1:28 PMDanilo Drobac
06/08/2022, 1:32 PMprefect cloud login --key <API Key I Just Created>
but I keep getting an error that says:
Unable to authenticate. Please ensure your credentials are correct.
Am I missing something?Nelson Griffiths
06/08/2022, 1:52 PMOllie Sellers
06/08/2022, 2:01 PMAlexandru Anghel
06/08/2022, 2:11 PMflow.register(project_name='my-project')
inside the python code. The while loop from the second task is preventing the register and it is unreachable. I want to register and run the flow using python file.py
More code inside the thread.
Is there a way to reach the register command of the flow inside the python code? Thank you!Ilhom Hayot o'g'li
06/08/2022, 2:20 PMTim Helfensdörfer
06/08/2022, 2:52 PMTim Enders
06/08/2022, 3:15 PMsqlite3.OperationalError: table flow already exists
Tim Enders
06/08/2022, 3:15 PMsqlite3.OperationalError: table flow already exists
Anna Geller
06/08/2022, 3:20 PMTim Enders
06/08/2022, 3:21 PMAnna Geller
06/08/2022, 3:21 PMTim Enders
06/08/2022, 3:21 PMAnna Geller
06/08/2022, 4:11 PMflurven
06/08/2022, 6:35 PMAnna Geller
06/08/2022, 9:09 PMTim Enders
06/08/2022, 9:10 PMAnna Geller
06/08/2022, 9:45 PM