Matthew Seligson
02/23/2022, 2:21 PMАндрій Демиденко
02/23/2022, 4:07 PMAmar Eid
02/23/2022, 4:09 PMfrom prefect.tasks.secrets import PrefectSecret
@task
def my_task(secret_credentials):
return secret_credentials
with Flow('test') as flow:
credentials = PrefectSecret('NAME_OF_SECRET')
my_task(credentials)
According to this documentation https://docs.prefect.io/orchestration/concepts/secrets.html#setting-local-secrets to be able to set these local secrets I have to
1. Create a custom config file by running:
cd ~/.prefect/ && touch config.toml
2. Add the local secrets into this file
But the “.prefect” file doesnt exist on my machine. I have tried to uninstall and install prefect again and it doesnt generate. Any idea why this is happening and what I should do to create this file?
Thanks in advance for the help! //fyi @Henrietta Salonen @Maximilian LutzE Li
02/23/2022, 4:21 PMwith Flow(...) as flow:
data = extract_some_data()
flow_a = create_flow_run(…, parameters={'param-key':data})
flow_b = create_flow_run(…,parameters={'param-key':output_from_flow_a})
Kevin Kho
Martim Lobao
02/23/2022, 4:55 PMList
and Dict
tasks in my DAG? this is taken from a barely modified example on prefect’s blog (code in thread)Frederick Thomas
02/23/2022, 5:36 PMFrederick Thomas
02/23/2022, 5:36 PMchris evans
02/23/2022, 6:22 PMWilliam Grim
02/23/2022, 6:41 PMDockerStorage
, if we can, so that registering flows brings along dependencies. We also want to use KubernetesRun
so that prefect can schedule jobs on k8s. For the latter, however, is KubernetesRun
still the right thing to do? Someone outside this community mentioned to us about dask for scheduling instead of the "kubernetes scheduler" (his words; I'm still new to this and figuring it out).
Basically, which way is the "right way"? Adding dask isn't something we're opposed to doing, but we have a lot of stuff going on and are aiming for the lowest effort path, haha. I promise we're not lazy, just overworked.Trevor Sweeney
02/23/2022, 6:53 PMmssql_fetch = SQLServerFetch(db_name='db', user='user', host='host', fetch='all')
Kevin Kho
chia berry
02/23/2022, 7:46 PMEmailTask
instead of the transform_and_show task. I want to send the data result in the body of the email. However, I am getting the message AttributeError: 'FunctionTask' object has no attribute 'encode'
. If I make it a string, I get an empty email. I’d like to send the actual result and not the functiontask.Chris Reuter
02/23/2022, 7:55 PMDaniel Komisar
02/23/2022, 8:22 PMAqib Fayyaz
02/23/2022, 8:51 PMMax Lei
02/23/2022, 8:55 PMprefecthq/prefect
and pip install my source code in the image be enough?Dexter Antonio
02/23/2022, 9:08 PMBen Muller
02/23/2022, 10:21 PMdata_update = Parameter("data_update", required=True)
put_dateformatted_data_to_s3.map(
df=data,
bucket_name=unmapped(get_key_value("bucket")),
key_name=unmapped("key"),
suffix=unmapped(f"_{data_update}"),
)
This writes a file with a name of 19_<Parameter: data_update>.parquet
I want it to be the actual param provided in the flowiñigo
02/23/2022, 10:26 PMiñigo
02/23/2022, 10:27 PMJason Motley
02/23/2022, 11:07 PMRio McMahon
02/23/2022, 11:23 PMsrc/
directory) what is the best way to import it? I tried following similar logic to this: https://docs.prefect.io/orchestration/flow_config/storage.html#loading-additional-files-with-git-storage but adding to the import path:
import pathlib, sys
file_path = pathlib.Path(__file__).resolve().parent
sys.path.append(file_path)
But keep getting this error:
[23 February 2022 4:22pm]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'src'")
Is there a best practice for importing external python code into a flow?Farid
02/24/2022, 12:12 AMIf the output of one mapped task is used as input to another mapped task, any failed or skipped task will make the subsequent task fail/skip by default.
How can I change this default behaviour so that the downstream mapped task runs even if the upstream mapped one fails?Josh
02/24/2022, 2:38 AMScheduled
to Submitted
state in the UI. But there is no docker container that is ever spun up and the agent doesn’t do anything. Is there a checklist similar to this, but for submitted state? https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-scheduled-state/73Nitin Bansal
02/24/2022, 5:31 AMHedgar
02/24/2022, 7:05 AMjcozar
02/24/2022, 8:27 AMSuccess
and the flow run just runs the rest of the tasks (obviously with no data dependencies between those tasks).
Thank you in advance!Arturo
02/24/2022, 9:36 AMFaisal k k
02/24/2022, 10:23 AMModuleNotFoundError: No module named 'prefect.environments'
.. It is there in 0.15.13