Jean-Baptiste Six
01/25/2022, 2:25 PMfrom dotenv import load_dotenv
load_dotenv()
@task()
def add(x, y=1):
return x + y
with Flow("Test GCSResult", result=GCSResult(bucket='thread-prefect-flows')) as GCSResult_flow:
first_result = add(1, y=2)
second_result = add(x=first_result, y=100)
Without success, however :
from dotenv import load_dotenv
load_dotenv()
def implicit():
from google.cloud import storage
# If you don't specify credentials when constructing the client, the
# client library will look for credentials in the environment.
storage_client = storage.Client()
# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)
implicit()
works well, then I'm not sure to understand : "_Make sure your Prefect installation can authenticate to Google's Cloud API_" (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
Then, I tried to use Local Secret (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
[context.secrets]
GOOGLE_APPLICATION_CREDENTIALS="auth/token.json"
GCP_CREDENTIALS="auth/token.json"
(don't know what to use between GCP_CREDENTIALS (prefect.utiliities.gcp) or GOOGLE_APPLICATION_CREDENTIALS (in the documentation of GCP), anyway I tried both and it didn't work neither.
What did I do wrong?
Please help 🙏Mukamisha jocelyne
01/25/2022, 2:36 PMSam Werbalowsky
01/25/2022, 4:57 PMMy Flow - today - today
rather than creating the date range
start_date = Parameter("start_date", default="today")
end_date = Parameter("end_date", default="today")
# this function handles parameter typing and converts today to YYYY-MM-DD
start_date_str = get_date_param_as_string.run(start_date)
end_date_str = get_date_param_as_string.run(end_date)
create_flow_run(
run_name=f"My Flow - {start_date_str.run()}-{end_date_str.run()}",
flow_name="My Flow Name",
project_name=prefect_project,
parameters={"start_date": start_date, "end_date": end_date, "file_type": "file_type"},
)
Jason Motley
01/25/2022, 4:58 PMSuresh R
01/25/2022, 5:33 PMJons Cyriac
01/25/2022, 5:43 PMFina Silva-Santisteban
01/25/2022, 5:55 PMAdi Gandra
01/25/2022, 6:26 PMprefect agent kubernetes install
what are the default mem and cpu limits? I have my tasks running on kubernetes spun up on EKS, and setup the agent using that command. However, the task is a download FTP file and it just stalls. No progress for hours. I’m wondering if this could be a possible mem and cpu limit, and if I should basically run the install command again - and will that overwrite my old manifest?Rob Landis
01/25/2022, 9:53 PMMathijs Miermans
01/25/2022, 10:33 PMStarting task run...
for tasks that are skipped because an upstream task fails? (Example code in thread.)Leon Kozlowski
01/25/2022, 11:39 PMjoshua mclellan
01/26/2022, 2:27 AMOvo Ojameruaye
01/26/2022, 5:54 AMTomek Florek
01/26/2022, 9:58 AMHenning Holgersen
01/26/2022, 11:52 AM'Connection aborted.', OSError(0, 'Error')
. I have added the RBAC stuff to the deployment, and opened outbound traffic to api.prefect.io on port 443, but I have not been able to get anything to run on the AKS agent. Any ideas?Romain
01/26/2022, 12:17 PMSidney Fox
01/26/2022, 2:48 PMValueError: The task result has no `location` so the result cannot be loaded. This often means that your task result has not been configured or has been configured incorrectly.
I’m not explicitly providing where or how results of the child flow should be stored, and I believe that is the root cause of the error thrown. Is that correct? If so, what is the preferred method for storing results? e.g. is PrefectResult preferred over S3Result or is it a matter of opinion?Sophie Courtemanche-Martel
01/26/2022, 4:09 PMstream = io.BytesIO(data.encode())
AttributeError: 'tuple' object has no attribute 'encode'
David Yang
01/26/2022, 4:12 PMKevin Kho
Eli Treuherz
01/26/2022, 5:20 PMTask.map()
it iterates through both of them together like zip()
. Is there a way to loop over every combination instead, like itertools.product()
?Heeje Cho
01/26/2022, 6:04 PMMatthew Seligson
01/26/2022, 7:03 PMJake
01/26/2022, 8:02 PMFailed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'cfmeta\'",)\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.6.15\', currently running with \'3.6.8\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.',)
I see that it is common and I read this thread. However, our flow needs to use specifically 3.6.15 because some of our dependencies require it and I do not see an agent that matches this version. I saw that script based storage was suggested, but we generate our flows dynamically so that is not an option. Any suggestions on how to resolve this? Thanks in advance!Saurabh Indoria
01/27/2022, 5:01 AMSergi de Pablos
01/27/2022, 6:43 AMmassumo
01/27/2022, 2:00 PMAndreas
01/27/2022, 3:10 PMshijas km
01/27/2022, 3:37 PMSuresh R
01/27/2022, 4:26 PM