Chris Keeley
04/11/2022, 1:09 PMwait_for_flow_run
method for creating flow of flows, but this seems designed to work with one tenantClaire Herdeman
04/11/2022, 1:39 PMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
Is a reliable way to avoid this or a best practice for launching multiple versions of a flow?Roger Webb
04/11/2022, 1:43 PMJelle Vegter
04/11/2022, 2:14 PMNico Neumann
04/11/2022, 2:39 PMChris Martinez
04/11/2022, 2:59 PM<http://prefect.io/flow_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx|prefect.io/flow_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>
<http://prefect.io/flow_run_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx|prefect.io/flow_run_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx> <http://prefect.io/identifier=xxxxxxxx|prefect.io/identifier=xxxxxxxx>
Stephen Lloyd
04/11/2022, 3:39 PM>>> from prefect import Client
>>> client = Client()
>>> client.set_secret(name="test", value="test")
>>> client.set_secret(name="AWS_ACCOUNT_ID-dev", value="123456789098")
It seems like I should be able to use
from prefect.client import Secret
aws_account_id = Secret('AWS_ACCOUNT_ID-' + RUN_ENV).get()
However, I receive the following error:
ValueError: Local Secret "AWS_ACCOUNT_ID-dev" was not found.
How can I retrieve a secret value outside of a task?Atsushi Saito
04/11/2022, 4:14 PMLocalRun
??
Can I use other virtual environments’ python path like virtual envs in pyenv or conda path?Matt Delacour
04/11/2022, 4:18 PMJosh
04/11/2022, 4:22 PMException raised while calling state handlers: KeyError('The secret SLACK_PREFECT_NOTIFICATIONS_WEBHOOK_URL was not found. Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>')
Anders Segerberg
04/11/2022, 4:27 PMcreate_flow_run(..., idempotency_key="{date}-"+MY_VAR)
function as expected? It's hard to tell, because I don't know yet how Prefect compiles the string templating as part of tasks. Would I need to generate this string as a task result, and then pass the result to the kwarg?Atsushi Saito
04/11/2022, 5:26 PMlate runs
icon on cloud UI.
However, cloud-UI shows neither success (green status) nor failure (red status)
Is this a issue about docker-login ? or are there other possible causes?Naimesh Chaudhari
04/11/2022, 5:28 PMSam Garvis
04/11/2022, 5:45 PMAnders Segerberg
04/11/2022, 6:03 PM13:53:26 INFO CloudFlowRunner
Flow run SUCCESS: all reference tasks succeeded
13:55:13 INFO
User marked flow run
as Failed
13:55:21 INFO
null restarted this flow run
13:55:22
INFO agent Submitted for execution: Container ID:
<id>
13:55:22 INFO
S3 Downloading flow from s3://<key>
13:55:23 INFO
S3 Flow successfully downloaded. ETag: <>, LastModified: <> VersionId: <>
13:55:23 INFO CloudFlowRunner
Beginning Flow run for <pipeline>
13:55:23 INFO CloudFlowRunner
Flow run SUCCESS: all reference tasks succeeded
I would expect the flow to re-run entirely. However, it doesn't look like it does (it just jumps straight into Success). Is this due to Prefect's default input caching?
To be specific, this flow is a flow-of-flows. I have two subflowsNaimesh Chaudhari
04/11/2022, 6:12 PMDavid Haynes
04/11/2022, 6:20 PMKarim Zaghw
04/11/2022, 6:27 PMNaimesh Chaudhari
04/11/2022, 6:33 PMVaikath Job
04/11/2022, 7:19 PMTony Yun
04/11/2022, 7:48 PMAnders Segerberg
04/11/2022, 8:57 PMVaikath Job
04/11/2022, 11:09 PMEddie Atkinson
04/12/2022, 2:35 AMECSRun
flow using the LocalDaskExecutor
with 30GB of RAM. For large jobs this flow OOMs and gets killed by the Prefect scheduler. My question is this: If I set up a Dask cluster to run these jobs would it gracefully handle memory issues?
That is to say if I had 30GB of RAM in the cluster and a job that required 50GB would Dask OOM or would it simply run slower? Do I need to modify my code to use Dask dataframes or is there some smarts here I’m not quite across?Atsushi Saito
04/12/2022, 5:19 AMlabels
be specified in both running environments (docker runner or local runner) and remote server-side (i.e. cloud UI or server UI) ??Alexander Belikov
04/12/2022, 8:59 AMRAISS Zineb
04/12/2022, 12:30 PMConstantino Schillebeeckx
04/12/2022, 1:55 PMDomenico Di Gangi
04/12/2022, 2:31 PMVasco Leitão
04/12/2022, 2:48 PMdefaults_from_attrs
helper decorator. What am I missing? 😅 (Code in the thread)Vasco Leitão
04/12/2022, 2:48 PMdefaults_from_attrs
helper decorator. What am I missing? 😅 (Code in the thread)from prefect.tasks.aws.parameter_store_manager import AWSParametersManager
...
@task(name="Fetch Invoices from FTP Server")
def fetch_invoice_files() -> list:
logger = prefect.context.get('logger')
try:
with paramiko.SSHClient() as s:
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
UPS_USERNAME = AWSParametersManager.run('ftp_user')
UPS_PW = AWSParametersManager.run('ftp_password')
s.connect(URL, 10022, username=UPS_USERNAME, password=UPS_PW)
...
return files
...
with Flow(
'parameter-test',
run_config=flow_run_config,
) as flow:
files = fetch_invoice_files()
...
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/Users/vascoleitao/Document/flows/ups_ftp_finance/flow.py", line 47, in fetch_invoice_files
UPS_USERNAME = AWSParametersManager.run('ftp_user')
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 455, in method
kwargs.setdefault(attr, getattr(self, attr))
AttributeError: 'str' object has no attribute 'parameter_name'
Kevin Kho
04/12/2022, 2:49 PMfetch_invoice_files
a task?UPS_USERNAME = AWSParametersManager().run('ftp_user')
Vasco Leitão
04/12/2022, 2:56 PMrun
method.
UPS_USERNAME = AWSParametersManager().run(parameter_name='ftp_user')
However, I'm getting unexpected values. I should get the clear value of the parameter, right?Kevin Kho
04/12/2022, 3:07 PMVasco Leitão
04/12/2022, 3:11 PMprint(UPS_USERNAME)
print(UPS_PW)
AQICAHjJcwEWLpsKGepQ9HpDWyYbuEDjdo5wDqcdcbs0xLhB4gGKDfIHfHZUA5CspbLvk+NKAAAAazBpBgkqhkiG9w0BBwagXDBaAgEAMFUGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMG79ax+1uE+LuAb7AAgEQgCiEqBR/t74O9x2qFLEJNAg
AQICAHjJcwEWLpsKGepQ9HpDWyYbuEDjdo5wDqcdcbs0xLhB4gFrFkanm+V4F1cko5UBGctcAAAAaTBnBgkqhkiG9w0BBwagWjBYAgEAMFMGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMTNnIXGejIWXNzUOYAgEQgCa
Kevin Kho
04/12/2022, 3:13 PMVasco Leitão
04/12/2022, 3:17 PMKevin Kho
04/12/2022, 3:36 PMAnna Geller
04/12/2022, 4:49 PM