Nelson
06/03/2020, 3:40 PMconfig.toml
to have different buckets per env:
source_data_bucket = "${environments.${environment}.source_data_bucket}"
transient_data_bucket = "${environments.${environment}.transient_data_bucket}"
[environments]
[environments.dev-nelson]
source_data_bucket = "<s3://REDACTED>"
transient_data_bucket = "<s3://REDACTED>"
[environments.prod]
source_data_bucket = "<s3://REDACTED>"
transient_data_bucket = "<s3://REDACTED>"
I can print the prefect.config.transient_data_bucket
inside a task, but when used as S3Result(bucket=prefect.config.transient_data_bucket)
it fails with Invalid bucket name ""
. How are others doing this? Note I’m providing this result as a task config
@task(
target="{date:%Y-%m-%d}/crunchbase-raw.cloudpickle",
result=S3Result(bucket=prefect.config.transient_data_bucket)
nicholas
06/03/2020, 3:55 PMNelson
06/03/2020, 3:59 PMnicholas
06/03/2020, 4:03 PMconfig.toml
(something modified post-installation) isn't available in your distributed environment. I think a better option would be to load your s3 bucket data as Parameter tasks, which can be passed as args to your downstream.Nelson
06/03/2020, 4:19 PM@task(result=S3Result(bucket=prefect.config.transient_data_bucket))
def magic():
...
So I cannot define one task decorator parameter with the output of another task right?
Generally the use case is: have different values per environment. A dask hostname, ports, buckets, etc. Here you showed how a bit: https://docs.prefect.io/core/concepts/configuration.html#configuration-interpolation
If it was via parameters, I’d have to define these values everywhere, when mostly it’s shared across a lot of flows hence config made sense? But also a task output is not available in contexts like this, or I missing anything? Thanks!nicholas
06/03/2020, 4:28 PMenvironment
set? I don't see it in that configNelson
06/03/2020, 4:36 PM-e PREFECT__ENVIRONMENT=nelson-dev
."${environments.${environment}.user}"
?Chris White
06/03/2020, 5:37 PMbucket=
becomes hardened as an attribute of your S3Result
when you register your Flow, so I think the pattern you’re going for here isn’t quite supportedNelson
06/03/2020, 5:42 PMChris White
06/03/2020, 5:46 PM