Vipul
12/08/2021, 2:59 PMJason Motley
12/08/2021, 3:09 PMLeon Kozlowski
12/08/2021, 4:18 PMEnvironment:*
from the agent? I am having issues persisting a service account and role ARN that give flows privileges to hit aws resources (details in thread)Sam Werbalowsky
12/08/2021, 4:31 PMStringFormatter
but pass in a file that is opened…is something like this bad practice:
@task
def format_sql_file(file):
with open(file, 'r') as sql:
string_to_format = sql.read()
return StringFormatter().run(var1='my_value', var2='other_value', template = string_to_format)
Pedro Machado
12/08/2021, 6:37 PMBrian S
12/08/2021, 7:30 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named 'xxx'")
when submitting the flow to and ECS(Fargate). The module in question is a custom Python module that I include in the agent Docker image and it's installed via pip install. I feel like I'm missing something here but from what I've read, that should work. Does anyone have a tip on how I might get the flow execution to be aware of this custom module? Is the Agent the proper place to install the module?Tom Shaffner
12/08/2021, 10:23 PMJacob Blanco
12/09/2021, 2:06 AMAnh Nguyen
12/09/2021, 3:42 AMKlemen Strojan
12/09/2021, 12:02 PMKlemen Strojan
12/09/2021, 12:56 PM1.22.2
After running
prefect agent kubernetes install \
-k ${PREFECT_DE_PROD_API} \
--namespace prefect-latest \
--mem-request=16Gi \
--mem-limit=128Gi \
--cpu-request=4 \
--cpu-limit=32 \
--image-pull-secrets azurecr-secret \
--label k8s \
--label prod \
--label latest \
--rbac | kubectl apply --namespace=prefect-latest -f -
we get:
deployment.apps/prefect-agent created
<http://role.rbac.authorization.k8s.io/prefect-agent-rbac|role.rbac.authorization.k8s.io/prefect-agent-rbac> unchanged
error: unable to recognize "STDIN": no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>"
We should be using
<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
What should we do?Vincent Chéry
12/09/2021, 1:40 PMLocalAgent
running which gets its orders from a private prefect server instance and runs flow runs locally, and I basically do not want to persist any result on disk. Following the docs I have defined all my tasks with @task(checkpoint=False)
, which does the job for custom tasks but does not give me a solution for prefect tasks like prefect.tasks.control_flow.merge
.
I have tried setting task checkpointing to false
globally:
[tasks]
checkpointing = false
[tasks.defaults]
checkpointing = false
without success.
Any idea? Thanks a lot in advance!Lucas Hosoya
12/09/2021, 2:16 PMRyan Brennan
12/09/2021, 2:56 PMKeyError('The secret <SECRET NAME> was not found. Please ensure that it was set correctly in your tenant
This seems to happen the most when I’m using task mapping but it seems pretty random. 100 iterations of the mapped task will work, then the 101st fails with the error above, then the next 100 are successful. Has anyone ran into this issue or have any ideas on what might be happening?Jason Motley
12/09/2021, 4:07 PMJosh
12/09/2021, 7:25 PMNikhil Sthalekar
12/09/2021, 7:31 PMLeon Kozlowski
12/09/2021, 9:12 PMprefect.context
inside a flow run?Joseph Mathes
12/09/2021, 10:41 PMKevin Kho
John T
12/10/2021, 12:33 AMthread.lock
issue. Is there anything I could do about this?
I also tried ResourceManager to see if I could avoid it, but that also is not working.Pedro Machado
12/10/2021, 12:51 AMMaria
12/10/2021, 6:32 AMpostgres_execute = PostgresExecute( db_name="abc", user="peter", host="my_host", port=5432,)
I have multiple databases (dev, test, demo, +prod db per client), and I need to query/load data into the relevant one. I have a config file and I was hoping I could use Parameter to tell what config line to choose - but it seems like I cannot override those params... What is the recommended approach for such use case?Cristian Toma
12/10/2021, 6:53 AMRuben Sik
12/10/2021, 8:23 AMEmma Rizzi
12/10/2021, 10:43 AMraw
format and I got lost here, if anyone has experience with it I'd gladly take some advice!Suresh R
12/10/2021, 11:11 AMdepends_on_past
in airflow. We wanted next run of flow to start only when previous run is in Success state. Can anyone help?haf
12/10/2021, 11:35 AMe2631f0e-2987-405f-bf3b-3fbb46acf90c
Ryan Brennan
12/10/2021, 1:55 PMFuETL
12/10/2021, 1:57 PMcontext
value inside the flow run (So i can retrieve later via graphql query), the issue is (not all my flow will have this custom context value), for what i understand the context values is passed in run()
call, is not possible to add/modify and retrieve this values later? If so, there a alternative i can use to get custom values from a flow?