. This is the task I am trying to run:
and it throws this exception: (see thread) All the times this task has been run in the past it passed just fine. Does anybody have an idea what might be the issue? Thanks very much in advance!
BigQueryTask( name="Get all statement IDs", query=""" SELECT * FROM temporary_tables.statements """, max_retries=3, retry_delay=timedelta(seconds=30), to_dataframe=True, )
(in bash, before invoking Python). Are their other ways of configuring this? (ie via constructor arguments at runtime)? ty!
But it is still possible to register flows as usual. I seem them in the UI, too. I switched back to v0.13.9 and the error is gone. I only changed the Prefect version -- no configuration and no flow code. I'd like to provide you with more information about this bug but the agent is ignoring
[2020-10-07 14:26:37,427] ERROR - agent1 | 400 Client Error: Bad Request for url: <http://localhost:4200/>
it seems. Is there any change in the enviroment for the agent? Why can I register flows but not start them?
and the other with label
), which also means that I'm using Docker Storage for the flows. Inside my flow I set the labels like this:
The thing is that I'd like the flow label to be "dynamic" in the sense that if I register that flow from a development environment, the label is automatically set to
flow.environment = LocalEnvironment(labels=['dev'])
, but if I want to register it for production, I'd like the flow label to be
. For now I register the flows from my own laptop, which automatically generates the Docker image and uploads it to my ECR for the Agents to use. Maybe I'm wrong with this, but I guess that if I set an environment variable in my local computer, and make the flow code something like
it will not work, because that variable will not exist in the container that runs the flow. Is this correct? Another alternative would be to be able to set flow labels when registering the flow using the CLI. Something like
flow.environment = LocalEnvironment(labels=[my_env_var])
, but according to the CLI help,
prefect register flow --file my_flow.py --name My-Flow --label dev
is not a valid parameter when registering flows. Do you have any recommendations for setting flow labels dynamically depending on the environment they where registered? Thanks!
Thomas La Piana