Edvard Kristiansen05/05/2022, 8:41 AM
Restarting the VM resolved the issue but no flows were able to run at all until it was restarted, all with the same or similar errors. Is there any way to fix this without restarting the VM?
Failed to load and execute flow run: RuntimeError("can't start new thread")
Michael Hadorn05/05/2022, 11:56 AM
. We can see them in the GUI, but not in the terminal output. Is this a known behavior?
prefect agent docker start --show-flow-logs
Lana Dann05/05/2022, 5:59 PM
Michael Smith05/06/2022, 1:58 PM
Edvard Kristiansen05/06/2022, 2:21 PM
At the same time, it would also be great to return any error messages on a failed run. Any idea how to do that?
run = DbtShellTask() send_notification = send_message(title = "Run Completed" , body = "The run has completed with status: " + run.status)
madhav05/06/2022, 7:11 PM
Our setup: • a GitHub agent packages and deploys a docker container to ECR • ECS pulls the ECR image and deploys a fargate instance • (For our proof of concept) When the machine runs, we register a flow that lives in the image/fargate instance • After that we run
Failed to load and execute flow run: ModuleNotFoundError("No module named '/root/'")
(both flow registration and starting the agent are run from a script that our ECS task definition runs on boot) • We do a “quick run” and get the module not found error. Other things to note: • all of our flows and supporting modules are on this container. • I’ve read some of the existing discussion surrounding this error and was not sure if the solution applied to our situation - given that we are running everything on the same Fargate instance
prefect local agent start
Luke Orland05/06/2022, 9:42 PM
Ievgenii Martynenko05/09/2022, 8:49 AM
after examining code we understood that issue comes from https://github.com/PrefectHQ/prefect/blob/9b5fb783d64299a180e5a54ed88ca6d4a1f8fb90/src/prefect/agent/kubernetes/agent.py#L490 . It could be an access issue or missing file, or even a typo in yaml file. There is no error handling or checks to point to root cause properly. MIght be a good minor story to add checks and throw proper exception.
'str' object has no attribute 'setdefault'
Architha Rao05/09/2022, 11:27 AM
Alexis Lucido05/09/2022, 4:56 PM
ek05/09/2022, 8:28 PM
looks like this:
backend = "server" [server] endpoint = "<http://prefect-server-apollo.prefect-server:4200>"
which it should build as docker storage when ever my github-runner exec
, it won’t build the docker image for me here is the error message:
Any thoughts/suggestions would be much appreciate it!
<http://localhost:2376/version>: Bad Request ("b'Client sent an HTTP request to an HTTPS server.'")
Lana Dann05/10/2022, 6:26 PM
Jeff Kehler05/11/2022, 3:09 AM
and connecting to the UI using the server IP address on port
. All that works fine, however I noticed that the url in Slack notifications are using
. How can I configure this part?
Mia05/11/2022, 9:10 AM
iñigo05/11/2022, 4:15 PM
Binoy Shah05/11/2022, 6:46 PM
Binoy Shah05/11/2022, 6:47 PM
Luiz Felipe05/12/2022, 6:53 PM
before the end of the task, the task logs appeared, but without it it does not work.
Michael Smith05/12/2022, 7:10 PM
Binoy Shah05/12/2022, 7:44 PM
Tom Baldwin05/12/2022, 9:25 PM
I searched the community and saw some recent discussion about
Variable "$input" got invalid value null at "input.tenant_id"; Expected non-nullable type UUID! not to be null.
in this forum related to the 1.0 release, but none in this exact context I think. Do I need to install a newer version of Prefect, or click an older version of the secret link, to make this work?
Cory Jez05/13/2022, 3:29 PM
jack05/14/2022, 2:45 PM
and then the next log line says
Flow run SUCCESS: all reference tasks succeeded
No heartbeat detected from the remote task; marking the run as failed.
YD05/16/2022, 11:35 PM
, a docker agent ? Can I have different docker agents running on a single server?
Mia05/17/2022, 2:14 PM
Kayvan Shah05/17/2022, 4:56 PM
Roey Brecher05/18/2022, 6:59 AM
directory reached a size of 428G.
Sebastian Beschke05/18/2022, 6:03 PM
, but does not exit. However it’s also does not seem to reestablish the connection once the server becomes available again. Is there some way to either 1. have the agent retry connecting some time after encountering a connection error, or 2. exit after a connection error so ECS can detect that and restart it? The way it looks now I have to manually restart the agent in this situation, which I don’t think is ideal… Am I missing something? Thanks in advance!
David Beck05/19/2022, 3:39 PM
function to modify the names of
for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an
When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.