Edvard Kristiansen
05/05/2022, 8:41 AMFailed to load and execute flow run: RuntimeError("can't start new thread")
Restarting the VM resolved the issue but no flows were able to run at all until it was restarted, all with the same or similar errors. Is there any way to fix this without restarting the VM?Michael Hadorn
05/05/2022, 11:56 AMprefect agent docker start --show-flow-logs
. We can see them in the GUI, but not in the terminal output. Is this a known behavior?Lana Dann
05/05/2022, 5:59 PMMichael Smith
05/06/2022, 1:58 PMEdvard Kristiansen
05/06/2022, 2:21 PMrun = DbtShellTask()
send_notification = send_message(title = "Run Completed" , body = "The run has completed with status: " + run.status)
At the same time, it would also be great to return any error messages on a failed run. Any idea how to do that?madhav
05/06/2022, 7:11 PMFailed to load and execute flow run: ModuleNotFoundError("No module named '/root/'")
Our setup:
• a GitHub agent packages and deploys a docker container to ECR
• ECS pulls the ECR image and deploys a fargate instance
• (For our proof of concept) When the machine runs, we register a flow that lives in the image/fargate instance
• After that we run prefect local agent start
(both flow registration and starting the agent are run from a script that our ECS task definition runs on boot)
• We do a “quick run” and get the module not found error.
Other things to note:
• all of our flows and supporting modules are on this container.
• I’ve read some of the existing discussion surrounding this error and was not sure if the solution applied to our situation - given that we are running everything on the same Fargate instanceLuke Orland
05/06/2022, 9:42 PMIevgenii Martynenko
05/09/2022, 8:49 AM'str' object has no attribute 'setdefault'
after examining code we understood that issue comes from https://github.com/PrefectHQ/prefect/blob/9b5fb783d64299a180e5a54ed88ca6d4a1f8fb90/src/prefect/agent/kubernetes/agent.py#L490 .
It could be an access issue or missing file, or even a typo in yaml file. There is no error handling or checks to point to root cause properly. MIght be a good minor story to add checks and throw proper exception.Architha Rao
05/09/2022, 11:27 AMAlexis Lucido
05/09/2022, 4:56 PMek
05/09/2022, 8:28 PM~/.prefect/config.toml
looks like this:
backend = "server"
[server]
endpoint = "<http://prefect-server-apollo.prefect-server:4200>"
• in .github/workflows/build.yml
, execute python flow.py
which it should build as docker storage
when ever my github-runner exec python flow.py
, it won’t build the docker image for me here is the error message:
<http://localhost:2376/version>: Bad Request ("b'Client sent an HTTP request to an HTTPS server.'")
Any thoughts/suggestions would be much appreciate it!Lana Dann
05/10/2022, 6:26 PMJeff Kehler
05/11/2022, 3:09 AM--expose
and connecting to the UI using the server IP address on port 8080
. All that works fine, however I noticed that the url in Slack notifications are using <http://localhost:4200>
. How can I configure this part?Mia
05/11/2022, 9:10 AMiñigo
05/11/2022, 4:15 PMBinoy Shah
05/11/2022, 6:46 PMBinoy Shah
05/11/2022, 6:47 PMLuiz Felipe
05/12/2022, 6:53 PMsleep(60)
before the end of the task, the task logs appeared, but without it it does not work.Michael Smith
05/12/2022, 7:10 PMBinoy Shah
05/12/2022, 7:44 PMTom Baldwin
05/12/2022, 9:25 PMVariable "$input" got invalid value null at "input.tenant_id"; Expected non-nullable type UUID! not to be null.
I searched the community and saw some recent discussion about tenant_id
in this forum related to the 1.0 release, but none in this exact context I think. Do I need to install a newer version of Prefect, or click an older version of the secret link, to make this work?Cory Jez
05/13/2022, 3:29 PMjack
05/14/2022, 2:45 PMFlow run SUCCESS: all reference tasks succeeded
and then the next log line says No heartbeat detected from the remote task; marking the run as failed.
YD
05/16/2022, 11:35 PMflow.register(...)
, a docker agent ?
Can I have different docker agents running on a single server?Mia
05/17/2022, 2:14 PMKayvan Shah
05/17/2022, 4:56 PMKayvan Shah
05/18/2022, 6:45 AMRoey Brecher
05/18/2022, 6:59 AM.prefect
directory reached a size of 428G.Sebastian Beschke
05/18/2022, 6:03 PMrequests.exceptions.ConnectionError
, but does not exit. However it’s also does not seem to reestablish the connection once the server becomes available again. Is there some way to either
1. have the agent retry connecting some time after encountering a connection error, or
2. exit after a connection error so ECS can detect that and restart it?
The way it looks now I have to manually restart the agent in this situation, which I don’t think is ideal… Am I missing something? Thanks in advance!David Beck
05/19/2022, 3:39 PM_log_record_context_injector
function to modify the names of PREFECT_LOG_RECORD_ATTRIBUTES
for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an upstream_task.
When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.