https://prefect.io logo
Powered by
Title
k

Khyaati Jindal

01/06/2023, 11:27 AM
Can somebody tell me , which space is the this error referring to 
"[Errno 28] No space left on device:
got this error during the agent run.
✅ 1
j

Jelle Vegter

01/06/2023, 11:44 AM
Where are you running your agent?
k

Khyaati Jindal

01/06/2023, 12:00 PM
Ec2 instance
c

Christopher Boyd

01/06/2023, 1:46 PM
It would be the filesystem the agent is running on
is likely either disk full, or inode full
k

Khyaati Jindal

01/06/2023, 1:50 PM
I have checked the disk space , about 15 gb is free. What is a innode space ?
Could it be, that multiple flows are tasked for a same time?
c

Christopher Boyd

01/06/2023, 1:57 PM
What’s the output of 
df
and 
df -i
k

Khyaati Jindal

01/06/2023, 2:01 PM
(env) ubuntu@~/5c-email-triggers$ df -h Filesystem Size Used Avail Use% Mounted on /dev/root 29G 15G 15G 51% / tmpfs 3.9G 8.0K 3.9G 1% /dev/shm tmpfs 1.6G 1.1M 1.6G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/xvda15 105M 5.3M 100M 5% /boot/efi tmpfs 795M 4.0K 795M 1% /run/user/1000
c

Christopher Boyd

01/06/2023, 2:12 PM
that looks fine. What about 
df -i
? What does the flow do? Is it writing to disk, or anything? What is the rest of the context of the error
k

Khyaati Jindal

01/06/2023, 2:13 PM
The rest of the error is literary just list of all my internal files from the vm
No the flow does not write into any disk, just runs a Scripts that reads a 5 rows long table and sends it as a email.
And this issue has been there , across various vm instance, i have tried to re deploying it in another 2 more instances where it existed the agent with , no disk space error
c

Christopher Boyd

01/06/2023, 2:15 PM
What does 
df -i
show
k

Khyaati Jindal

01/06/2023, 2:17 PM
I don't have the output of df-i now, since they shutdown the vm , but when. I checked it right after the agent crashing, it was something arround 2gb out of 2gb free
c

Christopher Boyd

01/06/2023, 2:18 PM
df -i
should show something like this; inodes are not about space, but about inodes (locations) to write data to. Space is how much data can be written to fill up the filesystem, inodes are the number of files that can be written 
Filesystem     512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk3s1s1  965595304  17235488 497464152     4%  348618 2487320760    0%   /
many tiny files can fill up a filesystem even if it’s only 2% used on space
without seeing the full context of the error,, it’s unknown if that’s what this is even referencing
k

Khyaati Jindal

01/06/2023, 7:12 PM
This is a snippet from the error trace, will this help ?
@Christopher Boyd
c

Christopher Boyd

01/06/2023, 7:31 PM
This doesn’t really showcase much - are you able to share the logs from the prefect agent running the flow
before and up to receiving this error
In addition to the output of 
df -i
This error in isolation just points to potentially a full filesystem
k

Khyaati Jindal

01/06/2023, 7:37 PM
here is the df -i log from the system 
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/root      3870720 <tel:5311793339541|531179 3339541>   14% /
tmpfs          1017213      4 1017209    1% /dev/shm
tmpfs           819200    686  818514    1% /run
tmpfs          1017213      3 1017210    1% /run/lock
/dev/xvda15          0      0       0     - /boot/efi
tmpfs           203442    306  203136    1% /run/user/1000
no error was recorded for the prefect agent. I am afraid, this is all I have.
✅ 1
c

Christopher Boyd

01/06/2023, 7:39 PM
Both of those (df and df -i) look fine. Where is this error originating from then?
k

Khyaati Jindal

01/06/2023, 7:43 PM
The error is generated in the screen where agent is running
c

Christopher Boyd

01/06/2023, 7:47 PM
I don’t think we have enough information here to really diagnose much without a traceback, or what was occurring, or where you received this error. I would suggest enabling debug logs on the agent (
prefect config set PREFECT_LOGGING_LEVEL=DEBUG
), and if this occurs again, sharing the logs immediately prior to receiving the error, and including the error
k

Khyaati Jindal

01/06/2023, 7:50 PM
thanks, I did not know we can enable logs for agents like that. this would help capture lot of issues. will do that.
full log.txt
full log.txt
@Christopher Boyd My agent ran into same issue . Here is the full log
here is the vm df-h output 
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       30297152 26698916   3581852  89% /
tmpfs            4068852        8   4068844   1% /dev/shm
tmpfs            1627544     1096   1626448   1% /run
tmpfs               5120        0      5120   0% /run/lock
/dev/xvda15       106858     5329    101529   5% /boot/efi
tmpfs             813768        4    813764   1% /run/user/1000
(env) ubuntu@ip-172-31-44-58:~/5c-email-triggers$ df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/root      3870720 1084070 2786650   29% /
tmpfs          1017213       4 1017209    1% /dev/shm
tmpfs           819200     708  818492    1% /run
tmpfs          1017213       3 1017210    1% /run/lock
/dev/xvda15          0       0       0     - /boot/efi
tmpfs           203442     384  203058    1% /run/user/1000
(env) ubuntu@ip-172-31-44-58:~/5c-email-triggers$ df -H
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        32G   28G  3.7G  89% /
tmpfs           4.2G  8.2k  4.2G   1% /dev/shm
tmpfs           1.7G  1.2M  1.7G   1% /run
tmpfs           5.3M     0  5.3M   0% /run/lock
/dev/xvda15     110M  5.5M  104M   5% /boot/efi
tmpfs           834M  4.1k  834M   1% /run/user/1000
c

Christopher Boyd

01/11/2023, 1:08 PM
It looks like your /tmp folder is filling up when this runs
You either need a bigger filesystem to store it, or clear it out routinely
or mount /tmp to a separate filesystem
k

Khyaati Jindal

01/11/2023, 1:41 PM
The temp folder seems to be filling up with snapshots of my prefect folder which contains lot of env files and thus, maxing out the limit. Is there a way to avoid this or prevent this from happening ?
c

Christopher Boyd

01/11/2023, 1:50 PM
cron to clear out your tmp folder routinely
k

Khyaati Jindal

01/11/2023, 1:52 PM
I can do that, but my concern is , with in a few hours ( 5-6 ), there seems to be a 10gb temp files creation. Even with corn, this does not seem to be an effective way of deploying the code.
c

Christopher Boyd

01/11/2023, 2:15 PM
https://discourse.prefect.io/t/can-i-change-the-location-of-the-agent-tmp-tmp-prefect-to-home-tmp-prefect-when-executing-a-flow/1343/7 When you are using the agent in this way, the default storage location for flows and executions are to /tmp which is a pretty common practice for any application, not specifically Prefect. I believe these should be cleaning themselves up on flow completion, but if the flow crashes mid run because it’s full, it doesn’t
I would look to see what the 10gb of data creation is as well, that seems like a lot
k

Khyaati Jindal

01/11/2023, 7:42 PM
So I had to clear the data to redeploy the project since VM hit the memory and crashed. But this is what the files had : a file with filename like with tmpxxxxxxprefect , and cd inside the folders was a exact snapshot of my project directory.
@Christopher Boyd
c

Christopher Boyd

01/11/2023, 7:44 PM
do you expect to have your entire project directory? or just the flow
k

Khyaati Jindal

01/12/2023, 8:08 AM
the temp files had entire project dir
c

Christopher Boyd

01/12/2023, 1:55 PM
10gb is a lot, and this is not something we have seen reported, so I suspect this is a configuration issue. How did you setup the flow, deployment, and storage (storage block in the deployment, or path). • What are the contents of the project dir • what is the path of the project dir? • What is the filesystem path of the flow? • Are the contents of this project dir required for the flow to run • Where is your prefect installation located?
More specifically - do you have an example I can reproduce the problem with?
Here’s what happens when I create a new deployment with a flow. Note specifically, with no options specified for infrastructure, or for storage, the defaults are 
Process
and 
LocalFileSystem
which uses your current path. 
09:02:05.357 | INFO    | prefect.agent - Submitting flow run '47c2b06d-1625-4757-9791-8be1c51debe6'
09:02:05.902 | INFO    | prefect.infrastructure.process - Opening process 'hallowed-monkey'...
09:02:05.904 | DEBUG   | prefect.infrastructure.process - Process 'hallowed-monkey' running command: /Users/christopherboyd/virtualenvs/prefect2/bin/python -m prefect.engine in /var/folders/_s/1vczgxc97y5003d22b40kpd80000gn/T/tmp8ldlb5thprefect
09:02:06.009 | INFO    | prefect.agent - Completed submission of flow run '47c2b06d-1625-4757-9791-8be1c51debe6'
/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
09:02:06.831 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/3/workspaces/e7c934ce-f75b-4d2f-a166-e25d40e647d7/>
09:02:07.083 | DEBUG   | Flow run 'hallowed-monkey' - Loading flow for deployment 'community_test'...
09:02:07.185 | DEBUG   | Flow run 'hallowed-monkey' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
09:02:07.186 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
09:02:07.194 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts//workspaces/e7c934ce-f75b-4d2f-a166-e25d40e647d7/>
09:02:07.909 | DEBUG   | Flow run 'hallowed-monkey' - Executing flow 'healthcheck' for flow run 'hallowed-monkey'...
09:02:07.910 | DEBUG   | Flow run 'hallowed-monkey' - Beginning execution...
09:02:08.060 | INFO    | Flow run 'hallowed-monkey' - Created task run 'log_platform_info-afea9710-0' for task 'log_platform_info'
09:02:08.061 | INFO    | Flow run 'hallowed-monkey' - Executing 'log_platform_info-afea9710-0' immediately...
09:02:08.332 | DEBUG   | Task run 'log_platform_info-afea9710-0' - Beginning execution...
09:02:08.334 | INFO    | Task run 'log_platform_info-afea9710-0' - Host's network name = Christophers-MacBook-Pro.local
09:02:08.335 | INFO    | Task run 'log_platform_info-afea9710-0' - Python version = 3.9.14
09:02:08.365 | INFO    | Task run 'log_platform_info-afea9710-0' - Platform information (instance type) = macOS-13.0.1-arm64-arm-64bit
09:02:08.367 | INFO    | Task run 'log_platform_info-afea9710-0' - OS/Arch = darwin/arm64
09:02:08.367 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect Version = 2.7.1 🚀
09:02:08.368 | INFO    | Task run 'log_platform_info-afea9710-0' - Prefect API Version = 0.8.3
09:02:08.594 | INFO    | Task run 'log_platform_info-afea9710-0' - Finished in state Completed()
09:02:08.727 | DEBUG   | prefect.task_runner.concurrent - Shutting down task runner...
09:02:08.727 | INFO    | Flow run 'hallowed-monkey' - Finished in state Completed('All states completed.')
09:02:08.738 | DEBUG   | prefect.client - Connecting to API at <https://api.prefect.cloud/api//workspaces/e7c934ce-f75b-4d2f-a166-e25d40e647d7/>
09:02:09.042 | INFO    | prefect.infrastructure.process - Process 'hallowed-monkey' exited cleanly.
Note this line: 
/Users/christopherboyd/virtualenvs/prefect2/bin/python -m prefect.engine in /var/folders/_s/1vczgxc97y5003d22b40kpd80000gn/T/tmp8ldlb5thprefect
If your flow is in the same path and location as the rest of the folder, it’s copying the entirety
the best way to go about that, would be to create the flow in a sub-directory, and create the deployment from the sub-folder so if your project is 
projectABC
and the flow is 
projectABC/flow.py
it would be better to do 
projectABC/flow_directory/flow.py
k

Khyaati Jindal

01/12/2023, 2:41 PM
That makes sense.! My project is indeed in the same path as my whole project.. which includes heavy env packages files. I will right away , change the dir structure of the project.
3 Views
#prefect-communityJoin Slack
Powered by