easimon / maximize-build-space Goto Github PK
View Code? Open in Web Editor NEWGithub action to maximize the available disk space on Github runners
License: MIT License
Github action to maximize the available disk space on Github runners
License: MIT License
Thanks for this useful action!
I'm currently getting the following error:
Unmounting and removing swap file.
swapoff: Not superuser.
Error: Process completed with exit code 1.
I find this confusing as I thought all actions were run as root
. Any ideas what's happening here? Here's the repository.
Full actions log is below.
Thanks for any help!
Run easimon/maximize-build-space@master
with:
remove-dotnet: true
remove-android: true
remove-haskell: true
root-reserve-mb: 1024
temp-reserve-mb: 100
swap-size-mb: 4096
overprovision-lvm: false
pv-loop-path: /pv.img
tmp-pv-loop-path: /mnt/tmp-pv.img
env:
GITHUB_PAT: ***
Run echo "Memory and swap:"
Memory and swap:
total used free shared buff/cache available
Mem: 7116968 570596 347668 32492 6198704 6203892
Swap: 4194300 2316 4191984
NAME TYPE SIZE USED PRIO
/mnt/swapfile file 4G 2.3M -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
overlay 84G 56G 29G 67% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/sda1 84G 56G 29G 67% /__w
tmpfs 696M 780K 695M 1% /run/docker.sock
tmpfs 3.4G 0 3.4G 0% /proc/acpi
tmpfs 3.4G 0 3.4G 0% /proc/scsi
tmpfs 3.4G 0 3.4G 0% /sys/firmware
Run set -euo pipefail
Arguments:
Root reserve: 1024 MiB
Temp reserve: 100 MiB
Swap space: 96 MiB
Overprovision LVM: false
Mount path: /__w/emma_model/emma_model
Root PV loop path: /pv.img
Temp PV loop path: /mnt/tmp-pv.img
Removing: dotnet android haskell
Removing unwanted software...
... done
Unmounting and removing swap file.
swapoff: Not superuser.
Error: Process completed with exit code 1.
We would like to use this action for our own build pipelines. But the project is missing a license that allows us to do so.
Can you please add a license to your project? I would suggest this license: https://choosealicense.com/licenses/mit/
Thank You!
Nico Dreher
GitHub Actions recently started upgrading Linux virtual environment from ubuntu-18.04 to ubuntu-20.04 (actions/runner-images#1816) that seemingly breaks maximize-build-space.
Run ***/maximize-build-space
with:
remove-dotnet: true
remove-android: true
root-reserve-mb: 1024
swap-size-mb: 4096
pv-loop-path: /pv.img
remove-haskell: false
Memory and swap:
total used free shared buff/cache available
Mem: 7121276 529448 4858396 6580 1733432 6279844
Swap: 4194300 0 4194300
NAME TYPE SIZE USED PRIO
/mnt/swapfile file 4G 0B -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
/dev/root 84G 60G 24G 72% /
devtmpfs 3.4G 0 3.4G 0% /dev
tmpfs 3.4G 12K 3.4G 1% /dev/shm
tmpfs 696M 1.1M 695M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/sda15 105M 9.2M 96M 9% /boot/efi
/dev/loop0 136M 136M 0 100% /snap/chromium/1466
/dev/loop2 56M 56M 0 100% /snap/core18/1944
/dev/loop1 138M 138M 0 100% /snap/chromium/1479
/dev/loop3 56M 56M 0 100% /snap/core18/1988
/dev/loop4 163M 163M 0 100% /snap/gnome-3-28-1804/145
/dev/loop5 65M 65M 0 100% /snap/gtk-common-themes/1514
/dev/loop6 70M 70M 0 100% /snap/lxd/19188
/dev/loop7 32M 32M 0 100% /snap/snapd/10707
/dev/loop8 32M 32M 0 100% /snap/snapd/11036
/dev/loop9 70M 70M 0 100% /snap/lxd/19032
/dev/sdb1 14G 4.1G 9.0G 32% /mnt
Arguments:
Root reserve: 1024 MiB
Swap space: 4096 MiB
Mount path: /home/runner/work/Cropbox.jl/Cropbox.jl
PV loop path: /pv.img
Removing: dotnet android
Removing unwanted software...
... done
Unmounting swap and temp disk.
Creating LVM Volume.
Physical volume "/dev/loop10" successfully created.
Can't open /dev/sdb1 exclusively. Mounted filesystem?
Can't open /dev/sdb1 exclusively. Mounted filesystem?
Error: Process completed with exit code 5.
This is a very useful action, and saved us a ton of headache!
From the README:
At the time of writing, public Github-hosted runners are using Azure DS2_v2 virtual machines, featuring a 84GB OS disk on / and a 14GB temp disk mounted on /mnt.
Today, when checking disk space before and after, I noticed the following instead:
# before
66G 74G /mnt /dev/sdb1
21G 73G / /dev/root
99M 105M /boot/efi /dev/sda15
# after
87G 87G /home/runner/work/repo/repo /dev/mapper/buildvg-buildlv
512M 73G / /dev/root
100M 74G /mnt /dev/sdb1
99M 105M /boot/efi /dev/sda15
87GB of free space, wow! It looks like whichever Azure VM they're using provides a much larger tempdisk now, which makes this action even more useful than it was before.
Since the Runner VMs are populated by a script, the knowledge of the contents of its configuration can help with identifying further target to remove.
For example:
https://github.com/actions/runner-images/blob/main/images/linux/toolsets/toolset-2204.json
How about either generating the options of this option directly from the JSON file or create some sort of rollback mechanism, than can take a similar script and just remove the software missing from the modified json file?
Updates last night seem to have broken this action for me. https://github.com/zenml-io/zenml/actions/runs/7016996228/job/19090533966 you can see the error message. Was just running it with build-mount-path: /var/lib/docker/
set. It was working without issue until last night / this morning.
When I rerun the action without setting that input to var/lib/docker
the action itself works again / completes, but then I run out of space on integration tests etc that I'm trying to run on that runner.
So not quite sure about the diagnosis / root cause, but pretty sure it relates to the changes made overnight.
For two weeks or so the root partition and the temp partition has been swapped on around 20% of the runners.
Which results in temp getting bigger and root staying small:
/dev/root 73G 73G 100M 100% /
Hey,
this is the only action that made our build work, so thank you.
Would it be possible to offer a less aggressive version or configurable version ideally which only deletes some usually never used stuff like the android sdk?
Some compilations need a lot of rams and sometimes, the runner will timeout and it isn't because of lack of ram as you can assign gigs of swap ram, it timeout because the swap ram is taking forever, I believe due to everybody fighting over IO on these drive, it is public runner afterall.
I did a fork and replaced the swap ram code with zramctl. My compilation which usually timeout and if it did not timeout, it usually compile in 45 minutes. With zram, it didn't timeout and compiled in 30 minutes, huge improvement.
With that said, I suggest (an elegant) implementation of zram as primary swap while using swap ram as backup swap. On a side note, the zram device takes a second to create on these public runner.
As composite-based actions now has support for conditionals, we can now run a separate script that runs on Windows and macOS to maximize available disk space.
The action is a great tool, bit it has a minor drawback.
The freed space is mounted the following way (output of the action itself, name of the repository changed)
/dev/mapper/buildvg-buildlv 39G 24K 39G 1% /home/runner/work/repo/repo
But I sometimes get the following message
System.IO.IOException: No space left on device : /home/runner/runners/2.308.0/_diag/Worker_20230912-055007-utc.log
[truncated]
I know, that I can change the free space of the root partition, but this doesn't really scale in terms in reusing the configuration in different projects.
Why not just changing the mount point of /dev/mapper/buildvg-buildlv
to /home/runner/
?
Before I knew this action I was using btrfs with compress to workaround my own space issue. I see this action is still using ext4, so this could be another way to accommodate large projects.
Git LFS initialized.
Fetching the repository
/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --filter=blob:none --depth=1 origin +refs/heads/maximize_space*:refs/remotes/origin/maximize_space* +refs/tags/maximize_space*:refs/tags/maximize_space*
The process '/usr/bin/git' failed with exit code 1
Waiting 20 seconds before trying again
/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --filter=blob:none --depth=1 origin +refs/heads/maximize_space*:refs/remotes/origin/maximize_space* +refs/tags/maximize_space*:refs/tags/maximize_space*
The process '/usr/bin/git' failed with exit code 1
Waiting 20 seconds before trying again
/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --filter=blob:none --depth=1 origin +refs/heads/maximize_space*:refs/remotes/origin/maximize_space* +refs/tags/maximize_space*:refs/tags/maximize_space*
Error: The process '/usr/bin/git' failed with exit code 1
Example config:
name: Build sample apk
on:
workflow_dispatch:
schedule:
- cron: '0 7 * * *' # run at 7 AM UTC
jobs:
build:
name: Build sample apk from develop branch
runs-on: ubuntu-latest
steps:
- name: Maximize space on the runner
uses: easimon/maximize-build-space@v7
with:
root-reserve-mb: 512
swap-size-mb: 1024
remove-haskell: 'true'
remove-docker-images: 'true'
# Checkout (with LFS)
- name: Checkout Repo
uses: actions/checkout@v3
with:
ref: maximize_space
lfs: true
sparse-checkout: |
Assets
Packages
ProjectSettings
Since the initial release I considered this action an ugly hack, to be used as a last resort only, because it depended on an undocumented "feature" of Github runners: an unused disk attached to the VM.
Over the last months, Github seems to regulary change the runner disk layout. The pattern is unclear for me. There seem to be differences between "free" runners and those used in Github Enterprise, but this does not seem to be everything.
Some runners now have both larger /
and a larger /mnt
, resulting in an enormous amount of total space, others have only 15GB free on / and seem to completely miss the temporary disk mounted on /mnt
, so there's nothing to add space from.
Finding all these bits and pieces to make this work back when I created the action was quite fiddly -- but at least all "default" runners looked the same, so it was constant for everyone. Now, given this very mixed experience, I doubt that it makes sense to keep the action alive "as is" -- if at all.
Input on this is welcome, until then, this issue should serve as an additional warning to those considering the use of this action.
I was trying out this action today on ubuntu-latest
and kept getting the following error:
Run easimon/maximize-build-space@master
Run echo "Memory and swap:"
Memory and swap:
/__w/_temp/c9089276-45ab-4176-ac40-c7afff117bd0.sh: line 2: free: command not found
Error: Process completed with exit code 127.
I wonder if we need to install packages separately? Or is it assumed that the commands required by the action comes pre-installed in the github runners?
Started getting this error on my actions. Assumed the repo had grown too big, but even re-running previously successful actions causes this error. Log follows.
Run easimon/maximize-build-space@master
Run echo "Memory and swap:"
Memory and swap:
total used free shared buff/cache available
Mem: 7110608 490944 5421356 22792 1198308 6284588
Swap: 4194300 0 4194300
NAME TYPE SIZE USED PRIO
/mnt/swapfile file 4G 0B -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
/dev/root 84G 58G 26G 69% /
tmpfs 3.4G 172K 3.4G 1% /dev/shm
tmpfs 1.4G 1.1M 1.4G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
/dev/sdb1 14G 4.1G 9.0G 31% /mnt
tmpfs 695M 12K 695M 1% /run/user/1001
Run set -euo pipefail
Arguments:
Root reserve: 30720 MiB
Temp reserve: 100 MiB
Swap space: 1024 MiB
Overprovision LVM: false
Mount path: /home/runner/work/map-repo/map-repo
Root PV loop path: /pv.img
Temp PV loop path: /mnt/tmp-pv.img
Removing: dotnet
Removing unwanted software...
... done
Unmounting and removing swap file.
Creating LVM Volume.
Creating LVM PV on root fs.
fallocate: invalid length value specified
Error: Process completed with exit code 1.
Are the LVM operations required? Would it be possible to just remove the software and skip the LVM step? I only ask because I'm not sure what maximum value I should provide for the root-reserve-mb
option. Thanks.
After a recent GitHub runner change, which changed the file structure of the runners to remove /mnt
, the root-reserve-mb flag no longer changes how much storage remains in the root directory.
Old filesystem:
Cleaner results on the old filesystem:
New filesystem:
Cleaner results on the new filesystem:
Arguments:
Note that the arguments we used with the old file system is identical to this except root-reserve-mb
was set to 30000.
is this still working?
runs-on: ubuntu-latest
0s
Run df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 84G 53G 31G 63% /
devtmpfs 3.4G 0 3.4G 0% /dev
tmpfs 3.4G 4.0K 3.4G 1% /dev/shm
tmpfs 695M 1.1M 694M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/1623
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop1 48M 48M 0 100% /snap/snapd/16778
/dev/sda15 105M 5.2M 100M 5% /boot/efi
/dev/sdb1 14G 4.1G 9.0G 31% /mnt
tmpfs 695M 0 695M 0% /run/user/1001
8s
Run easimon/maximize-build-space@master
with:
root-reserve-mb: 512
swap-size-mb: 1024
remove-dotnet: true
remove-android: true
remove-haskell: true
temp-reserve-mb: 100
overprovision-lvm: false
pv-loop-path: /pv.img
tmp-pv-loop-path: /mnt/tmp-pv.img
env:
CARGO_TERM_COLOR: always
Run echo "Memory and swap:"
echo "Memory and swap:"
free
echo
swapon --show
echo
echo "Available storage:"
df -h
echo
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
env:
CARGO_TERM_COLOR: always
Memory and swap:
total used free shared buff/cache available
Mem: 7110660 550796 5335396 12680 1224468 6247452
Swap: 4194300 0 4194300
NAME TYPE SIZE USED PRIO
/mnt/swapfile file 4G 0B -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
/dev/root 84G 53G 31G 63% /
devtmpfs 3.4G 0 3.4G 0% /dev
tmpfs 3.4G 4.0K 3.4G 1% /dev/shm
tmpfs 695M 1.1M 694M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/1623
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop1 48M 48M 0 100% /snap/snapd/16778
/dev/sda15 105M 5.2M 100M 5% /boot/efi
/dev/sdb1 14G 4.1G 9.0G 31% /mnt
tmpfs 695M 0 695M 0% /run/user/1001
Run set -euo pipefail
Arguments:
Root reserve: 512 MiB
Temp reserve: 100 MiB
Swap space: 1024 MiB
Overprovision LVM: false
Mount path: /home/runner/work/XXXXX/XXXXX
Root PV loop path: /pv.img
Temp PV loop path: /mnt/tmp-pv.img
Removing: dotnet android haskell
Removing unwanted software...
... done
Unmounting and removing swap file.
Creating LVM Volume.
Creating LVM PV on root fs.
Physical volume "/dev/loop3" successfully created.
Creating LVM PV on temp fs.
Physical volume "/dev/loop4" successfully created.
Volume group "buildvg" successfully created
Recreating swap
Logical volume "swap" created.
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=3a88015a-c8d7-412d-b453-14790523aa4d
Creating build volume
Logical volume "buildlv" created.
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 14713856 4k blocks and 3679200 inodes
Filesystem UUID: f54aed80-8c13-4e70-993f-967be6f29c96
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Allocating group tables: 0/450������� �������done
Writing inode tables: 0/450������� �������done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: 0/450������� �������done
Run echo "Memory and swap:"
Memory and swap:
total used free shared buff/cache available
Mem: 7110660 589880 5117596 12688 1403184 6208616
Swap: 1048572 0 1048572
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 1024M 0B -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
/dev/root 84G 83G 512M 100% /
devtmpfs 3.4G 0 3.4G 0% /dev
tmpfs 3.4G 4.0K 3.4G 1% /dev/shm
tmpfs 695M 1.1M 694M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/1623
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop1 48M 48M 0 100% /snap/snapd/16778
/dev/sda15 105M 5.2M 100M 5% /boot/efi
/dev/sdb1 14G 13G 100M 100% /mnt
tmpfs 695M 0 695M 0% /run/user/1001
/dev/mapper/buildvg-buildlv 55G 24K 55G 1% /home/runner/work/XXXXX/XXXXX
The first three steps will work for docker build, can you integrate it into the action?
jobs:
build_docker:
runs-on: ubuntu-latest
steps:
- name: Backup docker files
run: |
echo "backup moby/buildkit image"
sudo docker image save -o ${GITHUB_WORKSPACE}/images.tar moby/buildkit
echo "Back up /var/lib/docker folder structure and other files"
sudo rsync -aPq /var/lib/docker/ ${GITHUB_WORKSPACE}/docker
- name: Maximize build space
uses: easimon/maximize-build-space@master
with:
overprovision-lvm: 'true'
remove-dotnet: 'true'
# instead of using default value to mount to build path, /var/lib/docker/ is really the place we need more spaces.
build-mount-path: '/var/lib/docker/'
- name: Restore docker files
run: |
sudo rsync -aPq ${GITHUB_WORKSPACE}/docker/ /var/lib/docker
sudo rm -rf ${GITHUB_WORKSPACE}/docker
sudo ls ${GITHUB_WORKSPACE} -l
sudo docker image load -i ${GITHUB_WORKSPACE}/images.tar
sudo rm ${GITHUB_WORKSPACE}/images.tar
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: list images
run: docker images
- name: check storage
run: |
sudo df -h
Is it somehow possible to revert / change the space distribution after a specific point. I tried to rerun your action (with root-reserve-mb
and temp-reserve-mb
set), but unfortunately it doesn't work and I get the error fallocate: invalid length value specified
. I guess the action isn't made for something like this yet?
https://github.com/os-builds/kali/runs/7927017365
I'm trying to build a iso, which requires a lot of space. The github actions workflow works until I try to upload the iso. I guess that the upload-artifact action requires space on root oder temp fs?
A few days ago, it appears that Github Runners had their storage disk size decrease from 84GB to 73GB, since then this action has been failing with the error code fallocate: invalid length value specified
. Our actions were configured with the following parameters:
I'd like to know if others are seeing the same problem.
Logs at the point of failure:
...
untagged: ubuntu:20.04
untagged: ubuntu@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da85ff50c617732b8
deleted: sha256:f78909c2b360d866b3220655c0b079838258b8891a12ac25fc670f0cbb54229f
deleted: sha256:3a03f09d212915b240e9d216069aba5652ed4765c7e4b098c65e71860d47b8e1
Total reclaimed space: 3.122GB
... done
Unmounting and removing swap file.
Creating LVM Volume.
Creating LVM PV on root fs.
fallocate: invalid length value specified
Error: Process completed with exit code 1.
Hi,
Just wondering if any sort of advice has been sought regarding whether or not this breaches the Github T&Cs?
I could do with using this action but I'm concerned it may end up getting my account limited/banned.
Cheers,
James
I noticed our github actions started failing around 2 days ago and it keeps failing for the same reason. Looking into it, I noticed it's because the fallocate
command in this github action is failing claiming invalid length value even though, we use the default swap size and haven't made any changes to our github actions. See the log below:
My guess is Github changing its ubuntu-latest
version from 20.04 to 22.04 may have something to do with it.
Here is the link to our failed github action job.
Run easimon/maximize-build-space@v8
Run echo "Memory and swap:"
Memory and swap:
total used free shared buff/cache available
Mem: 16365024 607300 14552084 22820 1205640 15380000
Swap: 4194300 0 4194300
NAME TYPE SIZE USED PRIO
/mnt/swapfile file 4G 0B -2
Available storage:
Filesystem Size Used Avail Use% Mounted on
/dev/root 73G 54G 20G 74% /
tmpfs 7.9G 172K 7.9G 1% /dev/shm
tmpfs 3.2G 1.1M 3.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdb15 105M 6.1M 99M 6% /boot/efi
/dev/sda1 74G 4.1G 66G 6% /mnt
tmpfs 1.6G 12K 1.6G 1% /run/user/1001
Run set -euo pipefail
Arguments:
Root reserve: 40000 MiB
Temp reserve: 10000 MiB
Swap space: 4096 MiB
Overprovision LVM: false
Mount path: /home/runner/work/devbox/devbox
Root PV loop path: /pv.img
Temp PV loop path: /mnt/tmp-pv.img
Removing: dotnet android haskell codeql
Removing unwanted software...
... done
Unmounting and removing swap file.
Creating LVM Volume.
Creating LVM PV on root fs.
fallocate: invalid length value specified
Error: Process completed with exit code 1.
It would be useful to track how much time it took to free up the space with each of the options and update it in this table.
for me, it would help since I want to free up only around 8 GB and I want to pick up the quickest option/combination
Would you be up for having an input that disables everything except "removal of unnecessary software"?
I'd like to be able to just use (and benefit from) this:
https://github.com/easimon/maximize-build-space/blob/test-report/README.md
(as I have a setup that runs into fatal: write error: No space left on device
when using the full setup, which I didn't run into as early before)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.