Ubuntu Apt Cron Cache Has Broken Packages
One day I received an root mail from my machine, talking about some packages are broken in the cache. The investigation begins…
Problem Encountered
I received a root mail from my machine this morning, contents are shown below:
Subject: Cron <root@win> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
/etc/cron.daily/apt:
Cache has broken packages, exiting
Search for Solutions
First of all, collect all information which could be useful for later searching.
Unfortunately, the only information is that root mail. So I fed Google with the
error message, I found one post discussing about this problem. Someone said that
was because the root file system was low on free inodes, but not free space.
We can check this by df -i
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 327680 324403 3277 99% /
udev 253837 445 253392 1% /dev
tmpfs 256454 351 256103 1% /run
none 256454 2 256452 1% /run/lock
none 256454 1 256453 1% /run/shm
/dev/sdd1 327680 4013 323667 2% /lib/modules
/dev/md0 1310720 240845 1069875 19% /home
Ah-ha! This might be the problem! So our goal is turning to “how to reduce the inodes which are being used?”. The guy in the forum also mentioned that removing old linux-kernel-* and linux-headers-* will fix the problem. But how to remove them? Where are they?
Another article told us how to purge old and useless linux images and header
packages by using dpkg
# dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
After seeing this, I gave it a try but failed with the dependency warning. The warning said that I must install “linux-headers-3.2.0-72” because “linux-headers-3.2.0-72-generic” needs it. So I install the dependency according to the instruction it provided:
# apt-get -f install
Unfortunately, it failed again because there was no free disk space any more. I faced a dead lock between installation and uninstallation! What a joke…
My Solution, Ugly
Also, the running kernel version is not the newest, which means the system administrator has not reboot the machine since kernel upgrade.
$ uname -r
3.2.0-67-generic
I found many linux-headers-3.2.0-xx and linux-headers-3.2.0-xx-generic
are under /usr/src directory. So chances are that I may move those
directories to another place which is not under root directory. And I did it.
Now we can install the dependencies by sudo apt-get -f install linux-headers-3.2.0-72
After the dependencies were installed, we can remove all the previous kernel images, headers and modules, leaving only the current one intact. Using the one liner specified in last section will do the trick. (But it is important that the machine should reboot before purging the old kernel images. If you did not reboot, next time rebooting the machine will fail because it cannot find the kernel image.) After all, our precious disk space is returned!
To prove that our work does solve the original problem, we simply run the cron job manually:
$ cd /
# run-parts --report /etc/corn.daily
Waiting about 20 minutes, this time no more errors are shown. Hooray!