DiskSpace

Overview

The Chemistry Wiki is a hosted VM on the ICT ESX Farm. Disks are added to the VM at the ESX level, and Linux immediately sees them. If more space is needed, creating a service request to the Service Desk, detailing storage requirements is the preferred method.

The file-system layout is basic. There are no separate data partitions. The single root partition contains both system and data. Disk resizing is slightly more cumbersome because of this. The main disk is actually built up of two disks bound together using standard LVM2 logical volumes. A resizing operation involves adding on an extra disk and joining it to the root LVM Volume group before extending the root Logical volume. ( a pvmove operation can be used to live shift data from a previous small addon disk to the new larger addon disk before removal of the smaller disk). When complete a resize2fs operation grows the file system. All this is possible live and while the wiki is in use.

Troubleshooting

While there are advantages and economies inherent to being a compact and easy to manage monolithic entity, there are no dedicated data partitions and there is no separation of resources for different services. This means that one service can bring down other services or the entire system. Often seemingly unrelated errors can signal deeper problems.

Unchecked disk usage is the main mechanism by which one service can bring the entire server down. Luckily, space requirements are modest (~14GB for the database, and ~22GB for the physical web content), and the wiki grows relatively slowly through a succession of sudden if modest growth spurts as material is added by staff and students. Usually this gives plenty of time to manage storage issues.

Logging can build up if debugging options are left on, or an error state is left undiscovered. Deleting logs normally has a minimal effect on disk usage, but can give immediate respite, and a chance to solve underlying problems.

More troublesome is an image conversion process performed by the wiki. This uses the ImageMagick package and the convert binary. For some reason it dumps temporary files into the /tmp directory and does not clean them up afterwards. The files it produces are prefixed with magick- and there are associated .png image files also. The rate of creation is very high, and can quickly overwhelm the system. A (root owned)cron job was set up to run a daily clean up script (/usr/local/bin/cleantmp.sh). Because of very intense periods of image creation recently, I have increased the frequency to twice an hour.

Addendum: If the tmp directory fills up, then a login error can manifest, reporting that Cookies are required for login. Please enable them and try again. This occurs because cookies can no longer be created in the full /tmp directory. The wiki then chooses to blame the user rather than itself.