Tag: zfs

Ubuntu, ZFS and running out of drive space

For userland, system operations really need to be invisible.

I have been using Ubuntu as my desktop OS for over 15 years now. I came to it by way of using it on a backup computer to do a task that my primary computer, using Windows XP, failed at if I tried to do anything else concurrently. I didn’t want to buy another computer or a license for XP and put it on an old computer with specs that were very marginal for XP. So I thought, try this free OS and see whether I can use the old computer to do the task that my new XP computer would only do if nothing else was done at the same time.

Ubuntu got the job done. I recorded my vinyl albums to sound files, broke the sound files into tracks like on the albums, and then burned the tracks onto CDs so I could carry my music around more conveniently and listen to it in more places. XP did this too, but the sound files were corrupt or the burn failed if I did something else like open my word processor, spreadsheet, or web browser while recording or burning were going on.

Now I had two pcs running and would switch between them as needed to keep the vinyl to CD process going. That was a little inconvenient because my workspace didn’t let me put the two pcs right next to each other. Switching wasn’t a case of moving my hands from one keyboard and mouse to another, or flipping a switch on a KVM. Delaying getting to the Ubuntu pc after the album side was over meant extra time spent trimming the audio file to delete the tail of the file. This led to me trying some things on the Ubuntu pc like opening the word processor or spreadsheet or browsing the web while the recording or burning were going on to see if it caused problems. And amazing, it didn’t! A lower spec pc with Ubuntu could do more of what I wanted without errors than my much better XP desktop.

That led me to using the Ubuntu pc while converting my albums to CD and that led me to Ubuntu for home use. Professional life continued and continues to be Windows, but at home Ubuntu. And Ubuntu is still preferred at home because it doesn’t mysteriously prevent me from doing things, inconveniently interrupt me, or insist on having information I don’t want to share like Windows does. That is until ZFS in Ubuntu 20.04 started preventing me from doing updates on my primary and backup pc because of lack of space.

I’ve run out of space on Windows and Ubuntu before. It just meant time to finally do some housekeeping and get rid of large chunks of files, like virtual machines, that I hadn’t used in a while. Do that and boom, back to work! Not so with ZFS. Do that and gloom, still can’t do updates.

There were different error messages on the two pcs, one said bpool space free below 20%, the other was rpool space free below 20%. Rpool and bpool, what are they? And why, when there’s nearly 20% free space, is updating prevented? And why after deleting or moving tens of gigabytes of files off the drive and purging old kernels, a Linux thing, are updates still prevented and rpool and bpool still report less than 20% free? Gigs of files were just moved off the drive and these rpool and bpool things don’t reflect that!

My first experience with Ubuntu after more than 15 years using it where keeping it up to date wasn’t just a case of using it and running updates every once in a while.

Windows has a feature called “Recovery Points” that I’ve used to get back to a working system when things have been broken to the point of making it hard or impossible to use the pc. Ubuntu hasn’t really had anything equivalent until the introduction of ZFS. And as I’ve learned, that’s way too simple minded an explanation and doesn’t give credit to the capabilities of ZFS that go way beyond Windows Recovery Points. True, and so be it.

I dug through many ZFS web pages and tried many things until finally getting more than 20% rpool or bpool free on each pc, a list of links is at the end of this post. Now the pcs are back to updating without complaint.

What I’ve learned is Ubuntu has a way to go to make ZFS user friendly. Things I’d suggest to Canonical for desktop Ubuntu:

  • Double the recommended minimum drive size and/or tell end users they should have 2x the drive space they think they need if they already think they need more than the minimum
  • Reduce the default number of snapshots to 10
  • Provide a UI for setting the number of snapshots
  • Provide a UI for selectively removing snapshots from bpool or rpool when free space goes below the dreaded 20%
  • After prompting for confirmation automatically remove the oldest snapshots to get back to 20% free when the condition occurs

Both my pcs are now above 20% free space on rpool and bpool and updating without complaint. It took a while and some learning to make that happen. It wasn’t the type of thing an average end user would ever want to face or even know about.

77% and 48% free – will probably bump rpool space issues first again on this pc
89% and 90% free – plenty of room before bpool is a problem again on this pc

ZFS focus on Ubuntu 20.04 LTS: ZSys general presentation · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys general principle on state management · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys commands for state management · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys state collection · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys for system administrators · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys partition layout · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys dataset layout · ~DidRocks

ZFS focus on Ubuntu 20.04 LTS: ZSys properties on ZFS datasets · ~DidRocks

apt – Out of space on boot zpool and cant run updates anymore – Ask Ubuntu
For this link see especially Hannu‘s answer on Nov 19, 2020 at 17:22

docs.oracle.com | Displaying and Accessing ZFS Snapshots

docs.oracle.com | Destroying a ZFS File System

docs.oracle.com | Creating and Destroying ZFS Snapshots

Got vsftpd?

The path from “need a few files” to providing any time you like self service.

I tend to have computer components and a few spare computers hanging around. Both because I haven’t got hit with Marie Kondo fever (I’m not really bad) and because I help my kids with equipment selection, sometimes purchase, and benefit from getting their leave behinds to experiment with.

In this case one son had upgraded so I got the old laptop. It needed some work to be useful, badly damaged digitizer. He also wanted files from the hard drive but didn’t have opportunity to get them before leaving me the pc.

I replaced the digitizer and swapped out the hard drive with a loose one I had around so I could use the pc. Put the original drive in an external USB3 enclosure I had, labeled it not to erase, and set it aside.

Then said son asked for four files from the old drive. No problem I thought. Plug the drive into the USB port of my laptop, read them off the drive and send. Nope.

This son is one I’ve gotten to use Linux on several systems. I’d set up Linux for him on this system and used the default partition method at the time, LVM. Couldn’t read the drive. My system, using the current default, ZFS, didn’t have the ability to mount the drive.

Here’s one of the reasons I find Linux to be easy to use, all I needed to do was install LVM on my system and reboot. Presto I could read the external drive. It now automatically mounts when plugged in to USB. And the ZFS install of my system wasn’t affected at all.

Now try to read his files for him. Nope. He had been traveling internationally so I’d set up an encrypted home directory for him. Fortunately I’d kept the encryption passphrase in my password safe and was able to mount the encrypted home directory. I still wasn’t getting files in the clear though. It seemed related to the fact the drive was no longer the boot drive. Went down that rabbit hole for a bit and seemed to be making progress. Finally though, to get him the files, I just asked him if he recalled his login password. He did.

Booted the old pc and selected the external HD to boot from, it went right to login screen, enter password, and I’m logged in to the old system. Another Linux advantage, take an original host drive, plug it into USB on another pc, select that drive as boot source, and Linux boots without complaint.

I sent him the files he wanted. Then I thought to send him a list of all files in his home directory. After all he might want others and just not recall their names. Sure enough, he wanted a few more after getting the list.

Now I’m thinking, if he wants more files, then more work for me. What if, instead, he can get the files on his own any time he likes? Could I set up an ftp site he could connect to and get files whenever he wished?

This is where vsftpd finally enters the picture. My plan was boot from the old hard drive using a spare pc, make an ftp site that used an encrypted connection so not even username/password are sent in the clear and provide him the connection information.

vsftpd is an easy set up. Run the installation and it accepts anonymous connections by default. Didn’t want anonymous though and wanted connection to go to his home directory. Read the man, linux.die.net is my favorite man source, search for others’ descriptions of how to set up a credentialed, encrypted connection, and keep hacking at it until it worked.

The thing that really stymied me was the obscure failure message when vsftpd was failing to start after some of the config changes I made. I couldn’t find a parameter to boost the detail of the logging and was left with only “status=2/INVALIDARGUMENT” to try and figure out what parameter was the problem. Fortunately I came across Why my vsftp service can’t start?. It offered the tip to run /usr/sbin/vsftpd manually from the command prompt and the specific issue might be revealed. I tried, the problematic option was revealed, changed the option and presto, working vsftpd using TLSV1 for connections!

For your interest, here’s my working vsftpd.conf

anonymous_enable=NO
local_enable=YES
chroot_local_user=YES
allow_writeable_chroot=YES
listen=NO
listen_ipv6=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
user_sub_token=$USER
local_root=/home/$USER
rsa_cert_file=/etc/letsencrypt/live/fullchain16.pem
rsa_private_key_file=/etc/letsencrypt/live/privkey16.pem
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=YES
ssl_ciphers=HIGH
pasv_min_port=xxxxx
pasv_max_port=yyyyy
pam_service_name=vsftpd
implicit_ssl=NO