regex – headaches and solutions

Regex could save your bacon. Try it.

Need to parse a string that doesn’t have a consistent layout but does contain regular patterns that need to be keyed on to extract information (or change it)? Want to extract values from a string without complex nested IN(…, MID(…, SUBSTR(…, LEN(… functions?

Regex seems to be the tool to use, if you can wrap your mind around the syntax for patterns that need to be keyed on. There’s plenty of examples to be found that illustrate searching for phone numbers, social security numbers, ZIP codes, ZIP+4, IPv4 addresses and so on. Add in features like look ahead, look behind, greedy or non greedy searching and I get the feeling there’s more than enough material to develop an entire graduate level course to formulate and understand regex expressions.

My own forays into regular expressions have focused mostly on mundane simple patterns. I did however find a couple uses for the look behind to extract a substring from an irregular length string. And the substring to extract was not fixed length. To me the function based way of parsing the string would have been much more complex than working out the regex syntax.

In one case there was a substring for the size of a hard drive. That substring varied in length depending on the size of the drive. In another case strings for monitor make and model, which could each be different lengths because more than one manufacturer was in the dataset and model names varied.

In both cases the target string was preceded by a fixed string that could be used to locate the sought for value within the larger string. And the position of the target string within the larger string varied because the details before it changed from pc to pc. Rescued by the regex look behind!

I posted about it on LinkedIn if you’re interested to look. Samanage and Powershell – two tools to produce helpful reports.

The significant portions of the statements are below. They allowed looking into the string for a pattern, `Fixed hard disk media”~size: “` or `name: “` or `manufacturer: “`, and then extracting the portion of the string that immediately followed.

$regexdrv = '(?<=Fixed hard disk media"~size: ")([^"]+)'
Select-String -Pattern $regexdrv | %{$_.Matches}
$regexdisp = '(?<=name: ")([^"]+)|(?<=manufacturer: ")([^"]+)'
Select-String -Pattern $regexdisp -AllMatches | %{$_.Matches}

If you find yourself struggling to parse out some information from a string, take a look at regex. You could find the solution to your problem.

Remote Support

Using RealVNC and AeroAdmin to help people at their computer

I provide remote computer support to people. Mostly to family but also occasionally to friends and acquaintances.

For family I’ve been using RealVNC. For other folks, AeroAdmin.

RealVNC has several advantages from my perspective. Two important ones are there’s a native Linux version. Family doesn’t use Linux but I do and I want to be able to run something directly on my system rather than running it through layers. Secondly and most importantly for family, it’s running all the time on their system. If support is needed we get on the phone, I start the connection, a message pops up on their screen and they click OK to allow the connection. Shazam! remote support has started. There’s nothing for them to install, no program for them to start. Just ask for support then click OK.

Other pluses are it runs on Linux so I have remote access to my own system when I’m away from it and I can register up to five systems in my RealVNC account with no subscription license required.

For the person I’m tutoring now and others I’ve helped in the past AeroAdmin has been the way to go. There’s a free license that allows occasional usage without charge. When I say “occasional” I don’t mean to imply crazy limited. From my perspective the free license is very generous with the time and connections allowed. I have certainly never bumped into the usage limits.

Of course there’s drawbacks to both. With RealVNC the issue is running it on my system. RealVNC occasionally freezes and restarts. When it does I either have to wait out the freeze before I can begin using my pc again (same experience whether I’m connected remotely or at the keyboard) or I restart my system if I’m at the keyboard. An issue I choose to live with because it gives me remote access to my system and I know how to deal with it. When I’m at the keyboard I sometimes just shut down RealVNC so there’s no chance it interrupts me. I really should file a bug report but haven’t bothered yet. And since it’s me that’s affected, not family I provide remote support to, I know what’s going on and corrective actions I can take.


EDIT 3/2020: I found the issue back in January. A wall paper changer set to change wall paper every five minutes. Increased the interval to 15 minutes and the problem became less frequent. Switched to a different wallpaper changer and the problem has gone away


AeroAdmin is a different story. The biggest challenge is always getting the person who needs to use it to get it on their system, get it started, provide me the information I need to connect, and finally make the correct choices when the connection starts so I can actually help. That sounds hard but it’s not. But it is for many people.

I’ve made documentation and sent it to people. I’ve recently made this video, AeroAdmin, letting someone help you. But it is still a challenge. Getting the connection started so people can get help always seems to be the biggest issue.

Another AeroAdmin drawback, for me not the person supported, is that I need to run Wine to run AeroAdmin. I’ve never been a fan of Wine because whenever I tried it my system was changed in ways that I couldn’t easily undo. Plus, depending on the install package, it added things to my system that I didn’t want. Getting stuck with things I didn’t want was one of the reasons for getting away from Windows. However AeroAdmin does run well under Wine so, when I need it, I run it in an isolated Linux vm with Wine installed so Wine isn’t impacting my actual running system.

Certbot automatic authentication

Enable certificate auto renew after a manual renew.

I have a number of websites run from my own web server, like this one. Something I set up to experiment with web technologies and gain some insight into how things work.

One of the things I did was setup HTTPS for the websites once I found about about EFF‘s LetsEncrypt service. I wanted to see if I could provide secure connections to my sites even if they’re only for browsing.

I was able to get HTTPS working for my sites and have the certificates renew automatically. Then I changed ISPs. With TWC, now Spectrum, there was never a problem with the automated renewals. With Optimum the renewals didn’t work.

Emails alerting me to certificate expiration were my first indication there was a problem.

The logs indicated that files on my server couldn’t be manipulated to confirm my control of the website. Plus, entering the website address as boba.org or http://boba.org no longer connected to the website (externally, on the local network it still worked). Connection to any of my hosted sites now required prefixing https:// to the name. Automatic translation from http to https no longer worked.

After talking, chatting online actually, with Optimum they told me yup, that’s just the way it works. “We block port 80 to protect you” and “you can’t unblock it”.

Panic! How to maintain my certificates so https continues working? Fortunately certbot offers a manual option that requires updating DNS TXT records. It’s slow and cumbersome and NOT suitable for long term maintenance of even one certificate containing one domain but it works.

Sixty days pass and the certificate expiration emails start again. This time I determined that I’d speak to a person at Optimum and not use the chat. After some time with my Optimum support tech, and after she escalated to a supervisor, I was told there is in fact a way to open port 80. And it is a setting available to me via my account login. So I opened port 80 and thought all set now, renewals will happen automatically.

Not so. I got more certificate expiration warning emails. What to do? All the automated renewal tests I tried indicated a problem with a plugin. I read the certbot documentation, did searches for the error and tried to find a solution that was applied to the problem I had. I didn’t find it. But I did get a clue from a post that said once a manual certification has been done that setting needs to be removed before automated renewal will work again.

After more digging I discovered the certificate config files in /etc/letsencrypt/renewal. In them were two variables that seemed likely to be related to the auto renew problem. They were authenticator = and pref_challs =. The settings were manual and dns-01 respectively.

I never touched these files. It turns out doing manual renewal with DNS TXT records using the command sudo certbot certonly --manual --preferred-challenges dns --cert-name <name> -d <name1>,<name2>,etc just changes the config files in the background. Attempting auto renew later doesn’t work because the settings in the config files have now been changed to authenticator = manual and pref_challs = dns-01.

There was no help I could find that explicitly listed the acceptable values for these variables. And I didn’t have copies of these files from before the changes. After digging around in the help for a while I decided it was likely they should be authenticator = apache and pref_challs = http-01.

I made the change for one certificate and tested auto renew. Eureka, it worked!!

Next I changed the config files for all the certificates and did a test to see if it worked.

$ sudo certbot renew --dry-run
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/alanboba.net/fullchain.pem (success)
/etc/letsencrypt/live/andrewboba.org/fullchain.pem (success)
/etc/letsencrypt/live/danielboba.org/fullchain.pem (success)
/etc/letsencrypt/live/kevinkellypouredfoundations.com/fullchain.pem (success)
/etc/letsencrypt/live/www.anhnguyen.org/fullchain.pem (success)
/etc/letsencrypt/live/www.conorboba.org/fullchain.pem (success)
/etc/letsencrypt/live/www.mainguyen.org/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

It worked. All my certificates will again auto renew.

This website was created after the problems began. So I didn’t even attempt to make it https. Now that I’ve figured out how to have my certs auto renew again I’ll be converting this site over to https too.