Tag: certbot

Certbot headaches!

Modifying certificates with certbot. It works and it was a long journey to get it done.

If anyone’s reading this you may have noticed the URL is wp.boba.org. Possibly you entered www.wp.boba.org to get here and saw it change to wp.boba.org. Whatever. Until this evening (15 Jan, 2020) the URL’s protocol would have been http://. https:// wouldn’t have even connected. Now, even if http:// is entered it changes to https://. Hooray!!

The SSL certificate for this website is now part of the alanboba.net certificate. But, until tonight, I was unable to expand the domains in the alanboba.net certificate to include wp.boba.org and www.wp.boba.org.

My attempts to expand the alanboba.net certificate began nearly a month ago. Everything I tried failed. In desperation I posted on the LetsEncrypt community forum a little over three weeks ago, Apache certificate modification not successful, hoping someone would quickly recognize the problem and suggest a solution.

That didn’t pan out. Not a lot of respondents. The fix suggested didn’t address the issue the error message presented, the request was “unauthorized”, or suggest if the message might be misleading.

Domain: wp.boba.org
Type: unauthorized
Detail: Invalid response from http://wp.boba.org/.well-known/acme-challenge/QVV-1Skk-Xvrr6QAL-IvvDZuMGnhr2mNOfoAWbkYCnw [67.86.147.116]: "\n\n404 Not Found\n\n

More reading. More checking settings on this server. Some experimental configuration changes to see if the issue resolved and the command certbot –expand… would succeed in adding two additional domains to the existing certificate. None of the changes worked.

Finally came across a different command and decided to try it. As I understood it, it is meant to renew existing certificates not add domains to them. However it does include a “webroot” parameter and some of the documents I’d read suggested the webroot location might not be correctly interpreted by the command I was using.

The documentation I found doesn’t say anything to suggest the command can be used to expand the domain names covered by a certificate. I just had an inspiration and decided that if webroot might be the problem then explicitly specifying the webroot and adding domain names at the same time might turn the trick.

Tonight I tired the command with the webroot parameter and my additional domains appended to the list of domains already on the certificate. Surprise and delight! The domains were added to the certificate AND the protocol is now changed to https:// even if http:// is used in the URL name!

The following command…

sudo certbot run -a webroot -i apache -w /var/www/wp.boba.org/public_html -d alanboba.net,boba.org,sclc.boba.org,train.boba.org,training.boba.org,www.alanboba.net,www.boba.org,wp.boba.org,www.wp.boba.org

Produced the output below. Plus it added my two additional domains to the existing certificate and modified apache’s config for the website so http:// requests are rewritten as https://. Like I said at the beginning, hooray!!

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you requested (ref: /etc/letsencrypt/renewal/alanboba.net.conf)
It contains these names: alanboba.net, boba.org, sclc.boba.org, train.boba.org, training.boba.org, www.alanboba.net, www.boba.org
You requested these names for the new certificate: alanboba.net, boba.org, sclc.boba.org, train.boba.org, training.boba.org, www.alanboba.net, www.boba.org, wp.boba.org, www.wp.boba.org.
Do you want to expand and replace this existing certificate with the new certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: E
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wp.boba.org
http-01 challenge for www.wp.boba.org
Using the webroot path /var/www/wp.boba.org/public_html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/sclc.boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/train.boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/train.boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/boba.org-le-ssl.conf
Created an SSL vhost at /etc/apache2/sites-available/wp.boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/wp.boba.org-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/wp.boba.org-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/wp.boba.org-le-ssl.conf
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS.
You can undo this change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Enhancement redirect was already set.
Enhancement redirect was already set.
Enhancement redirect was already set.
Enhancement redirect was already set.
Enhancement redirect was already set.
Enhancement redirect was already set.
Enhancement redirect was already set.
Redirecting vhost in /etc/apache2/sites-enabled/wp.boba.org.conf to ssl vhost in /etc/apache2/sites-available/wp.boba.org-le-ssl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Your existing certificate has been successfully renewed, and the new certificate has been installed.
The new certificate covers the following domains: https://alanboba.net,https://boba.org, https://sclc.boba.org, https://train.boba.org,https://training.boba.org, https://www.alanboba.net, https://www.boba.org,https://wp.boba.org, and https://www.wp.boba.org

Certbot automatic authentication

Enable certificate auto renew after a manual renew.

I have a number of websites run from my own web server, like this one. Something I set up to experiment with web technologies and gain some insight into how things work.

One of the things I did was setup HTTPS for the websites once I found about about EFF‘s LetsEncrypt service. I wanted to see if I could provide secure connections to my sites even if they’re only for browsing.

I was able to get HTTPS working for my sites and have the certificates renew automatically. Then I changed ISPs. With TWC, now Spectrum, there was never a problem with the automated renewals. With Optimum the renewals didn’t work.

Emails alerting me to certificate expiration were my first indication there was a problem.

The logs indicated that files on my server couldn’t be manipulated to confirm my control of the website. Plus, entering the website address as boba.org or http://boba.org no longer connected to the website (externally, on the local network it still worked). Connection to any of my hosted sites now required prefixing https:// to the name. Automatic translation from http to https no longer worked.

After talking, chatting online actually, with Optimum they told me yup, that’s just the way it works. “We block port 80 to protect you” and “you can’t unblock it”.

Panic! How to maintain my certificates so https continues working? Fortunately certbot offers a manual option that requires updating DNS TXT records. It’s slow and cumbersome and NOT suitable for long term maintenance of even one certificate containing one domain but it works.

Sixty days pass and the certificate expiration emails start again. This time I determined that I’d speak to a person at Optimum and not use the chat. After some time with my Optimum support tech, and after she escalated to a supervisor, I was told there is in fact a way to open port 80. And it is a setting available to me via my account login. So I opened port 80 and thought all set now, renewals will happen automatically.

Not so. I got more certificate expiration warning emails. What to do? All the automated renewal tests I tried indicated a problem with a plugin. I read the certbot documentation, did searches for the error and tried to find a solution that was applied to the problem I had. I didn’t find it. But I did get a clue from a post that said once a manual certification has been done that setting needs to be removed before automated renewal will work again.

After more digging I discovered the certificate config files in /etc/letsencrypt/renewal. In them were two variables that seemed likely to be related to the auto renew problem. They were authenticator = and pref_challs =. The settings were manual and dns-01 respectively.

I never touched these files. It turns out doing manual renewal with DNS TXT records using the command sudo certbot certonly --manual --preferred-challenges dns --cert-name <name> -d <name1>,<name2>,etc just changes the config files in the background. Attempting auto renew later doesn’t work because the settings in the config files have now been changed to authenticator = manual and pref_challs = dns-01.

There was no help I could find that explicitly listed the acceptable values for these variables. And I didn’t have copies of these files from before the changes. After digging around in the help for a while I decided it was likely they should be authenticator = apache and pref_challs = http-01.

I made the change for one certificate and tested auto renew. Eureka, it worked!!

Next I changed the config files for all the certificates and did a test to see if it worked.

$ sudo certbot renew --dry-run
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/alanboba.net/fullchain.pem (success)
/etc/letsencrypt/live/andrewboba.org/fullchain.pem (success)
/etc/letsencrypt/live/danielboba.org/fullchain.pem (success)
/etc/letsencrypt/live/kevinkellypouredfoundations.com/fullchain.pem (success)
/etc/letsencrypt/live/www.anhnguyen.org/fullchain.pem (success)
/etc/letsencrypt/live/www.conorboba.org/fullchain.pem (success)
/etc/letsencrypt/live/www.mainguyen.org/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

It worked. All my certificates will again auto renew.

This website was created after the problems began. So I didn’t even attempt to make it https. Now that I’ve figured out how to have my certs auto renew again I’ll be converting this site over to https too.