I am getting the error below in trying to renew my certificate from the command line (and thus too from cron). From searching similar error reports, I understand that it means that I initially created a certificate manually and so cannot renew automatically. I am not sure what "manually" means in this context.

(I am not using Nginx integration or the like. Rather, I am running a Nodejs app which references the SSL certificate files in application code, loading an HTTPS listener alongside the HTTP listener. That works fine.)

What is the way to create certificate that I can renew automatically?

sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/shpil.joshuafox.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate abc.example.com with error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/abc.example.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
  • Did that script happen to use DNS hooking?
    – MechMK1
    Nov 10 '21 at 12:48
  • I have not set up any special hook. What is DNS hooking? Is it a process for automatically changing DNS zone records to prove ownership of a domain (as I did, manually). Is it what is described here? digitalocean.com/community/tutorials/…
    – Joshua Fox
    Nov 10 '21 at 13:57

Your question title and body ask two different questions. One asks "What does it mean for a certificate to be generated automatically?" and the other is "Why is my script not working?". I will attempt to answer both:

What does it mean to create a certificate "automatically"?

In order for a certificate authority to generate a certificate for a domain, they need to verify that the person, who requested the certificate, actually owned the domain. If they did not, you could request a certificate for security.stackexchange.com, and I am fairly confident that you don't own that domain.

Back in the day, when certificates cost an arm and a leg, that process was quite complicated. But with Let's Encrypt, there are are several different challenge types, but the two most popular are "HTTP-01" and "DNS-01".

With the HTTP-01 challenge, Let's Encrypt gives you a token, and then expects to find that token under http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>. Essentially, the idea here is that if you can prove that you can generate valid HTTP responses for that domain, you most likely control that domain.

With the DNS-01 challenge, Let's Encrypt gives you a token, and then expects to find a DNS TXT record under _acme-challenge.<YOUR_DOMAIN>, which contains said token. Again, the idea is that if you can publish valid DNS records for a domain, you most likely own that domain.

Armed with that knowledge, we can now look at automation. It's easy to do all of this manually, and for a long time, that's precisely what I did, four times a year. I'd issue a certificate request, go to my DNS console, add the request, wait for the DNS records to propagate and then confirm that the records can now be checked.

This process is long, tedious and involves a lot of cursing, possibly even a prayer or two to the Omnissiah. Automation would be a lot better! So, let's check what we need to automate:

We need some way to automatically check if our certificate is about to expire, and if so, request a new certificate, a new token, then automatically "deploy" that token and confirm to Let's Encrypt that they can now check the token. If the verification succeeded, we then need to overwrite the old certificate with the new one and possibly instruct the web server to reload the new certificate.

So to answer your first question: A certificate is considered to be generated "automatically" if it is done via some process, that does not require human intervention in any way.

Why does my script not work?

When you request a new certificate using certbot, you have several options:

(default) run   Obtain & install a certificate in your current webserver
certonly        Obtain or renew a certificate, but do not install it
renew           Renew all previously obtained certificates that are near expiry
enhance         Add security enhancements to your existing configuration

The two most important for generating a certificate is run and certonly, but for automation, it's renew. run is used when you trust Apache and nginx integration, certonly is when you use a different web server or have problems with the automatic integration.

Aside from the mode, there are also flags, which are where your issue stems from:

--apache          Use the Apache plugin for authentication & installation
--standalone      Run a standalone webserver for authentication
--nginx           Use the Nginx plugin for authentication & installation
--webroot         Place files in a server's webroot folder for authentication
--manual          Obtain certificates interactively, or using shell script hooks

The first four are all for the "HTTP-01" kind of challenge. Essentially, they try to figure out where to place the token, so that your web server will serve the token as requested. --manual on the other hand, as the documentation suggests, is when you do things...well...manually. Or if you write your own shell scripts to deal with that.

This is required for DNS challenges, since certbot does not include any functionality to automatically enter or modify DNS entries. You first created your certificate using the --manual flag, indicating that you would manually create the DNS entries and inform certbot when it's done.

In order to automate this process, the error message explains exactly what you need to do:

An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.

Essentially, you have to specify a script, which will deploy the DNS records for you - if your DNS provider supports that, that is. This thread in the Let's Encrypt forums lists DNS providers, for which integration is easy, and even attach several shell scripts, which can do this process for you.

If your DNS provider is not listed, you have the following options:

  • Switch to HTTP-01 challenges, if possible.
  • Write your own shell script, if possible.
  • Continue with manual renewal.

I personally switched from DNS-01 to HTTP-01, using the --standalone mode, as direct nginx integration gave me issues for servers, which were configured to not be accessible outside

The following lines in my renewal config specify that I want to use standalone mode and automatically shut down nginx before beginning the process, and starting it again after it is done.

authenticator = standalone
pre_hook = systemctl stop nginx.service
post_hook = systemctl start nginx.service
  • 1
    @JoshuaFox I find it easier to stop whichever server is listening on port 80, use the --standalone mode and then start it again. You can use the --dry-run flag for testing purposes.
    – MechMK1
    Nov 10 '21 at 14:57
  • 1
    @JoshuaFox For roughly 20 seconds every two months. If this is an issue, you may need to work on alternatives, such as the --webroot flag
    – MechMK1
    Nov 10 '21 at 15:09
  • 1
    @JoshuaFox Your server may not have a webroot and work purely as a reverse proxy. That's one instance where --webroot would not work.
    – MechMK1
    Nov 10 '21 at 20:52
  • 1
    @JoshuaFox You should be able to set up a cron job. In fact, you should try sudo certbot renew --dry-run and see if that gives you a success message. Depending on your distro, certbot may already come with a pre-configured crontab in /etc/cron.d/certbot
    – MechMK1
    Nov 11 '21 at 14:30
  • 1
    @JoshuaFox You don't need any of that in the crontab. The crontab was fine as-is. Those things should go into your renewal configs in /etc/letsencrypt/renewal
    – MechMK1
    Nov 12 '21 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.