Ghost HTTPS Redirect Loop 👻

This weekend we're cleaning up some last loose ends and putting up our first couple of posts.

The Problem

One of those loose ends was a redirect loop that we first ran into when trying to turn on HTTPS when we first set up our production Ghost install.

At the time, we were fighting with compatibility issues between MySQL 8 and Ghost, so we decided to just leave Ghost running on HTTP and troubleshoot the redirect loop later.

Then when we put up our first post, we got a bunch of Mixed Content errors from the images:

In gatsby-ghost-mysql-ubuntu/index.html:

    Insecure img urls:
        http://adampacholski.com/supercodebros/content/images/2020/11/Screen-Shot-2020-11-11-at-8.36.25-AM.png
...

Since we gave Ghost an HTTP url in our production config, all the images it was providing to Gatsby had HTTP urls, which were showing up in our nice HTTPS blog post 🙁

Some Background

Our frontend is a Gatsby app, which uses a production Ghost install on a Digital Ocean droplet as an API backend
Since Ghost is installed on a domain I plan to use for other things, we installed the production Ghost instance at a subdirectory - /supercodebros/

Try All The Things

Most of the troubleshooting advice we found for 'Ghost HTTPS Redirect Loop' centered around fixing an issue with the default Ghost nginx config:

proxy_set_header X-Forwarded-Proto https;

My guess is that the default nginx config generated by Ghost used http at some previous point for X-Forwarded-Proto, and sometime later that was fixed. The nginx config Ghost generated for us used $scheme, which should work fine for requests coming via HTTPS. We also tried changing $scheme to https - no luck, same redirect loop.

Next, we tried a cURL request, and reproduced the same error (but only once, since we didn't tell cURL to follow redirects):

curl -Is https://adampacholski.com/supercodebros/
HTTP/2 301
...
server: cloudflare
x-powered-by: Express

A couple of interesting things that came back in the response headers:

Oh yeah, I put Cloudflare in front of adampacholski.com! This was a while ago and I had forgotten - was playing with Cloudflare workers
Wait, the redirect is coming from Express??

That means the requests are getting to Ghost, and Ghost is redirecting. Nginx is fine. Confirmed that looking at the Ghost logs:

$ ghost log
[2020-11-14 18:21:51] INFO "HEAD /supercodebros/" 301 1ms
[2020-11-14 18:21:51] INFO "HEAD /supercodebros/" 301 1ms
[2020-11-14 18:21:51] INFO "HEAD /supercodebros/" 301 1ms
[2020-11-14 18:22:13] INFO "HEAD /supercodebros/" 301 1ms
...

And by shutting down Ghost via ghost stop, which gave us a 502 (progress! 😜).

Solution

This got me thinking... what if something about having Cloudflare in front of Ghost is causing Ghost to 301? Looking back at the github issue we looked at originally, we saw an answer with a Cloudflare suggestion:

We changed our SSL settings to Full (strict), and immediately got a 200 back from Ghost 🎉 👻

I was curious what that Full (strict) Cloudflare setting does - I don't think I changed it from the default setting, which is Flexible SSL. An exceptionally helpful Cloudflare troubleshooting article said this:

The Flexible SSL encryption mode in the Cloudflare SSL/TLS app Overview tab encrypts traffic between the browser and the Cloudflare network over HTTPS. However, when the Flexible SSL option is enabled, Cloudflare sends requests to your origin web server unencrypted over HTTP. Redirect loops occur if your origin web server is configured to redirect all HTTP requests to HTTPS when using the Flexible SSL option.

Yep, that would definitely cause Ghost to redirect to HTTPS, if Cloudflare was sending requests down via HTTP.

After that, it was easy to rebuild Gatsby and get rid of those Mixed Content errors 💥