Setup HAProxy with CloudFlare SSL termination on OpenWRT

Setup instructions for HAProxy with HTTP SSL termination on OpenWRT with CloudFlare Origin CA and Authenticated Origin Pulls.

Motivation

I activated IPv6 on my Pi. Due to the absence of NAT, I needed to update CloudFlare with Pi’s new IPv6 every time it changes and not forgetting to update the white-list of CloudFlare IP addresses in firewall for each Pi.

To ease managing my network, I decided to setup HAProxy as a front-end load balancer on OpenWRT that:

  1. Presents CloudFlare Origin CA to incoming connection so that CloudFlare Strict SSL option can be enabled
  2. Authenticates incoming connection to ensure only CloudFlare servers are allowed to pull content
  3. Terminates the SSL and proxies HTTP connections to and from my Raspberry Pi backend web server via non-SSL for higher performance

Besides the above, I am also sharing tips and tricks. :)

Assumptions

This tutorial details steps I used for my setup. To follow this tutorial, you need a fair bit of understanding on:

  1. Installing packages from OpenWRT repository
  2. Basic understanding on HTTP Load Balancing
  3. CloudFlare Origin CA and Authenticated Origin Pulls
  4. Firewall setup, IPv4 and IPv6

HAProxy setup

In OpenWRT Chaos Calmer repository, you should be able to find haproxy package, install it. At time of writing, the version available is 1.5.15-13.

After installing, edit its configuration at /etc/haproxy.cfg.

Add a front-end into the configuration file by appending the following lines. Please read the comments:

frontend https-in
    # Bind on port 443 only, CloudFlare Page Rule has been setup to redirect HTTP to HTTPS
    # cloudflare-origin-ca.pem = CloudFlare Origin CA certificate and key concatenated
    # cloudflare-origin-pull-pem = CloudFlare Authenticated Origin Pulls certificate
    bind :443 ssl strict-sni crt /etc/ssl/private/cloudflare-origin-ca.pem verify required ca-file /etc/ssl/private/cloudflare-origin-pull-ca.pem
    mode http

    # Allow CloudFlare IP addresses, deny others
    # I have these commented because my firewall is enforcing this
    #acl cloudflare_ipv4 src -f /etc/cloudflare/ips-v4
    #acl cloudflare_ipv6 src -f /etc/cloudflare/ips-v6
    #http-request allow if cloudflare_ipv4
    #http-request allow if cloudflare_ipv6
    #http-request deny

    # Do not need this line since CloudFlare sends it and HAProxy will send all incoming headers to backend
    #reqadd X-Forwarded-Proto:\ http

    # HTTP Strict Transport Security
    rspadd Strict-Transport-Security:\ max-age=15768000

    # Define hostnames that we are serving
    acl host_www_leowkahman_com hdr(host) -i www.leowkahman.com
    acl host_static_leowkahman_com hdr(host) -i static.leowkahman.com

    # Determine which backend to use
    use_backend backend_raspberrypi if host_www_leowkahman_com
    use_backend backend_raspberrypi if host_static_leowkahman_com

Next, add a backend:

backend backend_raspberrypi
    mode http
    balance leastconn
    # Replace [raspberry IP] with your backend server's IP address
    server raspberrypi [raspberry IP]:80 check

In case you are interested in seeing statistics, I suggest binding on a different port and one that is not exposed to the outside world:

listen stats
    bind :1936
    mode http
    stats enable
    stats uri /
    stats hide-version

I hardened my SSL security by appending the following lines into global section:

global
    # set default parameters to the modern configuration
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

Next, add a default section so that I do not have to put them all over the place:

defaults
    # Slowloris protection
    timeout http-request 5s
    timeout connect 5s
    timeout client 30s
    timeout server 30s
    timeout http-keep-alive 4s

    # Close the backend connection
    option http-server-close

Do not start/restart the HAProxy yet.

Apache on Raspberry Pi as my backend HTTP server

You may want to refer to my post on how to setup HTTP server on Pi. Note that you do not need to setup SSL on it since SSL is terminated at the load balancer.

To configure the Pi firewall to allow inbound traffic from LAN subnet only, follow these steps:

UFW must be installed:

sudo apt-get install ufw

Ensure that IPv6 is enabled in /etc/default/ufw, you should see IPV6=yes.

Configure default rules; deny incoming, allow outgoing:

sudo ufw default deny incoming
sudo ufw default allow outgoing

Allow all incoming traffic from your LAN (change the subnet accordingly):

sudo ufw allow from 192.168.0.0/24

Verify list of rules:

sudo ufw status

Enable UFW:

sudo ufw enable

OpenWRT firewall

Do not forward port 443

If you are currently forwarding port 443 to your backend web server, you need to remove the rule because HAProxy needs to listen on that port.

CloudFlare IP white-listing

By default, OpenWRT denies all incoming connections unless you have explicitly allowed the traffic through in your firewall configuration. I did not want to open that port to all inbound traffic. Instead, limiting to CloudFlare IP addresses only.

To accomplish this, I needed to create bash script download the latest CloudFlare IPv4 and IPv6 lists and subsequently load it into memory to be used by the firewall.

Create a folder to store the script and IP address lists:

mkdir -p /etc/cloudflare

Use your favourite text editor, I use nano to create a bash script at /etc/cloudflare/cloudflare-whitelist.sh with the following lines:

#!/bin/sh
set -e

# Pull the latest CloudFlare IP list into memory and overwrite existing copy
# only when changed to minimise non-volatile storage wear
mkdir -p /var/run/cloudflare
wget https://www.cloudflare.com/ips-v4 -O /var/run/cloudflare/ips-v4
wget https://www.cloudflare.com/ips-v6 -O /var/run/cloudflare/ips-v6
rsync --size-only /var/run/cloudflare/ips-v* /etc/cloudflare/
rm /var/run/cloudflare/ips-v*

# Create hash sets to store list of IP
ipset create -exist cloudflareipv4s hash:net family inet
ipset create -exist cloudflareipv6s hash:net family inet6
for cfip in $(cat /etc/cloudflare/ips-v4); do ipset add -exist cloudflareipv4s $cfip; done
for cfip in $(cat /etc/cloudflare/ips-v6); do ipset add -exist cloudflareipv6s $cfip; done

Give it permissions to run:

chmod 755 /etc/cloudflare/cloudflare-whitelist.sh

Try to run it:

/etc/cloudflare/cloudflare-whitelist.sh

Upon successful run, there should be two new files in the same folder:

  1. ips-v4
  2. ips-v6

If you run into issues, you are probably missing some packages that I use. If so, please install them (i.e. wget, rsync, ca-certificates, iptables, ip6tables, ipset) or you will have to find another way around.

Schedule this script to run weekly by putting into Cron:

# Every week, update CloudFlare IP white list
0 0 * * 1 /etc/cloudflare/cloudflare-whitelist.sh

In Firewall custom rules, add the following lines:

# Allow inbound HTTPS from CloudFlare IPs obtained from hash sets
ipset create -exist cloudflareipv4s hash:net family inet
ipset create -exist cloudflareipv6s hash:net family inet6
for cfip in $(cat /etc/cloudflare/ips-v4); do ipset add -exist cloudflareipv4s $cfip; done
for cfip in $(cat /etc/cloudflare/ips-v6); do ipset add -exist cloudflareipv6s $cfip; done
iptables -A INPUT -m set --match-set cloudflareipv4s src -p tcp -m multiport --dport 443 -j ACCEPT
ip6tables -A INPUT -m set --match-set cloudflareipv6s src -p tcp -m multiport --dport 443 -j ACCEPT

For firewall rules to take effect:

/etc/init.d/firewall restart

WordPress HTTP/HTTPS mismatch fix

Previously, I had my Pi serving SSL traffic. Now with SSL terminated at load balancer, WordPress on my Pi thinks that the incoming visitor is browsing without SSL thus giving http:// link prefixes. To correct this, I added the following lines to the top of wp-config.php:

define('FORCE_SSL_ADMIN', true);
if (strpos($_SERVER['HTTP_X_FORWARDED_PROTO'], 'https') !== false) {
    $_SERVER['HTTPS']='on';
    $_SERVER['SERVER_PORT'] = 443;
}

CloudFlare Dynamic DNS with Orange Cloud

Since I am using Dynamic IP address, it will change from time to time. CloudFlare must be notified of my IP changes so I have configured as below.

Refer to my post on how to update DNS on CloudFlare. Note that the script on OpenWRT repository sets CloudFlare cloud colour to grey. It is important that you refer to this link for instructions on how to make it set to orange.

In addition to that, I have added a IPv6 DDNS entry:

config service 'cloudflare_ipv6_leowkahman_com'
    option enabled '1'
    option use_ipv6 '1'
    option service_name 'CloudFlare'
    option domain 'leowkahman.com'
    option username '[CloudFlare username]'
    option password '[CloudFlare API key]'
    option use_https '1'
    option cacert '/etc/ssl/certs'
    option ip_source 'network'
    option interface 'wan6'
    option ip_network 'wan6'
    option use_syslog '2'
    option use_logfile '0'
    option check_interval '24'
    option check_unit 'hours'
    option force_interval '0'

Start using

Restart HAProxy:

/etc/init.d/haproxy restart

You should be good to go. :)

Now that I have a load balancer in place, I have the pre-requisites for adding more backend nodes in future.