Syncplify.me Server v8

Appearance

Web servers in front of automatic certificates ​

Automatic certificates are validated by the certificate authority on TCP port 443, and the validation is answered inside Syncplify Server!'s own TLS handshake. On a machine where a web server or a reverse proxy already owns port 443 that leaves exactly two honest arrangements:

  1. The web server passes port 443 through unchanged. It accepts the TCP connection and hands the bytes to the management console (port 6443 by default) or to the site's HTTPS listener without terminating TLS. The certificate authority then reaches Syncplify Server! as if nothing stood in between. Caddy with its layer4 plugin, nginx with its stream module and HAProxy in TCP mode can do this. IIS and Apache cannot.
  2. The two share the machine. Port 443 is split by IP address (the web server listens on one address, Syncplify Server! answers validations on another), or the web server moves to another port, or the router forwards public port 443 straight to one of the site's listeners. This works with every web server.

A server that terminates TLS on port 443 and proxies HTTP behind it (the stock configuration of every web server, including Caddy's reverse_proxy, nginx's proxy_pass inside an http block, Apache's mod_proxy, IIS with Application Request Routing) can never pass a TLS-ALPN-01 validation. The readiness check names it: "answered by something that is not this server's validation responder".

What the readiness check expects ​

Before anything can be enabled, the readiness check probes every name on port 443 from inside the cluster. With a pass through in place it reports the port as held, sees its own probe get through, and shows the holder as a warning rather than a blocking item: "Port 443 on node X is held by caddy (PID 1234), which passes validations through to this installation." Keep the per node responder switched off in Global Configuration in that case; it could not bind the port on that node anyway, and switched on it would report the failed bind at every validation. The responder is one switch for the whole cluster, so a cluster that mixes proxied nodes with nodes that rely on the responder lives with those reports on the proxied nodes.

Whenever a proxy sits in front of a listener, have it send the PROXY protocol header (version 1 or 2, both are accepted) and switch Behind HAProxy on for that listener: the Service Binding in Global Configuration for the console, the binding itself in the site's bindings for a site listener. The Shield, the allow lists and the logs then see the real client address instead of the proxy's. The two settings must agree: a header without the switch breaks every handshake, the switch without a header refuses every connection.

After the change, Test now on the Automatic certificates tab is the proof: it ships a throwaway validation and dials every address of every name through the same path the certificate authority will take, and the result lines show which address answered and from where.

Caddy (Linux, Windows, macOS) ​

Caddy's normal configuration terminates TLS and obtains its own certificates, so it needs the layer4 plugin for a raw pass through. Add it to an existing installation (the command downloads a build that includes the plugin and replaces the binary) or build it with xcaddy:

caddy add-package github.com/mholt/caddy-l4
xcaddy build --with github.com/mholt/caddy-l4

The simplest Caddyfile hands all of port 443 to the management console:

{
	layer4 {
		:443 {
			route {
				proxy {
					upstream 127.0.0.1:6443
					proxy_protocol v2
				}
			}
		}
	}
}

To keep serving Caddy's own sites on the same machine, route by server name and move Caddy's own HTTPS to another port. Caddy's HTTP to HTTPS redirects then point at that port, which only matters for sites Caddy serves itself:

{
	https_port 8443

	layer4 {
		:443 {
			@console tls sni admin.example.com
			route @console {
				proxy {
					upstream 127.0.0.1:6443
					proxy_protocol v2
				}
			}
			route {
				proxy 127.0.0.1:8443
			}
		}
	}
}

www.example.org {
	root * /var/www/example
	file_server
}

Replace 127.0.0.1:6443 with the site's HTTPS listener to pass validations to a virtual site instead of the console, and list every name the certificate covers in the sni matcher. Validate with caddy validate --config /etc/caddy/Caddyfile and restart Caddy. On the Syncplify Server! side switch Behind HAProxy on for the listener Caddy feeds.

nginx (Linux, Windows) ​

nginx passes TCP through with its stream module, which most distribution packages include (nginx -V lists --with-stream; on Debian and Ubuntu it comes from the libnginx-mod-stream package). The stream block lives at the top level of nginx.conf, beside http, never inside it. The simplest form hands all of port 443 to the console; nginx sends version 1 of the PROXY protocol header, which is accepted:

stream {
    server {
        listen 443;
        listen [::]:443;
        proxy_pass 127.0.0.1:6443;
        proxy_protocol on;
    }
}

To keep nginx's own HTTPS sites, route by server name with ssl_preread, which reads the server name from the hello without terminating TLS (the build must include --with-stream_ssl_preread_module), and move the http servers off port 443:

stream {
    map $ssl_preread_server_name $port443 {
        admin.example.com  syncplify;
        default            own_sites;
    }
    upstream syncplify { server 127.0.0.1:6443; }
    upstream own_sites { server 127.0.0.1:8443; }

    server {
        listen 443;
        listen [::]:443;
        ssl_preread on;
        proxy_pass $port443;
        proxy_protocol on;
    }
}

http {
    server {
        listen 127.0.0.1:8443 ssl proxy_protocol;
        set_real_ip_from 127.0.0.1;
        real_ip_header proxy_protocol;
        server_name www.example.org;
        ...
    }
}

No server in the http block may keep listen 443 once the stream server owns the port. Check with nginx -t, then reload.

HAProxy (Linux) ​

HAProxy passes TCP through in mode tcp, routes by server name after inspecting the hello, and sends the PROXY protocol header with send-proxy-v2:

frontend port443
    bind :::443 v4v6
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    use_backend syncplify if { req.ssl_sni -i admin.example.com }
    default_backend own_sites

backend syncplify
    mode tcp
    server console 127.0.0.1:6443 send-proxy-v2

backend own_sites
    mode tcp
    server local 127.0.0.1:8443

Drop the use_backend line and make syncplify the default backend to hand all of port 443 over. The site listeners accept the same header once Behind HAProxy is on for their bindings.

IIS and HTTP.sys (Windows) ​

On Windows, port 443 is rarely held by a process you can see. It is held by HTTP.sys, the kernel driver that serves IIS, WinRM over HTTPS, Windows Admin Center, Web Deploy, SQL Server Reporting Services and other Windows web services. The readiness check shows the holder as "System (PID 4)" and says so. HTTP.sys cannot pass a TCP connection through to another program and IIS's Application Request Routing terminates TLS, so with IIS the machine has to be shared, in one of three ways.

First find out what registered the port. The certificate bindings of HTTP.sys list every address and port it serves TLS on, and the request queues list the registered URLs with the process that owns each:

netstat -ano -p tcp | findstr :443
netsh http show sslcert
netsh http show servicestate view=requestq

A binding on 0.0.0.0:443 or [::]:443 means HTTP.sys took the port on every address.

Move IIS off port 443. When the IIS site is not the machine's public web site, edit its binding: in IIS Manager open the site, choose Bindings, select the https binding, choose Edit, set another port such as 8443 and keep the certificate. Do the same for any other HTTP.sys consumer on 443 (Windows Admin Center asks for its port during setup and can be changed by reinstalling it). Port 443 is then free, and the per node responder or a site listener bound to port 443 answers validations as on any other machine.

Share the machine by IP address. When IIS must keep port 443, give the server a second IP address and tell HTTP.sys to listen only on the first. By default the IP listen list of HTTP.sys is empty, which means every address; once it holds an address, HTTP.sys listens on that address only, on every port it uses:

netsh http show iplisten
netsh http add iplisten ipaddress=192.0.2.10

Add 127.0.0.1 as well if anything on the machine reaches an HTTP.sys service over loopback, and remember that the list applies to every HTTP.sys listener, WinRM on port 5985 included, so include every address those must stay reachable on. Then change the IIS bindings from "All Unassigned" to 192.0.2.10 (IIS Manager, Bindings, Edit, pick the address and select the certificate again, which binds it to the new address; netsh http show sslcert confirms the binding now reads 192.0.2.10:443), and restart the HTTP service and the services that depend on it, or reboot:

net stop http /y
net start w3svc

Syncplify Server! then answers validations on the second address: set the responder's bind address to 192.0.2.11 in Global Configuration, Automatic certificates, or bind the site's HTTPS or implicit FTPS listener to 192.0.2.11:443, and point the DNS names of the certificate at 192.0.2.11.

Forward port 443 at the router. When the public address is not the machine's own, forward public port 443 to the port of one of the site's HTTPS or implicit FTPS listeners (6444 or 990 by default); nothing on the machine changes and HTTP.sys keeps its port.

The firewall rules the installer creates are per program, so the management service and the workers may receive port 443 without a new rule. Caddy with the layer4 plugin also runs on Windows and can stand in front of both, routing IIS's names to IIS moved to another port and the certificate's names to Syncplify Server!; IIS does not understand the PROXY protocol, so its sites then log the proxy's address.

Apache httpd (Linux, Windows) ​

Apache has no TCP mode: mod_proxy terminates TLS, so Apache cannot pass a validation through. Share the machine instead. Apache binds port 443 on every address with the default Listen 443; replace it with the address Apache should keep, in ports.conf or httpd.conf and in every VirtualHost on that port:

Listen 192.0.2.10:443
<VirtualHost 192.0.2.10:443>
    ServerName www.example.org
    ...
</VirtualHost>

Port 443 on the machine's other addresses is then free, and the responder's bind address or a site listener takes one of them, exactly as described for IIS above. Moving Apache to another port, forwarding public port 443 at the router, or putting Caddy, nginx or HAProxy in front of both Apache and Syncplify Server! are the other ways out.