I have to take this decision for customers nearly once a week, and there is no simple answer to the « varnish or nginx reverse proxy » question. A few thoughts.
Varnish was created to be a reverse proxy. NGINX is more of a webserver than can also do reverse proxy. In my humble opinion, Varnish is a better reverse proxy ; it has more features as a proxy, and the configuration can be much more complex.
The main problem with Varnish is the lack of SSL support. I find it difficult to justify in today’s world. The main developer seems to think OpenSSL is a mess, and he does not want to include such a piece of crap inside Varnish. Funny thing is, they recently started a « commercial project » ( Varnish Plus ) that incorporates SSL termination. It seems to me that they only refrain from putting SSL termination in Varnish to sell Varnish Plus. In a world without NGINX, it could work…
The other problem with Varnish is a cryptic configuration system, with very complex VCL files that have to be compiled ( which requires a working GCC on the host ). This requirement has been mocked inside official Varnish doc ( « having a gcc compiler on a host is not a security problem since the 1970s« ) and I think they are wrong here. I’m at least simplifying a hacker’s life, especially script kiddies, by having a working GCC installed on a freaking reverse proxy often used to provide an additional layer of security.
In a world where everyone forces you to use SSL ( Apple, Google, etc) I’m not sure there is any kind of future for a reverse proxy product that doesn’t support SSL. The end user doesn’t care if OpenSSL source code is secure or insecure (and this argument coming from someone who forces use to install GCC on our proxies just makes me laugh).
Varnish 4 uses the new VCL 4.0 standard, which is not compatible with VCL 3.0 files, and forces you to edit your config files (no tool was created to help you with that migration). A good VCL file can only be created with a profound understanding of how varnish works. Copy-pasting from the internet is probably not gonna lead you anywhere with varnish.
So what about NGINX ?
Nginx is a decent reverse proxy. The configuration files are much easier. It has SSL termination, and it can do a lot of different things with a single configuration file ( reverse proxy, pfp-fpm server, status files server ).
The main problem with NGINX is that they also have a « commercial product », and they really cripple the free opensource version. Fundamental features are missing, I hope it’s not on purpose. There is no way to « flush » the entire cache from a simple GET or PURGE request. You have to either remove files and/or relaunch the freaking server to clear your cache. Oh, of course, you can ask the proxy to clear a single URL, but you can’t ask it to clear the entire cache ; something each and every customer that has a website behind a reverse proxy will ask sooner or later.
I’m afraid I have just no use of a reverse proxy with no « flush cache » feature – I know everyone is gonna ask for it. There are other missing features ; especially in the monitoring of back ends. In fact, it’s easy to figure out which ones ; just look at NGINX Plus’ feature list ! That is the list of features that are intentionally left behind in the opensource NGINX.
If NGINX plus wasn’t insanely priced ( in the thousands ), asking you to « take contact with a sales rep » (no way to simply pay and download the damn thing), maybe I would consider upgrading and sell it to my customers. But they wouldn’t even let me use a NGINX Plus copy to develop my own knowledge of the product… How am I suppose to sell it to customers then ? They simply want money too fast and don’t give a shit about freelance consultants – they want to sell their support as well.
I’m mostly installing Varnish. In the worst case scenario I can always add NGINX as a SSL termination software. I simply cannot cope with NGINX missing features. But I also cannot continue to install two softwares just to get SSL. As a reverse proxy ; one of these two software will disappear in the coming months. Which one will make the good move first ?