中文How EPAM Used NGINX Plus to Build Windows Containers Five Years Before the Rest of the World
Innovating while Cutting Costs and Migrating to the Public Cloud
Overview
Every business is uniquely challenged by the pressures of digitization. EPAM helps its customers become competitive – and stay competitive with its best-in-class software engineering with digital strategy and experience design, business consulting and technology innovation services. Its expert team and organized service practices help assess your market and design winning experiences, whether as a standalone engagement or a full end-to-end transformation program.
Challenge
EPAM Systems Inc. offers consulting‑based engineering services across a range of vertical markets. Since 1993, the company has partnered with the world’s leading companies to imagine, design, and engineer customer experiences that accelerate business transformation. One aspect of that work is InfoNgen, a service EPAM offers to customers which will alert them to important documents related to their market. InfoNgen crawls the Web to search millions of documents every day, sending related alerts to users in fields ranging from insurance to pharmacological research.
InfoNgen, originally a startup, was acquired by EPAM in 2010. Its technological core was moved to a private data center run by EPAM. After the acquisition, InfoNgen, as a business unit inside a larger consulting operation, required flexibility, stability, and affordability from its technology stack. By 2012, the biggest mandate for this business unit was to cut operational costs and to start migrating to the public cloud – which was, by that point, maturing to become a viable alternative for a robust, public‑facing service.
“Windows Containers were non‑existent essentially. So we decided to build our own containers on Windows. It’s not a big deal, it wasn’t a big challenge for us since all the operating system primitives were already there for building your own containers. So we just did it.”
While the application was hosted in EPAM’s datacenter, the company was using a Citrix NetScaler appliance to perform load balancing and reverse proxy functionality. Unfortunately, this prevented a move to the cloud, as the load balancer was a piece of hardware. Additionally, Kirill Lutarevich, senior delivery manager at EPAM, said the Citrix license was around $50,000/year, a high price for something which included functionality that wasn’t even being used by the team.
Solution
At first, Lutarevich replaced NetScaler with the free open source version of NGINX. This reduced load balancer and reverse proxy infrastructure costs to almost zero, and enabled the move to cloud, thanks to NGINX’s status as a multi‑platform, software‑based solution. The team quickly upgraded to NGINX Plus, however, in order to avail themselves of NGINX support and services. The upgrade still left the team with a tremendous cost savings over NetScaler.
InfoNegen’s solution for moving its legacy datacenter applications into the cloud and coupling them with microservices is now based entirely on NGINX Plus as the core traffic routing and balancing system.
InfoNgen uses an active/active pair of NGINX Plus instances in front of its microservices application. InfoNgen has 60–70 microservices running in production. The microservices are 80–85% hosted in custom‑built Windows containers; the rest are Linux. EPAM is not using the native container support found in Windows Server 2016, which came along four years too late for them. “The majority of our systems are on Windows Server 2012. Windows Server 2012 doesn’t support the concept of containers. So we built our own containers, custom containers,” said Lutarevich.
InfoNgen uses ZooKeeper for service discovery. With InfoNgen’s microservices architecture, the IP address and port number of services are constantly changing. All its services register with ZooKeeper, which maintains a registry. NGINX Plus then pulls information about services from ZooKeeper, and routes requests accordingly.
Since the transition, Lutarevich said that the team has had no issues or downtime related to its usage of NGINX Plus. Acting as an SSL terminator on the outside of its internal network, a single instance of NGINX Plus is able to handle all the traffic routing, balancing, and health monitoring tasks required by the system. The availability of robust health monitoring capabilities in NGINX Plus was, Lutarevich said, another deciding factor when the team chose to upgrade from the free version to the commercial edition.
Results
Cost Savings
Since the initial transition in 2012 to NGINX, and the subsequent move to NGINX Plus, Lutarevich was able to satisfy one of the most important needs of his company, which has been cutting infrastructure costs.
“Our budget is very limited; we have to do what we can. We are under constant pressure to lower our infrastructure costs. What was not acceptable to us with Citrix Netscaler was the licensing costs, which was exorbitant.”
By replacing Citrix Netscaler appliances with NGINX Plus, Lutarevich was able to cut costs significantly. Citrix Netscaler cost $50,000/year, according to Lutarevich, and NGIN Plus was a fraction of that cost.
Rolling Updates
The two internally hosted instances of NGINX Plus are used to enable rolling upgrades to services, said Lutarevich. NGINX Plus is able to reroute traffic to non‑upgraded nodes while software upgrades take place on others. The result, said Lutarevich, is that NGINX Plus has allowed InfoNgen to remain online 24/7, without any downtime for upgrades, since the team implemented rolling upgrades with NGINX Plus.
In addition to rolling upgrades, NGINX Plus has provided a stable environment for both legacy Windows applications and modern Linux ones. While InfoNgen would love to move off of Windows in order to save money on hosting, this is simply not possible because of the size and importance of the core .NET application.
“Internally, we’ve been using it as a load balancer for our microservices architecture, so basically it’s like a system of registration and discovery for our microservices ,which we controlled through the NGINX API. When we need to update or replace certain microservices endpoints, we turn them off, then turn them back on again, doing rolling upgrades by using the API provided by NGINX Plus,” said Lutarevich.
Community
The results of the move to NGINX Plus at InfoNgen are almost unexciting at this point. That’s because the transition was completed almost four years ago, and since that time, no issues have arisen in the NGINX layer. Lutarevich said that not only is NGINX stable and highly available; it’s also easy to find administrators and operators who know how to use it.
“NGINX is used almost everywhere, especially on the front end side,” said Lutarevich. “The people who have some skills or experience using NGINX – there are lots of them on the market. I can go ask questions on forums, on Stack Overflow, and hire someone there as well. There are lots of candidates with NGINX experience, so that makes it very useful for us.”
Next Steps
With the move to microservices, it doesn’t matter what operating system the services are hosted on, as InfoNgen utilizes a system of service discovery based on ZooKeeper and NGINX Plus. InfoNgen is able to build new services on Linux while still running its legacy Windows applications, all in the same environment, and with the same middleware and infrastructure in place for both types of systems.
Because InfoNgen is still using Windows Server 2012 for its .NET‑based microservices, supporting containers is tricky. This means containers are not supported out of the box in this operating system, and typical orchestration systems cannot work either, because they do not support Windows Server 2012. Still, InfoNgen has rolled its own containers for this environment and gotten them working, despite the older Windows version. This is bolstered by NGINX’s ability to rewrite the headers of incoming requests, allowing them to be forwarded internally to any number of systems, while external users continue to access the same URL.
“We are under constant pressure to lower our infrastructure costs, so people are asking us, ‘Are we using our EC2 instances effectively?’ If not, then we have to find a way to put as many microservices as possible on there. The only way to do that is through some sort of container infrastructure, like Docker,” said Lutarevich.
But, because Docker does not work in Windows Server 2012, NGINX Plus is called upon to bridge the gap between old‑school infrastructure and modern application architecture. Fortunately, NGINX Plus is just as happy in a virtual machine as it is in a container, and it can route traffic to both types of systems, regardless of their underlying operating system or environment.
And, because NGINX is so versatile, it’s not affected by this custom container environment. For enterprises, customization is the most important feature for many software packages, and NGINX offers more customizability than any other load balancer or webserver on the market.
About EPAM
EPAM helps customers around the world become competitive – and stay competitive. They combine best-in-class software engineering with digital strategy and experience design, business consulting and technology innovation services. EPAM was founded in 1993 and is headquartered in Newtown, Pennsylvania, USA.
