[Editor – The solution described in this blog relies on the NGINX Plus Status and Upstream Conf modules (enabled by the status
and upstream_conf
directives). Those modules are replaced and deprecated by the NGINX Plus API in NGINX Plus Release 13 (R13) and later, and will not be available after NGINX Plus R15. For the solution to continue working, update the files that refer to the two deprecated modules. See also Transitioning to the New NGINX Plus API for Configuration and Monitoring on our blog.]
Our previous blog post on service discovery with Consul discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. It described how to use NGINX Plus’ dynamic configuration API to dynamically add or remove load‑balanced servers that are registered with Consul.
In this blog post, we will go over another method to dynamically configure your upstream servers in NGINX Plus using DNS SRV
records obtained from Consul’s DNS interface. With this method, NGINX Plus periodically re‑resolves the service name using Consul DNS. If the list of IP addresses associated with the service has changed, NGINX Plus immediately starts load balancing across the updated group of servers.
Editor – Support for DNS SRV
records was introduced in NGINX Plus R9. For an overview of all the new features in that release, see Announcing NGINX Plus R9 on our blog.
Also check out our other blogs about service discovery for NGINX Plus:
- Service Discovery for NGINX Plus Using Consul APIs
- Service Discovery for NGINX Plus with etcd
- Service Discovery for NGINX Plus with ZooKeeper
To make it easier to combine dynamic configuration of NGINX Plus upstream groups with Consul DNS, we’ve created a sample demo, consul-dns-srv-demo, with step‑by‑step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker, Docker Compose, and Homebrew, you can spin up a Docker‑based environment in which NGINX Plus load balances HTTP traffic across a group of NGINX Open Source web servers, with all components running in separate Docker containers.
How the Demo Works
First we spin up a separate Docker container for each of the following apps:
- Consul – Performs service discovery and provides DNS service.
- Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
- hello – Simulates a backend server that can be scaled up or down. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page listing the client IP address, the request URI, and the web server’s hostname, IP address, port, and local time. (The diagram shows four instances of this app.)
- NGINX Plus R9 and later – Load balances the backend server apps.
The NGINX Plus container listens on the public port 80, and the built‑in NGINX Plus dashboard on port 8080. The Consul container listens on ports 8300, 8400, 8500, and 53 (the last mapped to port 8600 on the Docker host, which listens for DNS queries over both TCP and UDP). Consul advertises the IP address of the Docker host (configured with the environment variable HOST_IP
before starting the containers) for intercontainer communication.
Each time a new Docker container that has exposed ports is launched, Registrator registers the associated service with Consul. When a container quits or is removed, Registrator removes it from the Consul service catalog automatically.
By setting environment variables within the containers, we can be more explicit about how to register the services with Consul. For the hello container, we set the environment variable SERVICE_80_NAME
to http
to override the default service name of the service object created within Registrator, which later gets proxied to the registry backend (Consul in our case). This tells Consul to return the information (IP address, port, priority, and weight) for all the hello containers identified by service name http
when NGINX Plus sends a DNS SRV
query using the service=http
parameter discussed in the next paragraph.
The NGINX Plus configuration file app.conf included in the demo configures DNS‑based load balancing with these directives:
-
The
resolver
directive designates the container named consul, which is the Consul DNS service listening on port 53, as the DNS server. NGINX Plus obtains the IP address of the consul container from the /etc/hosts file in the NGINX Plus container.The
valid
parameter means that NGINX Plus ignores the TTL in the records provided by Consul DNS and instead re‑resolves them every 10 seconds. - The
service
parameter to theserver
directive, introduced in NGINX Plus R9, enables NGINX Plus to read DNSSRV
records and get not only the IP addresses for service.consul from the name server, but also port numbers, weights, and priorities. Consul doesn’t currently support weights and priorities, so in the demo NGINX Plus uses just the port numbers of the backend servers registered with Consul.
resolver consul:53 valid=10s;
upstream backend {
zone upstream_backend 64k;
server service.consul service=http resolve;
}
server {
# ...
location / {
proxy_pass http://backend;
}
}
Another feature introduced in NGINX Plus R9 facilitates the use of SRV
records – support for DNS lookups over TCP. SRV
records are larger than other types of DNS records, making it more likely that the complete DNS response won’t fit in a single 512 byte UDP datagram. The DNS server flags an incomplete response as “truncated”, and NGINX Plus immediately retries the request over TCP. (This feature also makes it easy for NGINX Plus to handle the very large deployments of servers that are common in microservices applications.) Note that you must also configure Consul to set the truncated flag when appropriate, by setting the enable_truncate
flag. In the demo, we do this in the consul_dns_config.json file.
Summary
Using Consul’s DNS interface to dynamically configure your upstream servers is an alternative to using the NGINX Plus’ dynamic configuration API, and is useful for organizations that frequently need to change the configuration of upstream server groups, such as those that use autoscaling. A major benefit is that you can manage the servers in the upstream group without modifying the NGINX Plus configuration directly.
Editor – Support for DNS SRV
records was introduced in NGINX Plus R9. For an overview of all the new features in that release, see Announcing NGINX Plus R9 on our blog.
Also check out our other blogs about service discovery for NGINX Plus:
- Service Discovery for NGINX Plus Using Consul APIs
- Service Discovery for NGINX Plus with etcd
- Service Discovery for NGINX Plus with ZooKeeper
Try out for yourself automated configuration of NGINX Plus upstream groups using Consul DNS for service discovery:
- Download consul-dns-srv-demo from our GitHub repository
- Start your free 30-day trial today or contact us to discuss your use cases