借助 Consul 的 DNS SRV 记录实现 NGINX Plus 的服务发现

Kunal Pariani of NGINX

Technical Solutions Architect

2016-04-22

[Editor – The solution described in this blog relies on the NGINX Plus Status and Upstream Conf modules (enabled by the status and upstream_conf directives). Those modules are replaced and deprecated by the NGINX Plus API in NGINX Plus Release 13 (R13) and later, and will not be available after NGINX Plus R15. For the solution to continue working, update the files that refer to the two deprecated modules. See also Transitioning to the New NGINX Plus API for Configuration and Monitoring on our blog.]

Our previous blog post on service discovery with Consul discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. It described how to use NGINX Plus’ dynamic configuration API to dynamically add or remove load‑balanced servers that are registered with Consul.

In this blog post, we will go over another method to dynamically configure your upstream servers in NGINX Plus using DNS SRV records obtained from Consul’s DNS interface. With this method, NGINX Plus periodically re‑resolves the service name using Consul DNS. If the list of IP addresses associated with the service has changed, NGINX Plus immediately starts load balancing across the updated group of servers.

Editor – Support for DNS SRV records was introduced in NGINX Plus R9. For an overview of all the new features in that release, see Announcing NGINX Plus R9 on our blog.

To make it easier to combine dynamic configuration of NGINX Plus upstream groups with Consul DNS, we’ve created a sample demo, consul-dns-srv-demo, with step‑by‑step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker, Docker Compose, and Homebrew, you can spin up a Docker‑based environment in which NGINX Plus load balances HTTP traffic across a group of NGINX Open Source web servers, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

• Consul – Performs service discovery and provides DNS service.
• Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
• hello – Simulates a backend server that can be scaled up or down. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page listing the client IP address, the request URI, and the web server’s hostname, IP address, port, and local time. (The diagram shows four instances of this app.)
• NGINX Plus R9 and later – Load balances the backend server apps.

The NGINX Plus container listens on the public port 80, and the built‑in NGINX Plus dashboard on port 8080. The Consul container listens on ports 8300, 8400, 8500, and 53 (the last mapped to port 8600 on the Docker host, which listens for DNS queries over both TCP and UDP). Consul advertises the IP address of the Docker host (configured with the environment variable HOST_IP before starting the containers) for intercontainer communication.

Each time a new Docker container that has exposed ports is launched, Registrator registers the associated service with Consul. When a container quits or is removed, Registrator removes it from the Consul service catalog automatically.

By setting environment variables within the containers, we can be more explicit about how to register the services with Consul. For the hello container, we set the environment variable SERVICE_80_NAME to http to override the default service name of the service object created within Registrator, which later gets proxied to the registry backend (Consul in our case). This tells Consul to return the information (IP address, port, priority, and weight) for all the hello containers identified by service name http when NGINX Plus sends a DNS SRV query using the service=http parameter discussed in the next paragraph.

The NGINX Plus configuration file app.conf included in the demo configures DNS‑based load balancing with these directives:

• The resolver directive designates the container named consul, which is the Consul DNS service listening on port 53, as the DNS server. NGINX Plus obtains the IP address of the consul container from the /etc/hosts file in the NGINX Plus container.

  The valid parameter means that NGINX Plus ignores the TTL in the records provided by Consul DNS and instead re‑resolves them every 10 seconds.

• The service parameter to the server directive, introduced in NGINX Plus R9, enables NGINX Plus to read DNS SRV records and get not only the IP addresses for service.consul from the name server, but also port numbers, weights, and priorities. Consul doesn’t currently support weights and priorities, so in the demo NGINX Plus uses just the port numbers of the backend servers registered with Consul.

resolver consul:53 valid=10s;

upstream backend {
    zone upstream_backend 64k;
    server service.consul service=http resolve;
}

server {
    # ...
    location / {
        proxy_pass http://backend;
    }
}

Another feature introduced in NGINX Plus R9 facilitates the use of SRV records – support for DNS lookups over TCP. SRV records are larger than other types of DNS records, making it more likely that the complete DNS response won’t fit in a single 512 byte UDP datagram. The DNS server flags an incomplete response as “truncated”, and NGINX Plus immediately retries the request over TCP. (This feature also makes it easy for NGINX Plus to handle the very large deployments of servers that are common in microservices applications.) Note that you must also configure Consul to set the truncated flag when appropriate, by setting the enable_truncate flag. In the demo, we do this in the consul_dns_config.json file.

Summary

Using Consul’s DNS interface to dynamically configure your upstream servers is an alternative to using the NGINX Plus’ dynamic configuration API, and is useful for organizations that frequently need to change the configuration of upstream server groups, such as those that use autoscaling. A major benefit is that you can manage the servers in the upstream group without modifying the NGINX Plus configuration directly.

Editor – Support for DNS SRV records was introduced in NGINX Plus R9. For an overview of all the new features in that release, see Announcing NGINX Plus R9 on our blog.

Try out for yourself automated configuration of NGINX Plus upstream groups using Consul DNS for service discovery:

• Download consul-dns-srv-demo from our GitHub repository
• Start your free 30-day trial today or contact us to discuss your use cases