Giter VIP home page Giter VIP logo

discodns's Introduction

discodns

Build Status

An authoritative DNS nameserver that queries an etcd database of domains and records.

Key Features

  • Full support for a variety of resource records
    • Both IPv4 (A) and IPv6 (AAAA) addresses
    • CNAME alias records
    • Delegation via NS and SOA records
    • SRV and PTR for service discovery and reverse domain lookups
  • Multiple resource records of different types per domain (where valid)
  • Support for wildcard domains
  • Support for TTLs
    • Global default on all records
    • Individual TTL values for individual records
  • Runtime and application metrics are captured regularly for monitoring (stdout or grahite)
  • Incoming query filters

Production Readyness

We've been running discodns in production for several months now, with no issue, though that is not to say it's bug free! If you find any issues, please submit a bug report or pull request.

Why did we build discodns?

When building infrastructure of sufficient complexity -- especially elastic infrastructure -- we've found it's really valuable to have a fast and flexible system for service identity and discovery. Crucially, it has to support different naming conventions and work with a wide variety of platform tooling and service software. DNS has proved itself to be capable in that role for over 25 years.

Since discodns is not a recursive resolver, nor does it implement it's own cache, you should front queries with a forwarder (BIND, for example) as seen in the diagram below.

         +-----------+   +---------+
         |           |   |         |
         |  Servers  |   |  Users  |
         |           |   |         |
         +------+----+   +----+----+
                |             |
         +------v-------------v----+
         |                         |
         |    Forwarders (BIND)    |
         |                         |
         +----+---+---------+------+
              |   |         |
+----------+  |   |         |
|          |  |   |         |
| discodns <--+   |         |
|          |      |         |
+----------+      |         |    +----------------+
                  |         |    |                |
        +---------v---+     +---->   Intertubes   |
        |             |          |                |
        |  Something  |          +----------------+
        |    Else     |
        |             |
        +-------------+

This "pluggable" DNS architecture allows us to mix a variety of tools to achieve a very flexible global discovery system throughout.

Why etcd?

We chose etcd as it's a simple and distributed k/v store. It's also commonly used (and designed) for cluster management, so can behave as the canonical point for discovering services throughout a network. Services can utilize the same etcd cluster to both publish and subscribe to changes in the domain name system, as well as other orchestration needs they may have.

Why not ZooKeeper? The etcd API is much simpler for users to use, and it uses RAFT instead of Paxos, which contributes to it being a simpler to understand and easier to manage.

Another attractive quality about etcd is the ability to continue serving (albeit stale) read queries even when a consensus cannot be reached, allowing the cluster to enter a semi-failed state where it cannot accept writes, but it will serve reads. This kind of graceful service degradation is very useful for a read-heavy system, such as DNS.

Currently, discodns has been tested against ETCD 3.1.2.

Getting Started

The discodns project is written in Go, and uses an extensive library (miekg/dns) to provide the actual implementation of the DNS protocol.

You'll need to compile from source, though a Makefile is provided to make this easier. Before starting, you'll need to ensure you have Go (1.8+) installed.

Building

It's simple enough to compile from source...

cd discodns
make

discodns uses godep to manage dependency versions. make will run godep restore, which modifies your go workspace to pin dependency versions.

If you change version dependencies (using go get -u ..pkg.., or manually bumping the git rev of the package in your go workspace), then you must run godep save and commit the changes to the Godeps dir

Running

It's as simple as launching the binary to start a DNS server listening on port 53 (tcp+udp) and accepting requests. You need to ensure you also have an etcd cluster up and running, which you can read about here.

Note: You can enable verbose logging using the -v argument

Note: Since port 53 is a privileged port, you'll need to run discodns as root. You should not do this in production.

cd discodns/build/
sudo ./bin/discodns --etcd=127.0.0.1:4001

Try it out

It's incredibly easy to see your own domains come to life, simply insert a key for your record into etcd and then you're ready to go! Here we'll insert a custom A record for discodns.net pointing to 10.1.1.1.

curl -L http://127.0.0.1:4001/v2/keys/net/discodns/.A -XPUT -d value="10.1.1.1"
{"action":"set","node":{"key":"/net/discodns/.A","value":"10.1.1.1","modifiedIndex":11,"createdIndex":11}}
$ @localhost discodns.net.
; <<>> DiG 9.8.3-P1 <<>> @localhost discodns.net.
; .. truncated ..

;; QUESTION SECTION:
;discodns.net.            IN  A

;; ANSWER SECTION:
discodns.net.     0   IN  A   10.1.1.1

Authority

If you're not familiar with the DNS specification, to behave correctly as an authoritative nameserver each domain needs to have its own SOA (stands for Start Of Authority) and NS records to assert its authority. Since discodns can support multiple authoritative domains, it's up to you to enter this SOA record for each domain you use. Here's an example of creating this record for discodns.net..

SOA

curl -L http://127.0.0.1:4001/v2/keys/net/discodns/.SOA -XPUT -d value=$'ns1.discodns.net.\tadmin.discodns.net.\t3600\t600\t86400\t10'
{"action":"set","node":{"key":"/net/discodns/.SOA","value":"...","modifiedIndex":11,"createdIndex":11}}

Let's break out the value and see what we've got.

ns1.discodns.net     << - This is the root, master nameserver for this delegated domain
admin.discodns.net   << - This is the "admin" email address, note the first segment is actually the user (`[email protected]`)
3600                 << - Time in seconds for any secondary DNS servers to cache the zone (used with `AXFR`)
600                  << - Interval in seconds for any secondary DNS servers to re-try in the event of a failed zone update
86400                << - Expiry time in seconds for any secondary DNS server to drop the zone data (too old)
10                   << - Minimum TTL that applies to all DNS entries in the zone

These are all tab-separated in the PUT request body. (The $'' is just a convenience to neatly escape tabs in bash; you could use regular bash strings, with \u0009 or %09 for the tab chars, too)

Note: If you're familiar with SOA records, you'll probably notice a value missing from above. The "Serial Number" (should be in the 3rd position) is actually filled in automatically by discodns, because it uses the current index of the etcd cluster to describe the current version of the zone. (TODO).

NS

Let's add the two NS records we need for our DNS cluster.

curl -L http://127.0.0.1:4001/v2/keys/net/discodns/.NS/ns1 -XPUT -d value=ns1.discodns.net.
{"action":"set","node":{"key":"/net/discodns/.NS/ns1","value":"...","modifiedIndex":12,"createdIndex":12}}
curl -L http://127.0.0.1:4001/v2/keys/net/discodns/.NS/ns2 -XPUT -d value=ns2.discodns.net.
{"action":"set","node":{"key":"/net/discodns/.NS/ns2","value":"...","modifiedIndex":13,"createdIndex":13}}

Don't forget to ensure you also add A records for the ns{1,2}.discodns.net domains to ensure they can resolve to IPs.

Storage

The record names are used as etcd key prefixes. They are in a reverse domain format, i.e discodns.net would equate to the key net/discodns. See the examples below;

  • discodns.net. -> A record -> [10.1.1.1, 10.1.1.2]
    • /net/discodns/.A/foo -> 10.1.1.1
    • /net/discodns/.A/bar -> 10.1.1.2

You'll notice the .A folder here on the end of the reverse domain, this signifies to the dns resolver that the values beneath are A records. You can have an infinite number of nested keys within this folder, allowing for some very interesting automation patterns. Multiple keys within this folder represent multiple records for the same dns entry. If you want to enforce only one value exists for a record type (CNAME for example) you can use a single key instead of a directory (/net/discodns/.CNAME -> foo.net).

$ dig @localhost discodns.net.
; <<>> DiG 9.8.3-P1 <<>> @localhost discodns.net.
; .. truncated ..

;; QUESTION SECTION:
;discodns.net.            IN  A

;; ANSWER SECTION:
discodns.net.     0   IN  A   10.1.1.1
discodns.net.     0   IN  A   10.1.1.2

Record Types

Only a select few of record types are supported right now. These are listed here:

  • A (ipv4)
  • AAAA (ipv6)
  • TXT
  • CNAME
  • NS
  • PTR
  • SRV

TTLs (Time To Live)

You can configure discodns with a default TTL (the default default is 300 seconds) using the --default-ttl command line option. This means every single DNS resource record returned will have a TTL of the default value, unless otherwise specified on a per-record basis.

To change the TTL of any given record, you can use the .ttl suffix. For example, to use a TTL of 60 minutes for the discodns.net. A record, the database might look like this...

  • /net/discodns/.A -> 10.1.1.1
  • /net/discodns/.A.ttl -> 18000

If you have multiple nested records, you can still use the suffix. In this example, the record identified by foo will use the default TTL, and bar will use the specified TTL of 60 minutes.

  • /net/discodns/.TXT/foo -> foo
  • /net/discodns/.TXT/bar -> bar
  • /net/discodns/.TXT/bar.ttl -> 18000

Value storage formats

All records in etcd are, of course, just strings. Most record types only require simple string values with no special considerations, except their natural constraints and types within DNS (valid IP addresses, for example)

In cases where multiple pieces of information are needed for a record, they are separated with single tab characters.

These more complex cases are:

SOA

Consists of the following tab-delimited fields in order:

  • Primary nameserver
  • 'Responsible Person' (admin email)
  • Refresh
  • Retry
  • Expire
  • Minimum TTL

See the SOA example above for more details

SRV

Consists of the following tab-delimited fields in order:

  • Priority
    • For clients wishing to choose between multiple service instances
    • 16bit unsigned int
  • Weight
    • For clients wishing to choose between multiple service instances
    • 16bit unsigned int
  • Port
    • The standard port number where the service can be found on the host
    • 16bit unsigned int
  • Target
    • a regular domain name for the host where the service can be found
    • must be resolvable to an A/AAAA record.

For more about the Priority and Weight fields, including the algorithm to use when choosing, see RFC2782.

Metrics

The discodns server will monitor a wide range of runtime and application metrics. By default these metrics are dumped to stderr every 30 seconds, but this can be configured using the -metrics argument, set to 0 to disable completely.

You can also use the -graphite arguments for shipping metrics to your own Graphite server instead.

Query Filters

In some situations, it can be useful to restrict the activities of a discodns nameserver to avoid querying etcd for certain domains or record types. For example, your network may not have support for IPv6 and therefore will never be storing any internal AAAA records, so it's a waste of effort querying etcd as they're never going to return with values.

This can be achieved with the --accept and --reject options to discodns. With these options, queries will be tested against the acceptance criteria before hitting etcd, or the internal resolver. This is a very cheap operation, and can drastically improve performance in some cases.

For example, if I only want to allow PTR lookups in the in-addr.arpa. domain space (for reverse domain queries) I can use the --accept="in-addr.arpa:PTR" argument. The nameserver is now going to reject any queries that aren't reverse lookups.

--accept="discodns.net:" # Accept any queries within the discodns.net domain
--accept="discodns.net:SRV,PTR" # Accept only PTR and SRV queries within the discodns domain
--reject="discodns.net:AAAA" # Reject any queries within the discodns.net domain that are for IPv6 lookups

Contributions

All contributions are welcome and encouraged! Please feel free to open a pull request no matter how large or small.

The project is licensed under MIT.

discodns's People

Contributors

oli-hall avatar splack avatar tarnfeld avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

discodns's Issues

When searching for an SOA record, query from the top down

Given most domains are several subdomains deep, when a DNS query cannot be satisfied with an answer and we look to return an authority record (SOA) it's more efficient to start looking from the top down, since in most cases the authority is close to the root.

Support for nsupdate

It'd be useful to support the nsupdate mechanism to allow users to more easily insert records into discodns. The tool seems to be widely installed, so wouldn't require any custom software and is seen as a standard.

For this to work, I believe we need to support DNSSEC authentication, which is something that'd be great to support anyway. The DNS library being used here already works with DNSSEC so it shouldn't be too complex to implement. I've opened #17 to track that.

Edit: DNSSEC is not the level authentication used here, it's the TSIG shared secret. My mistake.

Support for reverse domain lookups

Some distributed applications (aka Hadoop 2/HDFS) have an option enabled by default that verifies a host's IP against a reverse domain lookup and it's configured hostname. When using discodns these features have to be disabled.

  • Support for the PTR record type

The resolvers in front of discodns should be in control of whether to forward in-addr.arpa queries onto the nameserver for various IP ranges. The nameserver (with the above change) is capable of responding to those queries if the correct etcd entries are in place.

If a malformed DNS entry is entered into the database, the errors will be swallowed

If a user enters a malformed DNS entry into the database, discodns will handle the error "gracefully" by not falling over and pretending nothing happened.

This is clearly not a good user experience, and we should help the user validate their input before inserting it into the database. There are a few things here...

  • Implement a rich command line client/API (that communicates with etcd) allowing users to manage their DNS entries.
  • Notify users of bad entries
    • One suggestion was to (when sending back SERVFAIL) include a specially formatted TXT record describing the formatting error.

Discuss...

Multi-value records are delimited with a literal "\t"

....not an actual tab character. Fail ๐Ÿ˜›. It's unlikely that this will raise issues yet, but, it's still wrong โ€“ and when it does cause issues, it's hard to see / resolve.

I came across this writing something that writes records directly to etcd in a discodns-ready format; following the README I used actual tab characters, which turns out to be wrong and doesn't work.

I think this arose out of a rushed attempt to be curl- command-line friendly โ€“ as the curl examples in the README show literal '\t'...but that's what url-encoding is for (%08)

This currently affects SOA and SRV records currently.


Either the README needs to change, to explain the oddness...or, preferably, the implementation needs to change โ€“ which will involve some backwards-compatibility issues with existing records.

Support for SRV records

We should natively support the SRV record format.

  • Do the things that are required for the stuff

Tests are failing due to etcd client API changes

Travis builds are downloading the latest version of the go-etcd package, which targets etcd 2.0.x, but the build env and some of our tests assume etcd 0.4.6. This is the cause of the current travis build failures.

We should either vendor in dependencies fixed to the v0.4.6 tag or start fully targeting etcd 2

We should probably vendor deps soon anyway

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.