Merge branch 'master' into ns1-fallback-only-support

CHANGELOG.md

@@ -1,6 +1,29 @@
+## v0.9.12 - 2021-04-30 - Enough time has passed
+
+#### Noteworthy changes
+
+* Formal Python 2.7 support removed, deps and tooling were becoming
+  unmaintainable
+* octodns/octodns move, from github/octodns, more to come
+
+#### Stuff
+
+* ZoneFileSource supports specifying an extension & no files end in . to better
+  support Windows
+* LOC record type support added
+* Support for pre-release versions of PowerDNS
+* PowerDNS delete before create which allows A <-> CNAME etc.
+* Improved validation of fqdn's in ALIAS, CNAME, etc.
+* Transip support for NS records
+* Support for sending plan output to a file
+* DNSimple uses zone api rather than domain to support non-registered stuff,
+  e.g. reverse zones.
+* Support for fallback-only dynamic pools and related fixes to NS1 provider
+* Initial Hetzner provider
+
 ## v0.9.11 - 2020-11-05 - We still don't know edition
 
-#### Noteworthy changtes
+#### Noteworthy changes
 
 * ALIAS records only allowed at the root of zones - see `leient` in record docs
   for work-arounds if you really need them.

README.md

@@ -38,7 +38,7 @@ It is similar to [Netflix/denominator](https://github.com/Netflix/denominator).
 
 Running through the following commands will install the latest release of OctoDNS and set up a place for your config files to live. To determine if provider specific requirements are necessary see the [Supported providers table](#supported-providers) below.
 
-```
+```shell
 $ mkdir dns
 $ cd dns
 $ virtualenv env
@@ -48,6 +48,14 @@ $ pip install octodns <provider-specific-requirements>
 $ mkdir config
 ```
 
+#### Installing a specific commit SHA
+
+If you'd like to install a version that has not yet been released in a repetable/safe manner you can do the following. In general octoDNS is fairly stable inbetween releases thanks to the plan and apply process, but care should be taken regardless.
+
+```shell
+$ pip install -e git+https://git@github.com/github/octodns.git@<SHA>#egg=octodns
+```
+
 ### Config
 
 We start by creating a config file to tell OctoDNS about our providers and the zone(s) we want it to manage. Below we're setting up a `YamlProvider` to source records from our config files and both a `Route53Provider` and `DynProvider` to serve as the targets for those records. You can have any number of zones set up and any number of sources of data and targets for records for each. You can also have multiple config files, that make use of separate accounts and each manage a distinct set of zones. A good example of this this might be `./config/staging.yaml` & `./config/production.yaml`. We'll focus on a `config/production.yaml`.
@@ -113,7 +121,7 @@ Further information can be found in [Records Documentation](/docs/records.md).
 
 We're ready to do a dry-run with our new setup to see what changes it would make. Since we're pretending here we'll act like there are no existing records for `example.com.` in our accounts on either provider.
 
-```
+```shell
 $ octodns-sync --config-file=./config/production.yaml
 ...
 ********************************************************************************
@@ -137,7 +145,7 @@ There will be other logging information presented on the screen, but successful
 
 Now it's time to tell OctoDNS to make things happen. We'll invoke it again with the same options and add a `--doit` on the end to tell it this time we actually want it to try and make the specified changes.
 
-```
+```shell
 $ octodns-sync --config-file=./config/production.yaml --doit
 ...
 ```
@@ -168,7 +176,7 @@ If that goes smoothly, you again see the expected changes, and verify them with
 
 Very few situations will involve starting with a blank slate which is why there's tooling built in to pull existing data out of providers into a matching config file.
 
-```
+```shell
 $ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ example.com. route53
 2017-03-15T13:33:34  INFO  Manager __init__: config_file=tmp/production.yaml
 2017-03-15T13:33:34  INFO  Manager dump: zone=example.com., sources=('route53',)
@@ -197,6 +205,7 @@ The above command pulled the existing data out of Route53 and placed the results
 | [EnvVarSource](/octodns/source/envvar.py) | | TXT | No | read-only environment variable injection |
 | [GandiProvider](/octodns/provider/gandi.py) | | A, AAAA, ALIAS, CAA, CNAME, DNAME, MX, NS, PTR, SPF, SRV, SSHFP, TXT | No | |
 | [GoogleCloudProvider](/octodns/provider/googlecloud.py) | google-cloud-dns | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT  | No | |
+| [HetznerProvider](/octodns/provider/hetzner.py) | | A, AAAA, CAA, CNAME, MX, NS, SRV, TXT | No | |
 | [MythicBeastsProvider](/octodns/provider/mythicbeasts.py) | Mythic Beasts | A, AAAA, ALIAS, CNAME, MX, NS, SRV, SSHFP, CAA, TXT | No | |
 | [Ns1Provider](/octodns/provider/ns1.py) | ns1-python | All | Yes | Missing `NA` geo target |
 | [OVH](/octodns/provider/ovh.py) | ovh | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT, DKIM | No | |

octodns/__init__.py

@@ -3,4 +3,4 @@
 from __future__ import absolute_import, division, print_function, \
     unicode_literals
 
-__VERSION__ = '0.9.11'
+__VERSION__ = '0.9.12'

octodns/provider/hetzner.py

@@ -0,0 +1,339 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from collections import defaultdict
+from requests import Session
+import logging
+
+from ..record import Record
+from .base import BaseProvider
+
+
+class HetznerClientException(Exception):
+    pass
+
+
+class HetznerClientNotFound(HetznerClientException):
+
+    def __init__(self):
+        super(HetznerClientNotFound, self).__init__('Not Found')
+
+
+class HetznerClientUnauthorized(HetznerClientException):
+
+    def __init__(self):
+        super(HetznerClientUnauthorized, self).__init__('Unauthorized')
+
+
+class HetznerClient(object):
+    BASE_URL = 'https://dns.hetzner.com/api/v1'
+
+    def __init__(self, token):
+        session = Session()
+        session.headers.update({'Auth-API-Token': token})
+        self._session = session
+
+    def _do(self, method, path, params=None, data=None):
+        url = '{}{}'.format(self.BASE_URL, path)
+        response = self._session.request(method, url, params=params, json=data)
+        if response.status_code == 401:
+            raise HetznerClientUnauthorized()
+        if response.status_code == 404:
+            raise HetznerClientNotFound()
+        response.raise_for_status()
+        return response
+
+    def _do_json(self, method, path, params=None, data=None):
+        return self._do(method, path, params, data).json()
+
+    def zone_get(self, name):
+        params = {'name': name}
+        return self._do_json('GET', '/zones', params)['zones'][0]
+
+    def zone_create(self, name, ttl=None):
+        data = {'name': name, 'ttl': ttl}
+        return self._do_json('POST', '/zones', data=data)['zone']
+
+    def zone_records_get(self, zone_id):
+        params = {'zone_id': zone_id}
+        records = self._do_json('GET', '/records', params=params)['records']
+        for record in records:
+            if record['name'] == '@':
+                record['name'] = ''
+        return records
+
+    def zone_record_create(self, zone_id, name, _type, value, ttl=None):
+        data = {'name': name or '@', 'ttl': ttl, 'type': _type, 'value': value,
+                'zone_id': zone_id}
+        self._do('POST', '/records', data=data)
+
+    def zone_record_delete(self, zone_id, record_id):
+        self._do('DELETE', '/records/{}'.format(record_id))
+
+
+class HetznerProvider(BaseProvider):
+    '''
+    Hetzner DNS provider using API v1
+
+    hetzner:
+        class: octodns.provider.hetzner.HetznerProvider
+        # Your Hetzner API token (required)
+        token: foo
+    '''
+    SUPPORTS_GEO = False
+    SUPPORTS_DYNAMIC = False
+    SUPPORTS = set(('A', 'AAAA', 'CAA', 'CNAME', 'MX', 'NS', 'SRV', 'TXT'))
+
+    def __init__(self, id, token, *args, **kwargs):
+        self.log = logging.getLogger('HetznerProvider[{}]'.format(id))
+        self.log.debug('__init__: id=%s, token=***', id)
+        super(HetznerProvider, self).__init__(id, *args, **kwargs)
+        self._client = HetznerClient(token)
+
+        self._zone_records = {}
+        self._zone_metadata = {}
+        self._zone_name_to_id = {}
+
+    def _append_dot(self, value):
+        if value == '@' or value[-1] == '.':
+            return value
+        return '{}.'.format(value)
+
+    def zone_metadata(self, zone_id=None, zone_name=None):
+        if zone_name is not None:
+            if zone_name in self._zone_name_to_id:
+                zone_id = self._zone_name_to_id[zone_name]
+            else:
+                zone = self._client.zone_get(name=zone_name[:-1])
+                zone_id = zone['id']
+                self._zone_name_to_id[zone_name] = zone_id
+                self._zone_metadata[zone_id] = zone
+
+        return self._zone_metadata[zone_id]
+
+    def _record_ttl(self, record):
+        default_ttl = self.zone_metadata(zone_id=record['zone_id'])['ttl']
+        return record['ttl'] if 'ttl' in record else default_ttl
+
+    def _data_for_multiple(self, _type, records):
+        values = [record['value'].replace(';', '\\;') for record in records]
+        return {
+            'ttl': self._record_ttl(records[0]),
+            'type': _type,
+            'values': values
+        }
+
+    _data_for_A = _data_for_multiple
+    _data_for_AAAA = _data_for_multiple
+
+    def _data_for_CAA(self, _type, records):
+        values = []
+        for record in records:
+            value_without_spaces = record['value'].replace(' ', '')
+            flags = value_without_spaces[0]
+            tag = value_without_spaces[1:].split('"')[0]
+            value = record['value'].split('"')[1]
+            values.append({
+                'flags': int(flags),
+                'tag': tag,
+                'value': value,
+            })
+        return {
+            'ttl': self._record_ttl(records[0]),
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_CNAME(self, _type, records):
+        record = records[0]
+        return {
+            'ttl': self._record_ttl(record),
+            'type': _type,
+            'value': self._append_dot(record['value'])
+        }
+
+    def _data_for_MX(self, _type, records):
+        values = []
+        for record in records:
+            value_stripped_split = record['value'].strip().split(' ')
+            preference = value_stripped_split[0]
+            exchange = value_stripped_split[-1]
+            values.append({
+                'preference': int(preference),
+                'exchange': self._append_dot(exchange)
+            })
+        return {
+            'ttl': self._record_ttl(records[0]),
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_NS(self, _type, records):
+        values = []
+        for record in records:
+            values.append(self._append_dot(record['value']))
+        return {
+            'ttl': self._record_ttl(records[0]),
+            'type': _type,
+            'values': values,
+        }
+
+    def _data_for_SRV(self, _type, records):
+        values = []
+        for record in records:
+            value_stripped = record['value'].strip()
+            priority = value_stripped.split(' ')[0]
+            weight = value_stripped[len(priority):].strip().split(' ')[0]
+            target = value_stripped.split(' ')[-1]
+            port = value_stripped[:-len(target)].strip().split(' ')[-1]
+            values.append({
+                'port': int(port),
+                'priority': int(priority),
+                'target': self._append_dot(target),
+                'weight': int(weight)
+            })
+        return {
+            'ttl': self._record_ttl(records[0]),
+            'type': _type,
+            'values': values
+        }
+
+    _data_for_TXT = _data_for_multiple
+
+    def zone_records(self, zone):
+        if zone.name not in self._zone_records:
+            try:
+                zone_id = self.zone_metadata(zone_name=zone.name)['id']
+                self._zone_records[zone.name] = \
+                    self._client.zone_records_get(zone_id)
+            except HetznerClientNotFound:
+                return []
+
+        return self._zone_records[zone.name]
+
+    def populate(self, zone, target=False, lenient=False):
+        self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
+                       target, lenient)
+
+        values = defaultdict(lambda: defaultdict(list))
+        for record in self.zone_records(zone):
+            _type = record['type']
+            if _type not in self.SUPPORTS:
+                self.log.warning('populate: skipping unsupported %s record',
+                                 _type)
+                continue
+            values[record['name']][record['type']].append(record)
+
+        before = len(zone.records)
+        for name, types in values.items():
+            for _type, records in types.items():
+                data_for = getattr(self, '_data_for_{}'.format(_type))
+                record = Record.new(zone, name, data_for(_type, records),
+                                    source=self, lenient=lenient)
+                zone.add_record(record, lenient=lenient)
+
+        exists = zone.name in self._zone_records
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
+
+    def _params_for_multiple(self, record):
+        for value in record.values:
+            yield {
+                'value': value.replace('\\;', ';'),
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_A = _params_for_multiple
+    _params_for_AAAA = _params_for_multiple
+
+    def _params_for_CAA(self, record):
+        for value in record.values:
+            data = '{} {} "{}"'.format(value.flags, value.tag, value.value)
+            yield {
+                'value': data,
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _params_for_single(self, record):
+        yield {
+            'value': record.value,
+            'name': record.name,
+            'ttl': record.ttl,
+            'type': record._type
+        }
+
+    _params_for_CNAME = _params_for_single
+
+    def _params_for_MX(self, record):
+        for value in record.values:
+            data = '{} {}'.format(value.preference, value.exchange)
+            yield {
+                'value': data,
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_NS = _params_for_multiple
+
+    def _params_for_SRV(self, record):
+        for value in record.values:
+            data = '{} {} {} {}'.format(value.priority, value.weight,
+                                        value.port, value.target)
+            yield {
+                'value': data,
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_TXT = _params_for_multiple
+
+    def _apply_Create(self, zone_id, change):
+        new = change.new
+        params_for = getattr(self, '_params_for_{}'.format(new._type))
+        for params in params_for(new):
+            self._client.zone_record_create(zone_id, params['name'],
+                                            params['type'], params['value'],
+                                            params['ttl'])
+
+    def _apply_Update(self, zone_id, change):
+        # It's way simpler to delete-then-recreate than to update
+        self._apply_Delete(zone_id, change)
+        self._apply_Create(zone_id, change)
+
+    def _apply_Delete(self, zone_id, change):
+        existing = change.existing
+        zone = existing.zone
+        for record in self.zone_records(zone):
+            if existing.name == record['name'] and \
+               existing._type == record['type']:
+                self._client.zone_record_delete(zone_id, record['id'])
+
+    def _apply(self, plan):
+        desired = plan.desired
+        changes = plan.changes
+        self.log.debug('_apply: zone=%s, len(changes)=%d', desired.name,
+                       len(changes))
+
+        try:
+            zone_id = self.zone_metadata(zone_name=desired.name)['id']
+        except HetznerClientNotFound:
+            self.log.debug('_apply:   no matching zone, creating domain')
+            zone_id = self._client.zone_create(desired.name[:-1])['id']
+
+        for change in changes:
+            class_name = change.__class__.__name__
+            getattr(self, '_apply_{}'.format(class_name))(zone_id, change)
+
+        # Clear out the cache if any
+        self._zone_records.pop(desired.name, None)

tests/fixtures/hetzner-records.json

@@ -0,0 +1,223 @@
+{
+    "records": [
+        {
+            "id": "SOA",
+            "type": "SOA",
+            "name": "@",
+            "value": "hydrogen.ns.hetzner.com. dns.hetzner.com. 1 86400 10800 3600000 3600",
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "NS:sub:0",
+            "type": "NS",
+            "name": "sub",
+            "value": "6.2.3.4",
+            "ttl": 3600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "NS:sub:1",
+            "type": "NS",
+            "name": "sub",
+            "value": "7.2.3.4",
+            "ttl": 3600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "SRV:_srv._tcp:0",
+            "type": "SRV",
+            "name": "_srv._tcp",
+            "value": "10 20 30 foo-1.unit.tests",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "SRV:_srv._tcp:1",
+            "type": "SRV",
+            "name": "_srv._tcp",
+            "value": "12 20 30 foo-2.unit.tests",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "TXT:txt:0",
+            "type": "TXT",
+            "name": "txt",
+            "value": "\"Bah bah black sheep\"",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "TXT:txt:1",
+            "type": "TXT",
+            "name": "txt",
+            "value": "\"have you any wool.\"",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "A:@:0",
+            "type": "A",
+            "name": "@",
+            "value": "1.2.3.4",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "A:@:1",
+            "type": "A",
+            "name": "@",
+            "value": "1.2.3.5",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "A:www:0",
+            "type": "A",
+            "name": "www",
+            "value": "2.2.3.6",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "MX:mx:0",
+            "type": "MX",
+            "name": "mx",
+            "value": "10 smtp-4.unit.tests",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "MX:mx:1",
+            "type": "MX",
+            "name": "mx",
+            "value": "20 smtp-2.unit.tests",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "MX:mx:2",
+            "type": "MX",
+            "name": "mx",
+            "value": "30 smtp-3.unit.tests",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "MX:mx:3",
+            "type": "MX",
+            "name": "mx",
+            "value": "40 smtp-1.unit.tests",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "AAAA:aaaa:0",
+            "type": "AAAA",
+            "name": "aaaa",
+            "value": "2601:644:500:e210:62f8:1dff:feb8:947a",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "CNAME:cname:0",
+            "type": "CNAME",
+            "name": "cname",
+            "value": "unit.tests",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "A:www.sub:0",
+            "type": "A",
+            "name": "www.sub",
+            "value": "2.2.3.6",
+            "ttl": 300,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "TXT:txt:2",
+            "type": "TXT",
+            "name": "txt",
+            "value": "v=DKIM1;k=rsa;s=email;h=sha256;p=A/kinda+of/long/string+with+numb3rs",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "CAA:@:0",
+            "type": "CAA",
+            "name": "@",
+            "value": "0 issue \"ca.unit.tests\"",
+            "ttl": 3600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "CNAME:included:0",
+            "type": "CNAME",
+            "name": "included",
+            "value": "unit.tests",
+            "ttl": 3600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "SRV:_imap._tcp:0",
+            "type": "SRV",
+            "name": "_imap._tcp",
+            "value": "0 0 0 .",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        },
+        {
+            "id": "SRV:_pop3._tcp:0",
+            "type": "SRV",
+            "name": "_pop3._tcp",
+            "value": "0 0 0 .",
+            "ttl": 600,
+            "zone_id": "unit.tests",
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC"
+        }
+    ]
+}

tests/fixtures/hetzner-zones.json

@@ -0,0 +1,43 @@
+{
+    "zones": [
+        {
+            "id": "unit.tests",
+            "name": "unit.tests",
+            "ttl": 3600,
+            "registrar": "",
+            "legacy_dns_host": "",
+            "legacy_ns": [],
+            "ns": [],
+            "created": "0000-00-00 00:00:00.000 +0000 UTC",
+            "verified": "",
+            "modified": "0000-00-00 00:00:00.000 +0000 UTC",
+            "project": "",
+            "owner": "",
+            "permission": "",
+            "zone_type": {
+                "id": "",
+                "name": "",
+                "description": "",
+                "prices": null
+            },
+            "status": "verified",
+            "paused": false,
+            "is_secondary_dns": false,
+            "txt_verification": {
+                "name": "",
+                "token": ""
+            },
+            "records_count": null
+        }
+    ],
+    "meta": {
+        "pagination": {
+            "page": 1,
+            "per_page": 100,
+            "previous_page": 1,
+            "next_page": 1,
+            "last_page": 1,
+            "total_entries": 1
+        }
+    }
+}

tests/test_octodns_provider_hetzner.py

@@ -0,0 +1,341 @@
+#
+#
+#
+
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from mock import Mock, call
+from os.path import dirname, join
+from requests import HTTPError
+from requests_mock import ANY, mock as requests_mock
+from six import text_type
+from unittest import TestCase
+
+from octodns.record import Record
+from octodns.provider.hetzner import HetznerClientNotFound, \
+    HetznerProvider
+from octodns.provider.yaml import YamlProvider
+from octodns.zone import Zone
+
+
+class TestHetznerProvider(TestCase):
+    expected = Zone('unit.tests.', [])
+    source = YamlProvider('test', join(dirname(__file__), 'config'))
+    source.populate(expected)
+
+    def test_populate(self):
+        provider = HetznerProvider('test', 'token')
+
+        # Bad auth
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=401,
+                     text='{"message":"Invalid authentication credentials"}')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals('Unauthorized', text_type(ctx.exception))
+
+        # General error
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=502, text='Things caught fire')
+
+            with self.assertRaises(HTTPError) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals(502, ctx.exception.response.status_code)
+
+        # Non-existent zone doesn't populate anything
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=404,
+                     text='{"zone":{"id":"","name":"","ttl":0,"registrar":"",'
+                     '"legacy_dns_host":"","legacy_ns":null,"ns":null,'
+                     '"created":"","verified":"","modified":"","project":"",'
+                     '"owner":"","permission":"","zone_type":{"id":"",'
+                     '"name":"","description":"","prices":null},"status":"",'
+                     '"paused":false,"is_secondary_dns":false,'
+                     '"txt_verification":{"name":"","token":""},'
+                     '"records_count":0},"error":{'
+                     '"message":"zone not found","code":404}}')
+
+            zone = Zone('unit.tests.', [])
+            provider.populate(zone)
+            self.assertEquals(set(), zone.records)
+
+        # No diffs == no changes
+        with requests_mock() as mock:
+            base = provider._client.BASE_URL
+            with open('tests/fixtures/hetzner-zones.json') as fh:
+                mock.get('{}/zones'.format(base), text=fh.read())
+            with open('tests/fixtures/hetzner-records.json') as fh:
+                mock.get('{}/records'.format(base), text=fh.read())
+
+            zone = Zone('unit.tests.', [])
+            provider.populate(zone)
+            self.assertEquals(13, len(zone.records))
+            changes = self.expected.changes(zone, provider)
+            self.assertEquals(0, len(changes))
+
+        # 2nd populate makes no network calls/all from cache
+        again = Zone('unit.tests.', [])
+        provider.populate(again)
+        self.assertEquals(13, len(again.records))
+
+        # bust the cache
+        del provider._zone_records[zone.name]
+
+    def test_apply(self):
+        provider = HetznerProvider('test', 'token')
+
+        resp = Mock()
+        resp.json = Mock()
+        provider._client._do = Mock(return_value=resp)
+
+        domain_after_creation = {'zone': {
+            'id': 'unit.tests',
+            'name': 'unit.tests',
+            'ttl': 3600,
+        }}
+
+        # non-existent domain, create everything
+        resp.json.side_effect = [
+            HetznerClientNotFound,  # no zone in populate
+            HetznerClientNotFound,  # no zone during apply
+            domain_after_creation,
+        ]
+        plan = provider.plan(self.expected)
+
+        # No root NS, no ignored, no excluded, no unsupported
+        n = len(self.expected.records) - 9
+        self.assertEquals(n, len(plan.changes))
+        self.assertEquals(n, provider.apply(plan))
+        self.assertFalse(plan.exists)
+
+        provider._client._do.assert_has_calls([
+            # created the zone
+            call('POST', '/zones', None, {
+                'name': 'unit.tests',
+                'ttl': None,
+            }),
+            # created all the records with their expected data
+            call('POST', '/records', data={
+                'name': '@',
+                'ttl': 300,
+                'type': 'A',
+                'value': '1.2.3.4',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '@',
+                'ttl': 300,
+                'type': 'A',
+                'value': '1.2.3.5',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '@',
+                'ttl': 3600,
+                'type': 'CAA',
+                'value': '0 issue "ca.unit.tests"',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '_imap._tcp',
+                'ttl': 600,
+                'type': 'SRV',
+                'value': '0 0 0 .',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '_pop3._tcp',
+                'ttl': 600,
+                'type': 'SRV',
+                'value': '0 0 0 .',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '_srv._tcp',
+                'ttl': 600,
+                'type': 'SRV',
+                'value': '10 20 30 foo-1.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': '_srv._tcp',
+                'ttl': 600,
+                'type': 'SRV',
+                'value': '12 20 30 foo-2.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'aaaa',
+                'ttl': 600,
+                'type': 'AAAA',
+                'value': '2601:644:500:e210:62f8:1dff:feb8:947a',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'cname',
+                'ttl': 300,
+                'type': 'CNAME',
+                'value': 'unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'included',
+                'ttl': 3600,
+                'type': 'CNAME',
+                'value': 'unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'mx',
+                'ttl': 300,
+                'type': 'MX',
+                'value': '10 smtp-4.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'mx',
+                'ttl': 300,
+                'type': 'MX',
+                'value': '20 smtp-2.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'mx',
+                'ttl': 300,
+                'type': 'MX',
+                'value': '30 smtp-3.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'mx',
+                'ttl': 300,
+                'type': 'MX',
+                'value': '40 smtp-1.unit.tests.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'sub',
+                'ttl': 3600,
+                'type': 'NS',
+                'value': '6.2.3.4.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'sub',
+                'ttl': 3600,
+                'type': 'NS',
+                'value': '7.2.3.4.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'txt',
+                'ttl': 600,
+                'type': 'TXT',
+                'value': 'Bah bah black sheep',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'txt',
+                'ttl': 600,
+                'type': 'TXT',
+                'value': 'have you any wool.',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'txt',
+                'ttl': 600,
+                'type': 'TXT',
+                'value': 'v=DKIM1;k=rsa;s=email;h=sha256;'
+                         'p=A/kinda+of/long/string+with+numb3rs',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'www',
+                'ttl': 300,
+                'type': 'A',
+                'value': '2.2.3.6',
+                'zone_id': 'unit.tests',
+            }),
+            call('POST', '/records', data={
+                'name': 'www.sub',
+                'ttl': 300,
+                'type': 'A',
+                'value': '2.2.3.6',
+                'zone_id': 'unit.tests',
+            }),
+        ])
+        self.assertEquals(24, provider._client._do.call_count)
+
+        provider._client._do.reset_mock()
+
+        # delete 1 and update 1
+        provider._client.zone_get = Mock(return_value={
+            'id': 'unit.tests',
+            'name': 'unit.tests',
+            'ttl': 3600,
+        })
+        provider._client.zone_records_get = Mock(return_value=[
+            {
+                'type': 'A',
+                'id': 'one',
+                'created': '0000-00-00T00:00:00Z',
+                'modified': '0000-00-00T00:00:00Z',
+                'zone_id': 'unit.tests',
+                'name': 'www',
+                'value': '1.2.3.4',
+                'ttl': 300,
+            },
+            {
+                'type': 'A',
+                'id': 'two',
+                'created': '0000-00-00T00:00:00Z',
+                'modified': '0000-00-00T00:00:00Z',
+                'zone_id': 'unit.tests',
+                'name': 'www',
+                'value': '2.2.3.4',
+                'ttl': 300,
+            },
+            {
+                'type': 'A',
+                'id': 'three',
+                'created': '0000-00-00T00:00:00Z',
+                'modified': '0000-00-00T00:00:00Z',
+                'zone_id': 'unit.tests',
+                'name': 'ttl',
+                'value': '3.2.3.4',
+                'ttl': 600,
+            },
+        ])
+
+        # Domain exists, we don't care about return
+        resp.json.side_effect = ['{}']
+
+        wanted = Zone('unit.tests.', [])
+        wanted.add_record(Record.new(wanted, 'ttl', {
+            'ttl': 300,
+            'type': 'A',
+            'value': '3.2.3.4',
+        }))
+
+        plan = provider.plan(wanted)
+        self.assertTrue(plan.exists)
+        self.assertEquals(2, len(plan.changes))
+        self.assertEquals(2, provider.apply(plan))
+        # recreate for update, and delete for the 2 parts of the other
+        provider._client._do.assert_has_calls([
+            call('POST', '/records', data={
+                'name': 'ttl',
+                'ttl': 300,
+                'type': 'A',
+                'value': '3.2.3.4',
+                'zone_id': 'unit.tests',
+            }),
+            call('DELETE', '/records/one'),
+            call('DELETE', '/records/two'),
+            call('DELETE', '/records/three'),
+        ], any_order=True)