Merge branch 'main' into record-data-octodns

2 years ago · f6f7dbf853
--- a/.gitignore
+++ b/.gitignore
@ -10,6 +10,7 @@
 coverage.xml
 dist/
 env/
 examples/migrating-to-octodns/config/
 htmlcov/
 nosetests.xml
 octodns.egg-info/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,9 +1,14 @@
 ## v1.3.0 - 2023-??-?? - ???
 ## v1.?.0 - 2023-??-?? -

 * Record.lenient property added similar to other common/standard _octodns data

 ## v1.3.0 - 2023-11-14 - New and improved processors

 #### Noteworthy changes

 * Added octodns.__version__ to replace octodns.__VERSION__ as the former is more
  of a standard, per pep-8. __VERSION__ is deprecated and will go away in 2.x
 * Added `octodns.__version__` to replace `octodns.__VERSION__` as the former is
  more of a standard, per pep-8. `__VERSION__` is deprecated and will go away
  in 2.x
 * Fixed issues with handling of chunking large TXT values for providers that use
  the in-built `rrs` method
 * Removed code that included sha in module version number when installing from
@ -25,7 +30,7 @@
  horizon)
 * ExcludeRootNsChanges processor that will error (or warn) if plan includes a
  change to root NS records
 * Include the octodns special section info in Record __repr__, makes it easier
 * Include the octodns special section info in `Record.__repr__`, makes it easier
  to debug things with providers that have special functionality configured
  there.
 * Most processor.filter processors now support an include_target flag that can
--- a/examples/README.md
+++ b/examples/README.md
@ -0,0 +1,35 @@
 ### This is all a WIP

 This is a WIP and will definitely be rough around the edges, but in the spirit
 of not letting perfect get in the way of good enough, or really existing at
 all. It's being uploaded as a starting point. PRs/suggestions welcome as are
 ideas for other subjects to cover.

 ### Examples

 * Getting started with a [basic octoDNS configuration](basic/) - new to octoDNS
  this is the place to start. It'll walk you through the main pieces of DNS IaC
  with octoDNS including the process of planning and applying changes.
 * [Migrating to octoDNS](migrating-to-octodns/) - have an existing DNS setup
  you'd like to bring into octoDNS check this example out right after
  [basic](basic/). It'll walk you through the steps of using `octodns-dump` to
  pull the existing data out of your provider into matching YAML config files on
  disk.

 ### Running PowerDNS

 If you'd like to play around with running the examples in this directory
 interactively you'll need a target for pushing data to.
 [octodns-powerdns](https://github.com/octodns/octodns-powerdns) is the best
 stand-alone option for this and the examples directory makes extensive use of
 it. There is a [docker-compose.yml](docker-compose.yml) file that should get a
 fully functional copy of PowerDNS backed my MySQL with the API enabled along
 with other relivant functionality. For any of the examples that refer to the
 local PowerDNS instance the following instructions below should get it up and
 running.

 1. If you haven't already [install docker compose](https://docs.docker.com/compose/install/)
 1. If you don't already have a copy of octoDNS checked out run `git clone https://github.com/octodns/octodns.git`
 1. In a seperate terminal window or tab
 1. cd into the examples directory `cd octodns/examples`
 1. Run docker-compose up `docker-compose up`, this will start up MySQL and PowerDNS running them in the foreground with their logs printing to the terminal
--- a/examples/basic/README.md
+++ b/examples/basic/README.md
@ -0,0 +1,254 @@
 ## Basic octoDNS Setup

 This is the starting point octoDNS config, it's pretty similar to what you
 might see for managing a set of personal domains or a small business.

 Most of the actual documentation for this example is found in the comments in
 the YAML configuration files.

 * [config/octodns.yaml](config/octodns.yaml)
 * [config/my-domain.com.yaml](config/my-domain.com.yaml)
 * [config/unused-domain.io.yaml](config/unused-domain.io.yaml)

 From here on this README focuses on the general process of running octoDNS.

 ## Checking out the code and setting up the environment

 You would not normally need to check out octoDNS itself, you instead would have
 a git repo with only your configuration files. Here we're cloning the repo only
 to get a copy of the example files.

 ```console
 $ git clone https://github.com/octodns/octodns.git
 $ cd octodns/examples/basic/
 $ python3 -mvenv env
 $ source ../env.sh
 $ source env/bin/activate
 (env) $ pip install -r requirements.txt
 ```

 Finally check out [Running PowerDNS](../README.md#running-powerdns) to get a local
 instance of PowerDNS up and going before continuing.

 ## Running octoDNS the first time

 Once you have your configuration files and octoDNS installed you're ready to
 run the sync command to get it to plan an initial set of changes.

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml
 2023-08-23T15:09:51  [4577488384] INFO  Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.1)
 2023-08-23T15:09:51  [4577488384] INFO  Manager _config_executor: max_workers=1
 2023-08-23T15:09:51  [4577488384] INFO  Manager _config_include_meta: include_meta=False
 2023-08-23T15:09:51  [4577488384] INFO  Manager _config_auto_arpa: auto_arpa=False
 2023-08-23T15:09:51  [4577488384] INFO  Manager __init__: global_processors=[]
 2023-08-23T15:09:51  [4577488384] INFO  Manager __init__: provider=config (octodns.provider.yaml 1.0.1)
 2023-08-23T15:09:51  [4577488384] INFO  SpfSource[no-mail] __init__: id=no-mail, a_records=[], mx_records=[], ip4_addresses=[], ip6_addresses=[], includes=[], exists=[], soft_fail=False, merging_enabled=False, ttl=3600
 2023-08-23T15:09:51  [4577488384] INFO  Manager __init__: provider=no-mail (octodns_spf 0.0.2)
 2023-08-23T15:09:51  [4577488384] INFO  Manager __init__: provider=yaml (octodns.provider.yaml 1.0.1)
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync: eligible_zones=[], eligible_targets=[], dry_run=True, force=False, plan_output_fh=<stdout>
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   sources=['config', 'no-mail']
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   dynamic zone=*, sources=[YamlProvider, SpfSource]
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:      adding dynamic zone=my-domain.com.
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:      adding dynamic zone=unused-domain.io.
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   zone=my-domain.com.
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   sources=['config', 'no-mail']
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   targets=['yaml']
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   zone=unused-domain.io.
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   sources=['config', 'no-mail']
 2023-08-23T15:09:51  [4577488384] INFO  Manager sync:   targets=['yaml']
 2023-08-23T15:09:51  [4577488384] INFO  YamlProvider[config] populate:   found 9 records, exists=False
 2023-08-23T15:09:51  [4577488384] INFO  SpfSource[no-mail] populate:   found 0 records, exists=False
 2023-08-23T15:09:51  [4577488384] INFO  PowerDnsProvider[powerdns] plan: desired=my-domain.com.
 2023-08-23T15:09:51  [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com.
 2023-08-23T15:09:51  [4577488384] INFO  PowerDnsProvider[powerdns] plan:   Creates=9, Updates=0, Deletes=0, Existing Records=0
 2023-08-23T15:09:51  [4577488384] INFO  YamlProvider[config] populate:   found 0 records, exists=False
 2023-08-23T15:09:51  [4577488384] INFO  SpfSource[no-mail] populate:   found 1 records, exists=False
 2023-08-23T15:09:51  [4577488384] INFO  PowerDnsProvider[powerdns] plan: desired=unused-domain.io.
 2023-08-23T15:09:51  [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-domain.io.
 2023-08-23T15:09:51  [4577488384] INFO  PowerDnsProvider[powerdns] plan:   Creates=1, Updates=0, Deletes=0, Existing Records=0
 2023-08-23T15:09:51  [4577488384] INFO  Plan
 ********************************************************************************
 * unused-domain.io.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<unused-domain.io.>
 *   Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
 *   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************
 * my-domain.com.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<my-domain.com.>
 *   Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
 *   Create <TxtRecord TXT 3600, my-domain.com., ['some-verification=3becb991-932f-4433-a280-9df6f39b6194', 'v=spf1 -all', 'z-other-thing=this proves i have control over this domain']> (config)
 *   Create <ARecord A 3600, *.my-domain.com., ['203.0.113.45']> (config)
 *   Create <AaaaRecord AAAA 3600, *.my-domain.com., ['2001:db8::46']> (config)
 *   Create <TxtRecord TXT 3600, nadcbiqkbgq._companyname.my-domain.com., ['a-different-proof-of-ownership']> (config)
 *   Create <CnameRecord CNAME 3600, pointer.my-domain.com., look.over-here.net.> (config)
 *   Create <ARecord A 3600, www.my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, www.my-domain.com., ['2001:db8::44']> (config)
 *   Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************
 ```

 ### The log output

 It's always a good idea to scan over the logging output of an octoDNS run. Most
 of it is informational, telling you what has been configured and providing a
 big picture idea of what's happening while planning. You will sometimes see
 WARNINGS, these are generally telling you that there's something you should
 think about or look into.

 In the run above there was a warning about root/APEX NS records being supported
 by the provider, YamlProvider, and not present in the config. Not all providers
 support managing the root NS records, some just hard code their own name
 servers.

 ```console
 2023-08-23T15:09:51  [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com.
 ```

 ### The plan output

 ```console
 ********************************************************************************
 * unused-domain.io.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<unused-domain.io.>
 *   Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
 *   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************
 * my-domain.com.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<my-domain.com.>
 *   Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
 *   Create <TxtRecord TXT 3600, my-domain.com., ['some-verification=3becb991-932f-4433-a280-9df6f39b6194', 'v=spf1 -all', 'z-other-thing=this proves i have control over this domain']> (config)
 *   Create <ARecord A 3600, *.my-domain.com., ['203.0.113.45']> (config)
 *   Create <AaaaRecord AAAA 3600, *.my-domain.com., ['2001:db8::46']> (config)
 *   Create <TxtRecord TXT 3600, nadcbiqkbgq._companyname.my-domain.com., ['a-different-proof-of-ownership']> (config)
 *   Create <CnameRecord CNAME 3600, pointer.my-domain.com., look.over-here.net.> (config)
 *   Create <ARecord A 3600, www.my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, www.my-domain.com., ['2001:db8::44']> (config)
 *   Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************
 ```

 ### Applying the changes

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml --doit
 ...
 ********************************************************************************
 * unused-domain.io.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<unused-domain.io.>
 *   Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
 *   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************
 * my-domain.com.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<my-domain.com.>
 *   Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
 *   Create <TxtRecord TXT 3600, my-domain.com., ['some-verification=3becb991-932f-4433-a280-9df6f39b6194', 'v=spf1 -all', 'z-other-thing=this proves i have control over this domain']> (config)
 *   Create <ARecord A 3600, *.my-domain.com., ['203.0.113.45']> (config)
 *   Create <AaaaRecord AAAA 3600, *.my-domain.com., ['2001:db8::46']> (config)
 *   Create <TxtRecord TXT 3600, nadcbiqkbgq._companyname.my-domain.com., ['a-different-proof-of-ownership']> (config)
 *   Create <CnameRecord CNAME 3600, pointer.my-domain.com., look.over-here.net.> (config)
 *   Create <ARecord A 3600, www.my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
 *   Create <AaaaRecord AAAA 3600, www.my-domain.com., ['2001:db8::44']> (config)
 *   Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0
 ********************************************************************************

 2023-08-23T15:17:00  [4671815168] INFO  PowerDnsProvider[powerdns] apply: making 1 changes to unused-domain.io.
 2023-08-23T15:17:00  [4671815168] INFO  PowerDnsProvider[powerdns] apply: making 9 changes to my-domain.com.
 2023-08-23T15:17:00  [4671815168] INFO  Manager sync:   10 total changes
 ```

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml
 ********************************************************************************
 No changes were planned
 ********************************************************************************
 ```

 That's it. You're now managing your DNS with octoDNS.

 ## Making a change

 At some point down the road you need to make a change, maybe one of your web
 boxes was replaced and there's a new IP address. The first step is to open up
 the zone's YAML file and modify the YAML, removing 203.0.113.42 and adding
 203.0.113.44.

 ## Running octoDNS to see the plan

 Just like during the first run section above we'll first run octoDNS to see what changes it would make.
 We'd skim the log lines again looking for unexpected WARNINGS and then take a look at the changes.

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml
 ...
 ********************************************************************************
 * my-domain.com.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<my-domain.com.>
 *   Update
 *     <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
 *     <ARecord A 3600, my-domain.com., ['203.0.113.43', '203.0.113.44']> (config)
 *   Update
 *     <ARecord A 3600, www.my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
 *     <ARecord A 3600, www.my-domain.com., ['203.0.113.43', '203.0.113.44']> (config)
 *   Summary: Creates=0, Updates=2, Deletes=0, Existing Records=9
 ********************************************************************************
 ```

 Since we have used YAML anchors to share the values across both the root and
 www A's we see that octoDNS will be making changes to both those records.

 If there were unexpected things here we'd need to investigate what's changed.
 Maybe there were other alterations in the YAML that hadn't been applied yet or
 someone modified the records through another means.

 Here we only see the expected changes so we're good to move forward with
 applying them.

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml --doit
 ...
 ********************************************************************************
 * my-domain.com.
 ********************************************************************************
 * powerdns (PowerDnsProvider)
 *   Create Zone<my-domain.com.>
 *   Update
 *     <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
 *     <ARecord A 3600, my-domain.com., ['203.0.113.43', '203.0.113.44']> (config)
 *   Update
 *     <ARecord A 3600, www.my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
 *     <ARecord A 3600, www.my-domain.com., ['203.0.113.43', '203.0.113.44']> (config)
 *   Summary: Creates=0, Updates=2, Deletes=0, Existing Records=9
 ********************************************************************************
 ```

 If we want we can run another plan to make sure there are no further pending changes.

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml
 ********************************************************************************
 No changes were planned
 ********************************************************************************
 ```

 ## What's Next

 * Check out [migrating to octoDNS](../migrating-to-octodns) for an example of how to create zone configuration YAML files from your existing provider's configuration
 * For a complete list check out the [Examples Directory](../)
--- a/examples/basic/config/my-domain.com.yaml
+++ b/examples/basic/config/my-domain.com.yaml
@ -0,0 +1,58 @@
 ---
 # These are root/APEX records, things that live at the top-level of the zone.
 # There are multiple records so there's a YAML list, each element in that list
 # defines its own record.
 '':
  # We first have an A record, mapping A my-domain.com to two IPv4 addresses, we
  # also use a YAML anchor to effectively store the values for later use.
  - type: A
    values: &WEB_A_VALUES
      - 203.0.113.42
      - 203.0.113.43
  # Similar to the A above, this is a AAAA for my-domain.com with a single IPv6
  # addresse.
  - type: AAAA
    values: &WEB_AAAA_VALUES
      - 2001:DB8::44
  # Finally it's common to have to prove ownership of a domain with a TXT value
  # in an APEX record. The following are a couple made up examples of how that
  # would be done. As you'll see when you plan things The SpfSource will merge
  # its SPF value in with the values defined here
  - type: TXT
    values:
      - some-verification=3becb991-932f-4433-a280-9df6f39b6194
      - z-other-thing=this proves i have control over this domain
 # This is a wildcard record, any A or AAAA query not matching a more specific
 # answer defined in this file will match this name and receive the associated
 # value in response.
 '*':
  - type: A
    # Note here we have a single value rather than values and use `value`.
    value: 203.0.113.45
  - type: AAAA
    value: 2001:DB8::46
 # Note that the records in this zone are sorted, by default YamlProvider
 # enforces alphabetical sorting of records and even the keys within record
 # data. This is an organizational best practice, but can be disabled with the
 # enforce_order parameter to YamlProvider
 NADCBiQKBgQ._companyname:
  type: TXT
  value: a-different-proof-of-ownership
 # We want www and the APEX to return the same values for A/AAAA queries. We
 # stored the values earlier in YAML anchors so we can use them now on.
 # This is a CNAME record
 pointer:
  type: CNAME
  # CNAMEs can only have a single value so they require `value` to be used. As a
  # best practice the target name must end with a `.` If you have a specific
  # case where you need to omit the `.`, or other enforced best practices, see
  # the lenience example for more information.
  value: look.over-here.net.
 # This is an example of a record with a `.` in it's name, it would result for a
 # TXT query of NADCBiQKBgQ._companyname.my-domain.com. This specific example is
 # another commonly required proof of overship record.
 www:
  - type: A
    values: *WEB_A_VALUES
  - type: AAAA
    values: *WEB_AAAA_VALUES
--- a/examples/basic/config/octodns.yaml
+++ b/examples/basic/config/octodns.yaml
@ -0,0 +1,58 @@
 ---
 providers:
  # The primary source of record data in most setup is YAML files on disk. By
  # convention it's named config.
  config:
    class: octodns.provider.yaml.YamlProvider
    # There are a number of configuration options available to customize the
    # behavior of the provider, but the defaults are generally suitable and the
    # only required value is the name of the directory where the files can be
    # found. Convention is the same config directory in which this file lives.
    directory: ./config

  # Having a no mail SPF record is considered best practice for any domains
  # that you do not plan to use for email. The octodns-spf module provides
  # SpfSource which by default adds the SPF value `v=spf1 -all` to accomplish
  # this. If you have one or more domains that involve mail check out the email
  # example after you're done here.
  no-mail:
    class: octodns_spf.SpfSource

  # PowerDNS serves as a good option for a locally runnable standalone provider
  # to use in our examples. See the top-level example README for information on
  # how to run it and the documentation for octodns-powerdns for documentation
  # about the details of the config below.
  powerdns:
    class: octodns_powerdns.PowerDnsProvider
    host: 127.0.0.1
    port: 8081
    api_key: env/POWERDNS_API_KEY
    timeout: 10

  # If you wish to push your record data to multiple providers you'd define the
  # other here and include them in the targets list down below. See the
  # multiple-provider example for more details.

 zones:

  # This configuration is using dynamic zone config. Rather than having to
  # explicitly list out all of the zones here with their coresponding
  # configuration there's a single wildcard entry. Here the `config` source
  # will look for zone files defined in its configured directory, the files
  # alongside this one. For more details see the dynamic-zone-config example.

  '*':
    # This is the place(s) we look for the records in each zone
    sources:
      # The conig provider will be firs in line to provide records for the
      # zones, and since we're using dynamic zone config it will also be
      # responsible for defining what zones exist.
      - config
      # After config has added its records the we'll add an no email SPF value.
      # It'll create the APEX TXT record if necessary. Again see the email
      # example for more details.
      - no-mail
    # This is the place(s) we push the record data to when applying changes
    targets:
      # Here we're pushing things to our local PowerDNS instance
      - powerdns
--- a/examples/basic/config/unused-domain.io.yaml
+++ b/examples/basic/config/unused-domain.io.yaml
@ -0,0 +1,2 @@
 ---
 # Nothing in here, just an empty domain b/c it's not actually used for anything
--- a/examples/basic/requirements.txt
+++ b/examples/basic/requirements.txt
@ -0,0 +1,3 @@
 octodns>=1.0.1
 octodns_powerdns>=0.0.5
 octodns_spf>=0.0.2
--- a/examples/basic/target/my-domain.com.yaml
+++ b/examples/basic/target/my-domain.com.yaml
@ -0,0 +1,31 @@
 ---
 ? ''
 : - type: A
    values:
    - 203.0.113.42
    - 203.0.113.43
  - type: AAAA
    value: 2001:db8::44
  - type: TXT
    values:
    - some-verification=3becb991-932f-4433-a280-9df6f39b6194
    - v=spf1 -all
    - z-other-thing=this proves i have control over this domain
 '*':
 - type: A
  value: 203.0.113.45
 - type: AAAA
  value: 2001:db8::46
 nadcbiqkbgq._companyname:
  type: TXT
  value: a-different-proof-of-ownership
 pointer:
  type: CNAME
  value: look.over-here.net.
 www:
 - type: A
  values:
  - 203.0.113.42
  - 203.0.113.43
 - type: AAAA
  value: 2001:db8::44
--- a/examples/basic/target/unused-domain.io.yaml
+++ b/examples/basic/target/unused-domain.io.yaml
@ -0,0 +1,4 @@
 ---
 ? ''
 : type: TXT
  value: v=spf1 -all
--- a/examples/docker-compose.yml
+++ b/examples/docker-compose.yml
@ -0,0 +1,29 @@
 version: "3.9"  # optional since v1.27.0

 volumes:
  powerdns-db-data:
    name: powerdns-db-data

 services:
  db:
    image: mariadb:10.1
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: l3tmein
    volumes:
      - powerdns-db-data:/var/lib/mysql
  powerdns:
    image: psitrax/powerdns
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "8081:8081"
    environment:
      MYSQL_HOST: db
      MYSQL_PORT: 3306
      MYSQL_USER: root
      MYSQL_PASS: l3tmein
    depends_on:
      - db
    command: --api=yes --api-key=its@secret --loglevel=99 --webserver=yes --webserver-address=0.0.0.0 --webserver-allow-from=0.0.0.0/0 --webserver-password=its@secret --webserver-port=8081 --enable-lua-records=shared --edns-subnet-processing=yes
--- a/examples/env.sh
+++ b/examples/env.sh
@ -0,0 +1 @@
 export POWERDNS_API_KEY="its@secret"
--- a/examples/migrating-to-octodns/README.md
+++ b/examples/migrating-to-octodns/README.md
@ -0,0 +1,271 @@
 ## Migrating to octoDNS via octodns-dump

 Importing an existing DNS setup into octoDNS management is a very
 straightforward process and can generally be completed in minutes.

 Some relevant documentation for this example is in comments in the YAML
 configuration files.

 * [config/octodns.yaml](config/octodns.yaml)
 * [populate/octodns.yaml](populate/octodns.yaml)
 * [populate/my-dumpable.com.yaml](populate/my-dumpable.com.yaml)
 * [populate/unused-dumpable.com.yaml](populate/unused-dumpable.com.yaml)

 From here on this README focuses on the process of using `octodns-dump` to
 import your existing DNS data into octoDNS.

 ## Checking out the code and setting up the environment

 You would not normally need to check out octoDNS itself, you instead would have
 a git repo with only your configuration files. Here we're cloning the repo only
 to get a copy of the example files.

 ```console
 $ git clone https://github.com/octodns/octodns.git
 $ cd octodns/examples/basic/
 $ python3 -mvenv env
 $ source ../env.sh
 $ source env/bin/activate
 (env) $ pip install -r requirements.txt
 ```

 If you were instead creating a repo for your config from scratch it might look
 something like:

 ```console
 $ mkdir octodnsed
 $ cd octodnsed
 $ git init -b main
 $ mkdir config
 # Using your editor of choice create the main config file with your customized
 # version of things, examples/basic/config/octodns.yaml is a good place to
 # start.
 $ vim config/octodns.yaml
 # follow the process here and once complete
 $ git add config
 $ git commit -m "Importting existing records into octoDNS"
 $ git push -u origin main
 ```

 ## Populating some data that we can later dump

 We need zone & record data in our provider in order to have something to dump
 out into a YAML config, thus our first step is to populate PowerDNS. This is
 just to have something to work with the the actual process that begins in the
 next step. It's not something you'd normally do when migrating to octoDNS.

 Step 0 is to get a local PowerDNS instance running.
 Check out out [Running PowerDNS](../README.md#running-powerdns) for info on
 starting that up. Once you've done that run the following. You can ignore the
 output and move on to the next step.

 ```console
 (env) $ octodns-sync --config-file=populate/octodns.yaml --doit
 ```

 ## Running octodns-dump

 Once you have your configuration files and octoDNS installed you're ready to
 dump your zone configs. Here we've assumed that the provider being used supports
 `list_zones`, not all do. If you get an error to that effect see
 [dynamic-zone-config](../dynamic-zone-config) for details on how to explicitly
 list your zones in the config file.

 We first tell octodns-dump where to find our config file. We then tell it that
 we want to use the output provider defined in that file named `config` rather
 than having a default one auto-created. This would allow us to customize the
 details of the provider, e.g. things like the `default_ttl`.

 We'll then tell dump where we would like it to write our zone files, in this
 case we want it to put them into `config` provider's directory `./config`.

 The quoted * tells octodns-dump to dynamically source the list of zones and
 dump config files for all of them. If you would only like to dump a specific
 zone you can replace it with the zone name, e.g. `my-domain.com.`, making sure
 to include the trailing `.`

 The final argument passed to `octodns-dump` is the source of data you'd like to
 dump. Here that's our existing provider `powerdns`.

 Note that if you're previously run other examples you may see entries in the
 log output for those zones in addition to the ones we populated for this
 exercise. They shouldn't cause any problems and can be ignored.

 ```console
 (env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns
 2023-09-12T09:39:59  [4580544000] INFO  Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089)
 2023-09-12T09:39:59  [4580544000] INFO  Manager _config_executor: max_workers=1
 2023-09-12T09:39:59  [4580544000] INFO  Manager _config_include_meta: include_meta=False
 2023-09-12T09:39:59  [4580544000] INFO  Manager _config_auto_arpa: auto_arpa=False
 2023-09-12T09:39:59  [4580544000] INFO  Manager __init__: global_processors=[]
 2023-09-12T09:39:59  [4580544000] INFO  Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089)
 2023-09-12T09:40:00  [4580544000] INFO  Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2)
 2023-09-12T09:40:00  [4580544000] INFO  Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=False, split=False, sources=['powerdns']
 2023-09-12T09:40:00  [4580544000] INFO  Manager dump: using specified output_provider=config
 2023-09-12T09:40:00  [4580544000] INFO  Manager sync:   dynamic zone=*, sources=[PowerDnsProvider]
 2023-09-12T09:40:00  [4580544000] INFO  Manager sync:      adding dynamic zone=my-dumpable.com.
 2023-09-12T09:40:00  [4580544000] INFO  Manager sync:      adding dynamic zone=unused-domain.io.
 Traceback (most recent call last):
  File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/bin/octodns-dump", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/cmds/dump.py", line 51, in main
    manager.dump(
  File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/manager.py", line 868, in dump
    source.populate(zone, lenient=lenient)
  File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns_powerdns/__init__.py", line 433, in populate
    record = Record.new(
             ^^^^^^^^^^^
  File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/record/base.py", line 76, in new
    raise ValidationError(fqdn, reasons, context)
 octodns.record.exception.ValidationError: Invalid record "sshfp.my-dumpable.com."
  - unrecognized algorithm "42"
  - unrecognized algorithm "43"
 ```

 We've intentionally configured data in the provider without following
 (octoDNS's) best practices as it's common to run across these sorts of things
 when migrating. In this case there is a SSHFP record with invalid `algorithm`s.

 Regardless of whether or not you hit errors it is important to carefully look
 over the output of octodns-dump to ensure that it was able to make sense of and
 import everything it found. It usually can, but you may sometimes run across
 things that it can't make sense of or otherwise doesn't support.

 This is especially true if you have any advanced records configured in your
 provider. octoDNS generally cannot convert records with functionality like
 weights and/or geo-coding enabled. It'll generally import a "simple" version of
 that record and indicate it's doing so with a WARNING log message. If you hit
 this situation see [Dynamic Records](/docs/dynamic_records.md).

 ## Running octodns-dump again, now with --lenient

 Back to our example, the invalid `algorithm`s aren't a huge deal and octoDNS
 can be asked to ignore them with the `--lenient` argument.

 ```console
 (env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns --lenient
 2023-09-12T09:47:48  [4566433280] INFO  Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089)
 2023-09-12T09:47:48  [4566433280] INFO  Manager _config_executor: max_workers=1
 2023-09-12T09:47:48  [4566433280] INFO  Manager _config_include_meta: include_meta=False
 2023-09-12T09:47:48  [4566433280] INFO  Manager _config_auto_arpa: auto_arpa=False
 2023-09-12T09:47:48  [4566433280] INFO  Manager __init__: global_processors=[]
 2023-09-12T09:47:48  [4566433280] INFO  Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089)
 2023-09-12T09:47:48  [4566433280] INFO  Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2)
 2023-09-12T09:47:48  [4566433280] INFO  Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=True, split=False, sources=['powerdns']
 2023-09-12T09:47:48  [4566433280] INFO  Manager dump: using specified output_provider=config
 2023-09-12T09:47:48  [4566433280] INFO  Manager sync:   dynamic zone=*, sources=[PowerDnsProvider]
 2023-09-12T09:47:48  [4566433280] INFO  Manager sync:      adding dynamic zone=my-dumpable.com.
 2023-09-12T09:47:48  [4566433280] INFO  Manager sync:      adding dynamic zone=unused-dumpable.com.
 2023-09-12T09:47:48  [4566433280] WARNING Record Invalid record "sshfp.my-dumpable.com."
  - unrecognized algorithm "42"
  - unrecognized algorithm "43"
 2023-09-12T09:47:48  [4566433280] INFO  PowerDnsProvider[powerdns] populate:   found 8 records, exists=True
 2023-09-12T09:47:48  [4566433280] INFO  YamlProvider[config] plan: desired=my-dumpable.com.
 2023-09-12T09:47:48  [4566433280] INFO  YamlProvider[config] plan:   Creates=8, Updates=0, Deletes=0, Existing Records=0
 2023-09-12T09:47:48  [4566433280] INFO  YamlProvider[config] apply: making 8 changes to my-dumpable.com.
 2023-09-12T09:47:48  [4566433280] INFO  PowerDnsProvider[powerdns] populate:   found 1 records, exists=True
 2023-09-12T09:47:49  [4566433280] INFO  YamlProvider[config] plan: desired=unused-dumpable.com.
 2023-09-12T09:47:49  [4566433280] WARNING YamlProvider[config] root NS record supported, but no record is configured for unused-dumpable.com.
 2023-09-12T09:47:49  [4566433280] INFO  YamlProvider[config] plan:   Creates=1, Updates=0, Deletes=0, Existing Records=0
 2023-09-12T09:47:49  [4566433280] INFO  YamlProvider[config] apply: making 1 changes to unused-dumpable.com.
 ```

 With `--lenient` we now see a warning about the validation problems, but
 octoDNS does its best to continue anyway.

 ## Examining the results

 We can now take a look in the config/ directory to see the created zone files
 along side our main config.

 ```console
 (env) ls -1 config/
 my-dumpable.com.yaml
 octodns.yaml
 unused-dumpable.com.yaml
 ```

 Again note if you're run through other examples you may see more files for
 their zones in this directory. They can be ignored for the purpose of the
 example, but they will be imported and managed nonetheless.

 If you open up the my-dumpable.com.yaml file you'll note that the SSHFP
 record's values have 42 and 43 for their algorithm field.

 With those values if you tried to generate a plan with `octodns-sync` now it
 would fail with validation errors.

 When migrating it's recommended to make as few changes to your config as
 possible initially, that is you want to bring the setup as-is under octoDNS
 management. This is the safest approach, change as little as possible and
 incrementally work towards having a fully managed and compliant/best practice
 config.

 So for now we'll enable `lenient` on the SSHFP record by editing
 my-dumpable.com.yaml adding `octodns.lenient = true` as shown below. For more
 details on see [lenience](/docs/records.md#lenience).

 ```yaml
 ...
 sshpf:
  # TODO: look into bringing this record into compliance
  octodns:
    lenient: true
  type: SSHFP
  values:
  - algorithm: 42
    fingerprint: abcdef1234567890
    fingerprint_type: 1
  - algorithm: 43
    fingerprint: abcdef1234567890
    fingerprint_type: 1
 ...
 ```

 ## Generating our first plan

 We can now ask octoDNS to create a plan to see if anything would change. Since
 we've looked closely at the log output zone configuration files we have a pretty
 good idea the answer is no.

 ```console
 (env) $ octodns-sync --config-file=config/octodns.yaml
 ...
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] plan: desired=unused-dumpable.com.
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] populate:   found 1 records, exists=True
 2023-09-12T10:04:27  [4608876032] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-dumpable.com.
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] plan:   No changes
 2023-09-12T10:04:27  [4608876032] INFO  YamlProvider[config] populate:   found 5 records, exists=False
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] plan: desired=exxampled.com.
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] populate:   found 5 records, exists=True
 2023-09-12T10:04:27  [4608876032] INFO  PowerDnsProvider[powerdns] plan:   No changes
 2023-09-12T10:04:27  [4608876032] INFO  Plan
 ********************************************************************************
 No changes were planned
 ********************************************************************************
 ```

 Most of the output above is omitted, this is the final few lines showing that
 no changes need to be made to get `powerdns` to match `config`. It does have a
 warning about the lack of root NS records in `unused-dumpable.com.`, but we'll
 ignore that in this example.

 If you had unsupported record types or advanced features in use this may not be
 the case and you may see changes. It's completely safe to generate a plan,
 octoDNS won't make any of the changes listed.

 If you see things in the plan output section it's time to triple check and make
 sure they're, hopefully minimal, modifications your OK with making.

 If the changes are due to advanced functionality you'll need to step back and
 plan a careful migration over to [Dynamic Records](/docs/dynamic_records.md)
 which is beyond the scope of this example.

 ## What's Next

 So now you can commit your config and start managing you DNS with octoDNS rather
 than clicking buttons in UIs or using whatever you previous had used.

 * Check out [octoDNS basic example](../basic) for an example of how to create zone configuration YAML files from your existing provider's configuration
 * For a complete list check out the [Examples Directory](../)
--- a/examples/migrating-to-octodns/config/octodns.yaml
+++ b/examples/migrating-to-octodns/config/octodns.yaml
@ -0,0 +1,31 @@
 ---
 providers:
  # See the basic example for descriptions of the basics of this config file,
  # here we'll focus on things related to octodns-dump.
  
  # This is our config provider, it sources YAML from disk, but to start we
  # don't have any files in the ./config directory. We need to dump what's in
  # our provider, in this case a local PowerDNS setup, into this config
  # directory as a starting point.
  config:
    class: octodns.provider.yaml.YamlProvider
    directory: ./config

  # This is our existing provider, the one we want to bring under octoDNS
  # management. This example will walk through dumping the existing data in
  # this provider into the config provider's directory above. At that point we
  # should be ready to start managing things with octoDNS going forward.
  powerdns: 
    class: octodns_powerdns.PowerDnsProvider
    host: 127.0.0.1
    port: 8081
    api_key: env/POWERDNS_API_KEY
    timeout: 10

 zones:

  '*':
    sources:
      - config
    targets:
      - powerdns
--- a/examples/migrating-to-octodns/populate/my-dumpable.com.yaml
+++ b/examples/migrating-to-octodns/populate/my-dumpable.com.yaml
@ -0,0 +1,41 @@
 ---
 # Just some records so we'll have something to dump, see the basic example for
 # documentation/descriptions of what's in here.
 '':
  - type: A
    values:
      - 203.0.113.42
      - 203.0.113.43
  - type: AAAA
    values: &WEB_AAAA_VALUES
      - 2001:DB8::44
  - type: NS
    values:
      - ns1.some-provider.com.
      - ns2.some-provider.com.
      - ns3.some-provider.com.
      - ns4.some-provider.com.
  - type: TXT
    values:
      - some-verification=3becb991-932f-4433-a280-9df6f39b6194
      - z-other-thing=this proves i have control over this domain
 '*':
  - type: A
    value: 203.0.113.45
  - type: AAAA
    value: 2001:DB8::46
 sshfp:
  type: SSHFP
  values:
    # These values won't pass validation. They'll be used to walk through the
    # process of safely bringing an existing config up to octoDNS's
    # recommended best practices.
    - algorithm: 42
      fingerprint: abcdef1234567890
      fingerprint_type: 1
    - algorithm: 43
      fingerprint: abcdef1234567890
      fingerprint_type: 1
 www:
  - type: CNAME
    value: my-dumpable.com.
--- a/examples/migrating-to-octodns/populate/octodns.yaml
+++ b/examples/migrating-to-octodns/populate/octodns.yaml
@ -0,0 +1,26 @@
 ---
 providers:
  # This is just a quick and easy way to load some data into PowerDNS that we
  # can dump while working through the example. See the basic example for
  # documentation of what's in here
  config:
    class: octodns.provider.yaml.YamlProvider
    directory: ./populate

  powerdns: 
    class: octodns_powerdns.PowerDnsProvider
    host: 127.0.0.1
    port: 8081
    api_key: env/POWERDNS_API_KEY
    timeout: 10

 zones:

  '*':
    # we're disabling strict validation here so we can load up data that won't
    # meet octoDNS's best practices so that we can show
    lenient: true
    sources:
      - config
    targets:
      - powerdns
--- a/examples/migrating-to-octodns/populate/unused-dumpable.com.yaml
+++ b/examples/migrating-to-octodns/populate/unused-dumpable.com.yaml
@ -0,0 +1,5 @@
 ---
 '':
  type: TXT
  values:
    - v=spf1 -all
--- a/examples/migrating-to-octodns/requirements.txt
+++ b/examples/migrating-to-octodns/requirements.txt
@ -0,0 +1,3 @@
 octodns>=1.0.1
 octodns_powerdns>=0.0.5
 octodns_spf>=0.0.2
--- a/octodns/init.py
+++ b/octodns/init.py
@ -1,4 +1,4 @@
 'OctoDNS: DNS as code - Tools for managing DNS across multiple providers'

 # TODO: remove __VERSION__ w/2.x
 __version__ = __VERSION__ = '1.2.1'
 __version__ = __VERSION__ = '1.3.0'
--- a/octodns/processor/restrict.py
+++ b/octodns/processor/restrict.py
@ -59,7 +59,7 @@ class TtlRestrictionFilter(BaseProcessor):

    def process_source_zone(self, zone, *args, **kwargs):
        for record in zone.records:
            if record._octodns.get('lenient'):
            if record.lenient:
                continue
            if self.allowed_ttls and record.ttl not in self.allowed_ttls:
                raise RestrictionException(
--- a/octodns/processor/spf.py
+++ b/octodns/processor/spf.py
@ -137,7 +137,7 @@ class SpfDnsLookupProcessor(BaseProcessor):
            if record._type != 'TXT':
                continue

            if record._octodns.get('lenient'):
            if record.lenient:
                continue

            self._check_dns_lookups(record, record.values, 0)
--- a/octodns/record/base.py
+++ b/octodns/record/base.py
@ -235,6 +235,10 @@ class Record(EqualityTupleMixin):
        except KeyError:
            return 443

    @property
    def lenient(self):
        return self._octodns.get('lenient', False)

    def changes(self, other, target):
        # We're assuming we have the same name and type if we're being compared
        if self.ttl != other.ttl: