diff --git a/.gitignore b/.gitignore index 5192821..8bf8edf 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ coverage.xml dist/ env/ +examples/migrating-to-octodns/config/ htmlcov/ nosetests.xml octodns.egg-info/ diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..f3adfa1 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,35 @@ +### This is all a WIP + +This is a WIP and will definitely be rough around the edges, but in the spirit +of not letting perfect get in the way of good enough, or really existing at +all. It's being uploaded as a starting point. PRs/suggestions welcome as are +ideas for other subjects to cover. + +### Examples + +* Getting started with a [basic octoDNS configuration](basic/) - new to octoDNS + this is the place to start. It'll walk you through the main pieces of DNS IaC + with octoDNS including the process of planning and applying changes. +* [Migrating to octoDNS](migrating-to-octodns/) - have an existing DNS setup + you'd like to bring into octoDNS check this example out right after + [basic](basic/). It'll walk you through the steps of using `octodns-dump` to + pull the existing data out of your provider into matching YAML config files on + disk. + +### Running PowerDNS + +If you'd like to play around with running the examples in this directory +interactively you'll need a target for pushing data to. +[octodns-powerdns](https://github.com/octodns/octodns-powerdns) is the best +stand-alone option for this and the examples directory makes extensive use of +it. There is a [docker-compose.yml](docker-compose.yml) file that should get a +fully functional copy of PowerDNS backed my MySQL with the API enabled along +with other relivant functionality. For any of the examples that refer to the +local PowerDNS instance the following instructions below should get it up and +running. + +1. If you haven't already [install docker compose](https://docs.docker.com/compose/install/) +1. If you don't already have a copy of octoDNS checked out run `git clone https://github.com/octodns/octodns.git` +1. In a seperate terminal window or tab +1. cd into the examples directory `cd octodns/examples` +1. Run docker-compose up `docker-compose up`, this will start up MySQL and PowerDNS running them in the foreground with their logs printing to the terminal diff --git a/examples/basic/README.md b/examples/basic/README.md new file mode 100644 index 0000000..f210881 --- /dev/null +++ b/examples/basic/README.md @@ -0,0 +1,254 @@ +## Basic octoDNS Setup + +This is the starting point octoDNS config, it's pretty similar to what you +might see for managing a set of personal domains or a small business. + +Most of the actual documentation for this example is found in the comments in +the YAML configuration files. + +* [config/octodns.yaml](config/octodns.yaml) +* [config/my-domain.com.yaml](config/my-domain.com.yaml) +* [config/unused-domain.io.yaml](config/unused-domain.io.yaml) + +From here on this README focuses on the general process of running octoDNS. + +## Checking out the code and setting up the environment + +You would not normally need to check out octoDNS itself, you instead would have +a git repo with only your configuration files. Here we're cloning the repo only +to get a copy of the example files. + +```console +$ git clone https://github.com/octodns/octodns.git +$ cd octodns/examples/basic/ +$ python3 -mvenv env +$ source ../env.sh +$ source env/bin/activate +(env) $ pip install -r requirements.txt +``` + +Finally check out [Running PowerDNS](../README.md#running-powerdns) to get a local +instance of PowerDNS up and going before continuing. + +## Running octoDNS the first time + +Once you have your configuration files and octoDNS installed you're ready to +run the sync command to get it to plan an initial set of changes. + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml +2023-08-23T15:09:51 [4577488384] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.1) +2023-08-23T15:09:51 [4577488384] INFO Manager _config_executor: max_workers=1 +2023-08-23T15:09:51 [4577488384] INFO Manager _config_include_meta: include_meta=False +2023-08-23T15:09:51 [4577488384] INFO Manager _config_auto_arpa: auto_arpa=False +2023-08-23T15:09:51 [4577488384] INFO Manager __init__: global_processors=[] +2023-08-23T15:09:51 [4577488384] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.1) +2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] __init__: id=no-mail, a_records=[], mx_records=[], ip4_addresses=[], ip6_addresses=[], includes=[], exists=[], soft_fail=False, merging_enabled=False, ttl=3600 +2023-08-23T15:09:51 [4577488384] INFO Manager __init__: provider=no-mail (octodns_spf 0.0.2) +2023-08-23T15:09:51 [4577488384] INFO Manager __init__: provider=yaml (octodns.provider.yaml 1.0.1) +2023-08-23T15:09:51 [4577488384] INFO Manager sync: eligible_zones=[], eligible_targets=[], dry_run=True, force=False, plan_output_fh= +2023-08-23T15:09:51 [4577488384] INFO Manager sync: sources=['config', 'no-mail'] +2023-08-23T15:09:51 [4577488384] INFO Manager sync: dynamic zone=*, sources=[YamlProvider, SpfSource] +2023-08-23T15:09:51 [4577488384] INFO Manager sync: adding dynamic zone=my-domain.com. +2023-08-23T15:09:51 [4577488384] INFO Manager sync: adding dynamic zone=unused-domain.io. +2023-08-23T15:09:51 [4577488384] INFO Manager sync: zone=my-domain.com. +2023-08-23T15:09:51 [4577488384] INFO Manager sync: sources=['config', 'no-mail'] +2023-08-23T15:09:51 [4577488384] INFO Manager sync: targets=['yaml'] +2023-08-23T15:09:51 [4577488384] INFO Manager sync: zone=unused-domain.io. +2023-08-23T15:09:51 [4577488384] INFO Manager sync: sources=['config', 'no-mail'] +2023-08-23T15:09:51 [4577488384] INFO Manager sync: targets=['yaml'] +2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 9 records, exists=False +2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 0 records, exists=False +2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: desired=my-domain.com. +2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com. +2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: Creates=9, Updates=0, Deletes=0, Existing Records=0 +2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 0 records, exists=False +2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 1 records, exists=False +2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: desired=unused-domain.io. +2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-domain.io. +2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: Creates=1, Updates=0, Deletes=0, Existing Records=0 +2023-08-23T15:09:51 [4577488384] INFO Plan +******************************************************************************** +* unused-domain.io. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create () +* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** +* my-domain.com. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** +``` + +### The log output + +It's always a good idea to scan over the logging output of an octoDNS run. Most +of it is informational, telling you what has been configured and providing a +big picture idea of what's happening while planning. You will sometimes see +WARNINGS, these are generally telling you that there's something you should +think about or look into. + +In the run above there was a warning about root/APEX NS records being supported +by the provider, YamlProvider, and not present in the config. Not all providers +support managing the root NS records, some just hard code their own name +servers. + +```console +2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com. +``` + +### The plan output + +```console +******************************************************************************** +* unused-domain.io. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create () +* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** +* my-domain.com. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** +``` + +### Applying the changes + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml --doit +... +******************************************************************************** +* unused-domain.io. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create () +* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** +* my-domain.com. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Create (config) +* Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0 +******************************************************************************** + +2023-08-23T15:17:00 [4671815168] INFO PowerDnsProvider[powerdns] apply: making 1 changes to unused-domain.io. +2023-08-23T15:17:00 [4671815168] INFO PowerDnsProvider[powerdns] apply: making 9 changes to my-domain.com. +2023-08-23T15:17:00 [4671815168] INFO Manager sync: 10 total changes +``` + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml +******************************************************************************** +No changes were planned +******************************************************************************** +``` + +That's it. You're now managing your DNS with octoDNS. + +## Making a change + +At some point down the road you need to make a change, maybe one of your web +boxes was replaced and there's a new IP address. The first step is to open up +the zone's YAML file and modify the YAML, removing 203.0.113.42 and adding +203.0.113.44. + +## Running octoDNS to see the plan + +Just like during the first run section above we'll first run octoDNS to see what changes it would make. +We'd skim the log lines again looking for unexpected WARNINGS and then take a look at the changes. + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml +... +******************************************************************************** +* my-domain.com. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Update +* -> +* (config) +* Update +* -> +* (config) +* Summary: Creates=0, Updates=2, Deletes=0, Existing Records=9 +******************************************************************************** +``` + +Since we have used YAML anchors to share the values across both the root and +www A's we see that octoDNS will be making changes to both those records. + +If there were unexpected things here we'd need to investigate what's changed. +Maybe there were other alterations in the YAML that hadn't been applied yet or +someone modified the records through another means. + +Here we only see the expected changes so we're good to move forward with +applying them. + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml --doit +... +******************************************************************************** +* my-domain.com. +******************************************************************************** +* powerdns (PowerDnsProvider) +* Create Zone +* Update +* -> +* (config) +* Update +* -> +* (config) +* Summary: Creates=0, Updates=2, Deletes=0, Existing Records=9 +******************************************************************************** +``` + +If we want we can run another plan to make sure there are no further pending changes. + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml +******************************************************************************** +No changes were planned +******************************************************************************** +``` + +## What's Next + +* Check out [migrating to octoDNS](../migrating-to-octodns) for an example of how to create zone configuration YAML files from your existing provider's configuration +* For a complete list check out the [Examples Directory](../) diff --git a/examples/basic/config/my-domain.com.yaml b/examples/basic/config/my-domain.com.yaml new file mode 100644 index 0000000..f447217 --- /dev/null +++ b/examples/basic/config/my-domain.com.yaml @@ -0,0 +1,58 @@ +--- +# These are root/APEX records, things that live at the top-level of the zone. +# There are multiple records so there's a YAML list, each element in that list +# defines its own record. +'': + # We first have an A record, mapping A my-domain.com to two IPv4 addresses, we + # also use a YAML anchor to effectively store the values for later use. + - type: A + values: &WEB_A_VALUES + - 203.0.113.42 + - 203.0.113.43 + # Similar to the A above, this is a AAAA for my-domain.com with a single IPv6 + # addresse. + - type: AAAA + values: &WEB_AAAA_VALUES + - 2001:DB8::44 + # Finally it's common to have to prove ownership of a domain with a TXT value + # in an APEX record. The following are a couple made up examples of how that + # would be done. As you'll see when you plan things The SpfSource will merge + # its SPF value in with the values defined here + - type: TXT + values: + - some-verification=3becb991-932f-4433-a280-9df6f39b6194 + - z-other-thing=this proves i have control over this domain +# This is a wildcard record, any A or AAAA query not matching a more specific +# answer defined in this file will match this name and receive the associated +# value in response. +'*': + - type: A + # Note here we have a single value rather than values and use `value`. + value: 203.0.113.45 + - type: AAAA + value: 2001:DB8::46 +# Note that the records in this zone are sorted, by default YamlProvider +# enforces alphabetical sorting of records and even the keys within record +# data. This is an organizational best practice, but can be disabled with the +# enforce_order parameter to YamlProvider +NADCBiQKBgQ._companyname: + type: TXT + value: a-different-proof-of-ownership +# We want www and the APEX to return the same values for A/AAAA queries. We +# stored the values earlier in YAML anchors so we can use them now on. +# This is a CNAME record +pointer: + type: CNAME + # CNAMEs can only have a single value so they require `value` to be used. As a + # best practice the target name must end with a `.` If you have a specific + # case where you need to omit the `.`, or other enforced best practices, see + # the lenience example for more information. + value: look.over-here.net. +# This is an example of a record with a `.` in it's name, it would result for a +# TXT query of NADCBiQKBgQ._companyname.my-domain.com. This specific example is +# another commonly required proof of overship record. +www: + - type: A + values: *WEB_A_VALUES + - type: AAAA + values: *WEB_AAAA_VALUES diff --git a/examples/basic/config/octodns.yaml b/examples/basic/config/octodns.yaml new file mode 100644 index 0000000..02d1148 --- /dev/null +++ b/examples/basic/config/octodns.yaml @@ -0,0 +1,58 @@ +--- +providers: + # The primary source of record data in most setup is YAML files on disk. By + # convention it's named config. + config: + class: octodns.provider.yaml.YamlProvider + # There are a number of configuration options available to customize the + # behavior of the provider, but the defaults are generally suitable and the + # only required value is the name of the directory where the files can be + # found. Convention is the same config directory in which this file lives. + directory: ./config + + # Having a no mail SPF record is considered best practice for any domains + # that you do not plan to use for email. The octodns-spf module provides + # SpfSource which by default adds the SPF value `v=spf1 -all` to accomplish + # this. If you have one or more domains that involve mail check out the email + # example after you're done here. + no-mail: + class: octodns_spf.SpfSource + + # PowerDNS serves as a good option for a locally runnable standalone provider + # to use in our examples. See the top-level example README for information on + # how to run it and the documentation for octodns-powerdns for documentation + # about the details of the config below. + powerdns: + class: octodns_powerdns.PowerDnsProvider + host: 127.0.0.1 + port: 8081 + api_key: env/POWERDNS_API_KEY + timeout: 10 + + # If you wish to push your record data to multiple providers you'd define the + # other here and include them in the targets list down below. See the + # multiple-provider example for more details. + +zones: + + # This configuration is using dynamic zone config. Rather than having to + # explicitly list out all of the zones here with their coresponding + # configuration there's a single wildcard entry. Here the `config` source + # will look for zone files defined in its configured directory, the files + # alongside this one. For more details see the dynamic-zone-config example. + + '*': + # This is the place(s) we look for the records in each zone + sources: + # The conig provider will be firs in line to provide records for the + # zones, and since we're using dynamic zone config it will also be + # responsible for defining what zones exist. + - config + # After config has added its records the we'll add an no email SPF value. + # It'll create the APEX TXT record if necessary. Again see the email + # example for more details. + - no-mail + # This is the place(s) we push the record data to when applying changes + targets: + # Here we're pushing things to our local PowerDNS instance + - powerdns diff --git a/examples/basic/config/unused-domain.io.yaml b/examples/basic/config/unused-domain.io.yaml new file mode 100644 index 0000000..d513fd6 --- /dev/null +++ b/examples/basic/config/unused-domain.io.yaml @@ -0,0 +1,2 @@ +--- +# Nothing in here, just an empty domain b/c it's not actually used for anything diff --git a/examples/basic/requirements.txt b/examples/basic/requirements.txt new file mode 100644 index 0000000..00239f7 --- /dev/null +++ b/examples/basic/requirements.txt @@ -0,0 +1,3 @@ +octodns>=1.0.1 +octodns_powerdns>=0.0.5 +octodns_spf>=0.0.2 diff --git a/examples/basic/target/my-domain.com.yaml b/examples/basic/target/my-domain.com.yaml new file mode 100644 index 0000000..e7735ae --- /dev/null +++ b/examples/basic/target/my-domain.com.yaml @@ -0,0 +1,31 @@ +--- +? '' +: - type: A + values: + - 203.0.113.42 + - 203.0.113.43 + - type: AAAA + value: 2001:db8::44 + - type: TXT + values: + - some-verification=3becb991-932f-4433-a280-9df6f39b6194 + - v=spf1 -all + - z-other-thing=this proves i have control over this domain +'*': +- type: A + value: 203.0.113.45 +- type: AAAA + value: 2001:db8::46 +nadcbiqkbgq._companyname: + type: TXT + value: a-different-proof-of-ownership +pointer: + type: CNAME + value: look.over-here.net. +www: +- type: A + values: + - 203.0.113.42 + - 203.0.113.43 +- type: AAAA + value: 2001:db8::44 diff --git a/examples/basic/target/unused-domain.io.yaml b/examples/basic/target/unused-domain.io.yaml new file mode 100644 index 0000000..00cf2f7 --- /dev/null +++ b/examples/basic/target/unused-domain.io.yaml @@ -0,0 +1,4 @@ +--- +? '' +: type: TXT + value: v=spf1 -all diff --git a/examples/docker-compose.yml b/examples/docker-compose.yml new file mode 100644 index 0000000..599c751 --- /dev/null +++ b/examples/docker-compose.yml @@ -0,0 +1,29 @@ +version: "3.9" # optional since v1.27.0 + +volumes: + powerdns-db-data: + name: powerdns-db-data + +services: + db: + image: mariadb:10.1 + ports: + - "3306:3306" + environment: + MYSQL_ROOT_PASSWORD: l3tmein + volumes: + - powerdns-db-data:/var/lib/mysql + powerdns: + image: psitrax/powerdns + ports: + - "53:53/tcp" + - "53:53/udp" + - "8081:8081" + environment: + MYSQL_HOST: db + MYSQL_PORT: 3306 + MYSQL_USER: root + MYSQL_PASS: l3tmein + depends_on: + - db + command: --api=yes --api-key=its@secret --loglevel=99 --webserver=yes --webserver-address=0.0.0.0 --webserver-allow-from=0.0.0.0/0 --webserver-password=its@secret --webserver-port=8081 --enable-lua-records=shared --edns-subnet-processing=yes diff --git a/examples/env.sh b/examples/env.sh new file mode 100644 index 0000000..7501e7c --- /dev/null +++ b/examples/env.sh @@ -0,0 +1 @@ +export POWERDNS_API_KEY="its@secret" diff --git a/examples/migrating-to-octodns/README.md b/examples/migrating-to-octodns/README.md new file mode 100644 index 0000000..15c1742 --- /dev/null +++ b/examples/migrating-to-octodns/README.md @@ -0,0 +1,271 @@ +## Migrating to octoDNS via octodns-dump + +Importing an existing DNS setup into octoDNS management is a very +straightforward process and can generally be completed in minutes. + +Some relevant documentation for this example is in comments in the YAML +configuration files. + +* [config/octodns.yaml](config/octodns.yaml) +* [populate/octodns.yaml](populate/octodns.yaml) +* [populate/my-dumpable.com.yaml](populate/my-dumpable.com.yaml) +* [populate/unused-dumpable.com.yaml](populate/unused-dumpable.com.yaml) + +From here on this README focuses on the process of using `octodns-dump` to +import your existing DNS data into octoDNS. + +## Checking out the code and setting up the environment + +You would not normally need to check out octoDNS itself, you instead would have +a git repo with only your configuration files. Here we're cloning the repo only +to get a copy of the example files. + +```console +$ git clone https://github.com/octodns/octodns.git +$ cd octodns/examples/basic/ +$ python3 -mvenv env +$ source ../env.sh +$ source env/bin/activate +(env) $ pip install -r requirements.txt +``` + +If you were instead creating a repo for your config from scratch it might look +something like: + +```console +$ mkdir octodnsed +$ cd octodnsed +$ git init -b main +$ mkdir config +# Using your editor of choice create the main config file with your customized +# version of things, examples/basic/config/octodns.yaml is a good place to +# start. +$ vim config/octodns.yaml +# follow the process here and once complete +$ git add config +$ git commit -m "Importting existing records into octoDNS" +$ git push -u origin main +``` + +## Populating some data that we can later dump + +We need zone & record data in our provider in order to have something to dump +out into a YAML config, thus our first step is to populate PowerDNS. This is +just to have something to work with the the actual process that begins in the +next step. It's not something you'd normally do when migrating to octoDNS. + +Step 0 is to get a local PowerDNS instance running. +Check out out [Running PowerDNS](../README.md#running-powerdns) for info on +starting that up. Once you've done that run the following. You can ignore the +output and move on to the next step. + +```console +(env) $ octodns-sync --config-file=populate/octodns.yaml --doit +``` + +## Running octodns-dump + +Once you have your configuration files and octoDNS installed you're ready to +dump your zone configs. Here we've assumed that the provider being used supports +`list_zones`, not all do. If you get an error to that effect see +[dynamic-zone-config](../dynamic-zone-config) for details on how to explicitly +list your zones in the config file. + +We first tell octodns-dump where to find our config file. We then tell it that +we want to use the output provider defined in that file named `config` rather +than having a default one auto-created. This would allow us to customize the +details of the provider, e.g. things like the `default_ttl`. + +We'll then tell dump where we would like it to write our zone files, in this +case we want it to put them into `config` provider's directory `./config`. + +The quoted * tells octodns-dump to dynamically source the list of zones and +dump config files for all of them. If you would only like to dump a specific +zone you can replace it with the zone name, e.g. `my-domain.com.`, making sure +to include the trailing `.` + +The final argument passed to `octodns-dump` is the source of data you'd like to +dump. Here that's our existing provider `powerdns`. + +Note that if you're previously run other examples you may see entries in the +log output for those zones in addition to the ones we populated for this +exercise. They shouldn't cause any problems and can be ignored. + +```console +(env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns +2023-09-12T09:39:59 [4580544000] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089) +2023-09-12T09:39:59 [4580544000] INFO Manager _config_executor: max_workers=1 +2023-09-12T09:39:59 [4580544000] INFO Manager _config_include_meta: include_meta=False +2023-09-12T09:39:59 [4580544000] INFO Manager _config_auto_arpa: auto_arpa=False +2023-09-12T09:39:59 [4580544000] INFO Manager __init__: global_processors=[] +2023-09-12T09:39:59 [4580544000] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089) +2023-09-12T09:40:00 [4580544000] INFO Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2) +2023-09-12T09:40:00 [4580544000] INFO Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=False, split=False, sources=['powerdns'] +2023-09-12T09:40:00 [4580544000] INFO Manager dump: using specified output_provider=config +2023-09-12T09:40:00 [4580544000] INFO Manager sync: dynamic zone=*, sources=[PowerDnsProvider] +2023-09-12T09:40:00 [4580544000] INFO Manager sync: adding dynamic zone=my-dumpable.com. +2023-09-12T09:40:00 [4580544000] INFO Manager sync: adding dynamic zone=unused-domain.io. +Traceback (most recent call last): + File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/bin/octodns-dump", line 8, in + sys.exit(main()) + ^^^^^^ + File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/cmds/dump.py", line 51, in main + manager.dump( + File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/manager.py", line 868, in dump + source.populate(zone, lenient=lenient) + File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns_powerdns/__init__.py", line 433, in populate + record = Record.new( + ^^^^^^^^^^^ + File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/record/base.py", line 76, in new + raise ValidationError(fqdn, reasons, context) +octodns.record.exception.ValidationError: Invalid record "sshfp.my-dumpable.com." + - unrecognized algorithm "42" + - unrecognized algorithm "43" +``` + +We've intentionally configured data in the provider without following +(octoDNS's) best practices as it's common to run across these sorts of things +when migrating. In this case there is a SSHFP record with invalid `algorithm`s. + +Regardless of whether or not you hit errors it is important to carefully look +over the output of octodns-dump to ensure that it was able to make sense of and +import everything it found. It usually can, but you may sometimes run across +things that it can't make sense of or otherwise doesn't support. + +This is especially true if you have any advanced records configured in your +provider. octoDNS generally cannot convert records with functionality like +weights and/or geo-coding enabled. It'll generally import a "simple" version of +that record and indicate it's doing so with a WARNING log message. If you hit +this situation see [Dynamic Records](/docs/dynamic_records.md). + +## Running octodns-dump again, now with --lenient + +Back to our example, the invalid `algorithm`s aren't a huge deal and octoDNS +can be asked to ignore them with the `--lenient` argument. + +```console +(env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns --lenient +2023-09-12T09:47:48 [4566433280] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089) +2023-09-12T09:47:48 [4566433280] INFO Manager _config_executor: max_workers=1 +2023-09-12T09:47:48 [4566433280] INFO Manager _config_include_meta: include_meta=False +2023-09-12T09:47:48 [4566433280] INFO Manager _config_auto_arpa: auto_arpa=False +2023-09-12T09:47:48 [4566433280] INFO Manager __init__: global_processors=[] +2023-09-12T09:47:48 [4566433280] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089) +2023-09-12T09:47:48 [4566433280] INFO Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2) +2023-09-12T09:47:48 [4566433280] INFO Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=True, split=False, sources=['powerdns'] +2023-09-12T09:47:48 [4566433280] INFO Manager dump: using specified output_provider=config +2023-09-12T09:47:48 [4566433280] INFO Manager sync: dynamic zone=*, sources=[PowerDnsProvider] +2023-09-12T09:47:48 [4566433280] INFO Manager sync: adding dynamic zone=my-dumpable.com. +2023-09-12T09:47:48 [4566433280] INFO Manager sync: adding dynamic zone=unused-dumpable.com. +2023-09-12T09:47:48 [4566433280] WARNING Record Invalid record "sshfp.my-dumpable.com." + - unrecognized algorithm "42" + - unrecognized algorithm "43" +2023-09-12T09:47:48 [4566433280] INFO PowerDnsProvider[powerdns] populate: found 8 records, exists=True +2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] plan: desired=my-dumpable.com. +2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] plan: Creates=8, Updates=0, Deletes=0, Existing Records=0 +2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] apply: making 8 changes to my-dumpable.com. +2023-09-12T09:47:48 [4566433280] INFO PowerDnsProvider[powerdns] populate: found 1 records, exists=True +2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] plan: desired=unused-dumpable.com. +2023-09-12T09:47:49 [4566433280] WARNING YamlProvider[config] root NS record supported, but no record is configured for unused-dumpable.com. +2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] plan: Creates=1, Updates=0, Deletes=0, Existing Records=0 +2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] apply: making 1 changes to unused-dumpable.com. +``` + +With `--lenient` we now see a warning about the validation problems, but +octoDNS does its best to continue anyway. + +## Examining the results + +We can now take a look in the config/ directory to see the created zone files +along side our main config. + +```console +(env) ls -1 config/ +my-dumpable.com.yaml +octodns.yaml +unused-dumpable.com.yaml +``` + +Again note if you're run through other examples you may see more files for +their zones in this directory. They can be ignored for the purpose of the +example, but they will be imported and managed nonetheless. + +If you open up the my-dumpable.com.yaml file you'll note that the SSHFP +record's values have 42 and 43 for their algorithm field. + +With those values if you tried to generate a plan with `octodns-sync` now it +would fail with validation errors. + +When migrating it's recommended to make as few changes to your config as +possible initially, that is you want to bring the setup as-is under octoDNS +management. This is the safest approach, change as little as possible and +incrementally work towards having a fully managed and compliant/best practice +config. + +So for now we'll enable `lenient` on the SSHFP record by editing +my-dumpable.com.yaml adding `octodns.lenient = true` as shown below. For more +details on see [lenience](/docs/records.md#lenience). + +```yaml +... +sshpf: + # TODO: look into bringing this record into compliance + octodns: + lenient: true + type: SSHFP + values: + - algorithm: 42 + fingerprint: abcdef1234567890 + fingerprint_type: 1 + - algorithm: 43 + fingerprint: abcdef1234567890 + fingerprint_type: 1 +... +``` + +## Generating our first plan + +We can now ask octoDNS to create a plan to see if anything would change. Since +we've looked closely at the log output zone configuration files we have a pretty +good idea the answer is no. + +```console +(env) $ octodns-sync --config-file=config/octodns.yaml +... +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: desired=unused-dumpable.com. +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] populate: found 1 records, exists=True +2023-09-12T10:04:27 [4608876032] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-dumpable.com. +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: No changes +2023-09-12T10:04:27 [4608876032] INFO YamlProvider[config] populate: found 5 records, exists=False +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: desired=exxampled.com. +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] populate: found 5 records, exists=True +2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: No changes +2023-09-12T10:04:27 [4608876032] INFO Plan +******************************************************************************** +No changes were planned +******************************************************************************** +``` + +Most of the output above is omitted, this is the final few lines showing that +no changes need to be made to get `powerdns` to match `config`. It does have a +warning about the lack of root NS records in `unused-dumpable.com.`, but we'll +ignore that in this example. + +If you had unsupported record types or advanced features in use this may not be +the case and you may see changes. It's completely safe to generate a plan, +octoDNS won't make any of the changes listed. + +If you see things in the plan output section it's time to triple check and make +sure they're, hopefully minimal, modifications your OK with making. + +If the changes are due to advanced functionality you'll need to step back and +plan a careful migration over to [Dynamic Records](/docs/dynamic_records.md) +which is beyond the scope of this example. + +## What's Next + +So now you can commit your config and start managing you DNS with octoDNS rather +than clicking buttons in UIs or using whatever you previous had used. + +* Check out [octoDNS basic example](../basic) for an example of how to create zone configuration YAML files from your existing provider's configuration +* For a complete list check out the [Examples Directory](../) diff --git a/examples/migrating-to-octodns/config/octodns.yaml b/examples/migrating-to-octodns/config/octodns.yaml new file mode 100644 index 0000000..0ad49c1 --- /dev/null +++ b/examples/migrating-to-octodns/config/octodns.yaml @@ -0,0 +1,31 @@ +--- +providers: + # See the basic example for descriptions of the basics of this config file, + # here we'll focus on things related to octodns-dump. + + # This is our config provider, it sources YAML from disk, but to start we + # don't have any files in the ./config directory. We need to dump what's in + # our provider, in this case a local PowerDNS setup, into this config + # directory as a starting point. + config: + class: octodns.provider.yaml.YamlProvider + directory: ./config + + # This is our existing provider, the one we want to bring under octoDNS + # management. This example will walk through dumping the existing data in + # this provider into the config provider's directory above. At that point we + # should be ready to start managing things with octoDNS going forward. + powerdns: + class: octodns_powerdns.PowerDnsProvider + host: 127.0.0.1 + port: 8081 + api_key: env/POWERDNS_API_KEY + timeout: 10 + +zones: + + '*': + sources: + - config + targets: + - powerdns diff --git a/examples/migrating-to-octodns/populate/my-dumpable.com.yaml b/examples/migrating-to-octodns/populate/my-dumpable.com.yaml new file mode 100644 index 0000000..62482be --- /dev/null +++ b/examples/migrating-to-octodns/populate/my-dumpable.com.yaml @@ -0,0 +1,41 @@ +--- +# Just some records so we'll have something to dump, see the basic example for +# documentation/descriptions of what's in here. +'': + - type: A + values: + - 203.0.113.42 + - 203.0.113.43 + - type: AAAA + values: &WEB_AAAA_VALUES + - 2001:DB8::44 + - type: NS + values: + - ns1.some-provider.com. + - ns2.some-provider.com. + - ns3.some-provider.com. + - ns4.some-provider.com. + - type: TXT + values: + - some-verification=3becb991-932f-4433-a280-9df6f39b6194 + - z-other-thing=this proves i have control over this domain +'*': + - type: A + value: 203.0.113.45 + - type: AAAA + value: 2001:DB8::46 +sshfp: + type: SSHFP + values: + # These values won't pass validation. They'll be used to walk through the + # process of safely bringing an existing config up to octoDNS's + # recommended best practices. + - algorithm: 42 + fingerprint: abcdef1234567890 + fingerprint_type: 1 + - algorithm: 43 + fingerprint: abcdef1234567890 + fingerprint_type: 1 +www: + - type: CNAME + value: my-dumpable.com. diff --git a/examples/migrating-to-octodns/populate/octodns.yaml b/examples/migrating-to-octodns/populate/octodns.yaml new file mode 100644 index 0000000..c2ce511 --- /dev/null +++ b/examples/migrating-to-octodns/populate/octodns.yaml @@ -0,0 +1,26 @@ +--- +providers: + # This is just a quick and easy way to load some data into PowerDNS that we + # can dump while working through the example. See the basic example for + # documentation of what's in here + config: + class: octodns.provider.yaml.YamlProvider + directory: ./populate + + powerdns: + class: octodns_powerdns.PowerDnsProvider + host: 127.0.0.1 + port: 8081 + api_key: env/POWERDNS_API_KEY + timeout: 10 + +zones: + + '*': + # we're disabling strict validation here so we can load up data that won't + # meet octoDNS's best practices so that we can show + lenient: true + sources: + - config + targets: + - powerdns diff --git a/examples/migrating-to-octodns/populate/unused-dumpable.com.yaml b/examples/migrating-to-octodns/populate/unused-dumpable.com.yaml new file mode 100644 index 0000000..16c9dc3 --- /dev/null +++ b/examples/migrating-to-octodns/populate/unused-dumpable.com.yaml @@ -0,0 +1,5 @@ +--- +'': + type: TXT + values: + - v=spf1 -all diff --git a/examples/migrating-to-octodns/requirements.txt b/examples/migrating-to-octodns/requirements.txt new file mode 100644 index 0000000..00239f7 --- /dev/null +++ b/examples/migrating-to-octodns/requirements.txt @@ -0,0 +1,3 @@ +octodns>=1.0.1 +octodns_powerdns>=0.0.5 +octodns_spf>=0.0.2