WIP: major restructuring of docs, README/doc split

.changelog/1088e071ae0d4bff859ade49561aad91.md

@@ -0,0 +1,4 @@
+---
+type: none
+---
+WIP: major restructuring of docs, README/doc split

docs/conf.py

@@ -9,7 +9,7 @@ from octodns.__init__ import __version__
 
 ### sphinx config ###
 
-project = "octodns"
+project = "octoDNS"
 copyright = "2017-present"  # noqa
 author = "Ross McFarland"
 release = __version__
@@ -69,10 +69,7 @@ todo_include_todos = True
 
 ### myst ###
 
-myst_enable_extensions = [
-    "colon_fence",
-    "deflist",
-]
+myst_enable_extensions = ["colon_fence", "deflist"]
 myst_heading_anchors = 3
 
 

docs/source/api.rst

@@ -0,0 +1,9 @@
+.. _api:
+
+Developer Interface
+===================
+
+.. module:: octodns
+
+.. autoclass:: octodns.provider.yaml.YamlProvider
+   :inherited-members:

docs/source/getting-started.rst

@@ -0,0 +1,326 @@
+Getting started
+===============
+
+Workspace
+---------
+
+Running through the following commands will install the latest release of
+octoDNS and set up a place for your config files to live. To determine if
+provider specific requirements are necessary see the providers_ below.::
+
+  $ mkdir dns
+  $ cd dns
+  $ python -m venv env
+  ...
+  $ source env/bin/activate
+  # provider-specific-requirements would be things like: octodns-route53 octodns-azure
+  $ pip install octodns <provider-specific-requirements>
+  $ mkdir config
+
+Installing a specific commit SHA
+................................
+
+If you'd like to install a version that has not yet been released in a
+repeatable/safe manner you can do the following. In general octoDNS is fairly
+stable in between releases thanks to the plan and apply process, but care
+should be taken regardless.::
+
+    $ pip install -e git+https://git@github.com/octodns/octodns.git@<SHA>#egg=octodns
+
+Config
+------
+
+We start by creating a config file to tell octoDNS about our providers and the
+zone(s) we want it to manage. Below we're setting up a ``YamlProvider`` to
+source records from our config files and both a ``Route53Provider`` and
+``DynProvider`` to serve as the targets for those records. You can have any
+number of zones set up and any number of sources of data and targets for
+records for each. You can also have multiple config files, that make use of
+separate accounts and each manage a distinct set of zones. A good example of
+this this might be ``./config/staging.yaml`` & ``./config/production.yaml``.
+We'll focus on a ``config/production.yaml``.
+
+Dynamic Zone Config
+...................
+
+octoDNS supports dynamically building the list of zones it will work with when
+source providers support it. The most common use of this would be with
+``YamlProvider`` and a single dynamic entry to in effect use the files that
+exist in the provider's directory as the source of truth. Other providers may
+support the ``list_zones`` method and be available to populate zones
+dynamically as well. This can be especially useful when using ``octodns-dump``
+to create an initial setup from an existing provider.
+
+An example config would look something like::
+
+  ---
+  providers:
+    config:
+      class: octodns.provider.yaml.YamlProvider
+      directory: ./config
+      default_ttl: 3600
+      enforce_order: True
+    ns1:
+      class: octodns_ns1.Ns1Provider
+      api_key: env/NS1_API_KEY
+    route53:
+      class: octodns_route53.Route53Provider
+      access_key_id: env/AWS_ACCESS_KEY_ID
+      secret_access_key: env/AWS_SECRET_ACCESS_KEY
+
+  zones:
+    # This is a dynamic zone config. The source(s), here `config`, will be
+    # queried for a list of zone names and each will dynamically be set up to
+    # match the dynamic entry.
+    '*':
+      sources:
+        - config
+      targets:
+        - ns1
+        - route53
+
+Static Zone Config
+..................
+
+In cases where finer grained control is desired and the configuration of
+individual zones varies ``zones`` can be an explicit list with each configured
+zone listed along with its specific setup. As exemplified below ``alias`` zones
+can be useful when two zones are exact copies of each other, with the same
+configuration and records. YAML anchors are also helpful to avoid duplication
+where zones share config, but not records.::
+
+  ---
+  manager:
+    include_meta: True
+    max_workers: 2
+
+  providers:
+    config:
+      class: octodns.provider.yaml.YamlProvider
+      directory: ./config
+      default_ttl: 3600
+      enforce_order: True
+    ns1:
+      class: octodns_ns1.Ns1Provider
+      api_key: env/NS1_API_KEY
+    route53:
+      class: octodns_route53.Route53Provider
+      access_key_id: env/AWS_ACCESS_KEY_ID
+      secret_access_key: env/AWS_SECRET_ACCESS_KEY
+
+  zones:
+    example.com.: &dual_target
+      sources:
+        - config
+      targets:
+        - ns1
+        - route53
+
+    # these have the same setup as example.com., but will have their own files
+    # in the configuration directory for records.
+    third.tv.: *dual_target
+    fourth.tv.: *dual_target
+
+    example.net.:
+      # example.net. is an exact copy of example.com., there will not be an
+      # example.net.yaml file in the config directory as `alias` includes
+      # duplicating the records of the aliased zone along with its config.
+      alias: example.com.
+
+    other.com.:
+      lenient: True
+      sources:
+        - config
+      targets:
+        - ns1
+
+General Configuration Concepts
+..............................
+
+``class`` is a special key that tells octoDNS what python class should be
+loaded.  Any other keys will be passed as configuration values to that
+provider. In general any sensitive or frequently rotated values should come
+from environmental variables. When octoDNS sees a value that starts with
+``env/`` it will look for that value in the process's environment and pass the
+result along.
+
+Further information can be found in the docstring of each source and provider
+class.
+
+The ``include_meta`` key in the ``manager`` section of the config controls the
+creation of a TXT record at the root of a zone that is managed by octoDNS. If
+set to ``True``, octoDNS will create a TXT record for the root of the zone with
+the value ``provider=<target-provider>``. If not specified, the default value for
+``include_meta`` is ``False``.
+
+The ``max_workers`` key in the ``manager`` section of the config enables threading
+to parallelize the planning portion of the sync.
+
+Quick Example Record
+....................
+
+Now that we have something to tell octoDNS about our providers & zones we need
+to tell it about our records. We'll keep it simple for now and just create a
+single ``A`` record at the top-level of the domain.
+
+``config/example.com.yaml``::
+
+  ---
+  '':
+    ttl: 60
+    type: A
+    values:
+      - 1.2.3.4
+      - 1.2.3.5
+
+Further information can be found in Records_ Documentation
+
+Noop
+----
+
+We're ready to do a dry-run with our new setup to see what changes it would
+make. Since we're pretending here we'll act like there are no existing records
+for ``example.com.`` in our accounts on either provider.::
+
+  $ octodns-sync --config-file=./config/production.yaml
+  ...
+  ********************************************************************************
+  * example.com.
+  ********************************************************************************
+  * route53 (Route53Provider)
+  *   Create <ARecord A 60, example.com., [u'1.2.3.4', '1.2.3.5']>
+  *   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
+  * dyn (DynProvider)
+  *   Create <ARecord A 60, example.com., [u'1.2.3.4', '1.2.3.5']>
+  *   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
+  ********************************************************************************
+  ...
+
+There will be other logging information presented on the screen, but successful
+runs of sync will always end with a summary like the above for any providers &
+zones with changes. If there are no changes a message saying so will be printed
+instead. Above we're creating a new zone in both providers so they show the
+same change, but that doesn't always have to be the case. If, to start, one of
+them had a different state, you would see the changes octoDNS intends to make
+to sync them up.
+
+Making changes
+--------------
+
+**WARNING**: octoDNS assumes ownership of any domain you point it to. When you
+tell it to act it will do whatever is necessary to try and match up states
+including deleting any unexpected records. Be careful when playing around with
+octoDNS. It's best to experiment with a fake zone or one without any data that
+matters until you're comfortable with the system.
+
+Now it's time to tell octoDNS to make things happen. We'll invoke it again with
+the same options and add a ``--doit`` on the end to tell it this time we
+actually want it to try and make the specified changes.::
+
+  $ octodns-sync --config-file=./config/production.yaml --doit
+  ...
+
+The output here would be the same as before with a few more log lines at the
+end as it makes the actual changes. After which the config in Route53 and Dyn
+should match what's in the yaml file.
+
+Workflow
+--------
+
+In the above case we manually ran octoDNS from the command line. That works and
+it's better than heading into the provider GUIs and making changes by clicking
+around, but octoDNS is designed to be run as part of a deploy process. The
+implementation details are well beyond the scope of this README, but here is an
+example of the workflow we use at GitHub. It follows the way `GitHub itself is
+branch deployed`_.
+
+.. _GitHub itself is branch deployed: https://githubengineering.com/deploying-branches-to-github-com/
+
+The first step is to create a PR with your changes.
+
+.. image:: assets/pr.png
+   :alt: GitHub user interface of a pull request
+
+Assuming the code tests and config validation statuses are green the next step
+is to do a noop deploy and verify that the changes octoDNS plans to make are
+the ones you expect.
+
+.. image:: assets/noop.png
+   :alt: Output of a noop deployment command
+
+After that comes a set of reviews. One from a teammate who should have full
+context on what you're trying to accomplish and visibility into the changes
+you're making to do it. The other is from a member of the team here at GitHub
+that owns DNS, mostly as a sanity check and to make sure that best practices
+are being followed. As much of that as possible is baked into
+``octodns-validate``.
+
+After the reviews it's time to branch deploy the change.
+
+.. image:: assets/deploy.png
+   :alt: Output of a deployment command
+
+If that goes smoothly, you again see the expected changes, and verify them with
+``dig`` and/or ``octodns-report`` you're good to hit the merge button. If there
+are problems you can quickly do a ``.deploy dns/main`` to go back to the
+previous state.
+
+Other Uses
+----------
+
+Syncing between providers
+.........................
+
+
+While the primary use-case is to sync a set of yaml config files up to one or
+more DNS providers, octoDNS has been built in such a way that you can easily
+source and target things arbitrarily. As a quick example the config below would
+sync ``githubtest.net.`` from Route53 to Dyn.::
+
+  ---
+  providers:
+    route53:
+      class: octodns.provider.route53.Route53Provider
+      access_key_id: env/AWS_ACCESS_KEY_ID
+      secret_access_key: env/AWS_SECRET_ACCESS_KEY
+    dyn:
+      class: octodns.provider.dyn.DynProvider
+      customer: env/DYN_CUSTOMER
+      username: env/DYN_USERNAME
+      password: env/DYN_PASSWORD
+
+  zones:
+    githubtest.net.:
+      sources:
+        - route53
+      targets:
+        - dyn
+
+Dynamic sources
+...............
+
+Internally we use custom sources to create records based on dynamic data that
+changes frequently without direct human intervention. An example of that might
+look something like the following. For hosts this mechanism is janitorial, run
+periodically, making sure the correct records exist as long as the host is
+alive and ensuring they are removed after the host is destroyed. The host
+provisioning and destruction processes do the actual work to create and destroy
+the records.::
+
+  ---
+  providers:
+    gpanel-site:
+      class: github.octodns.source.gpanel.GPanelProvider
+      host: 'gpanel.site.github.foo'
+      token: env/GPANEL_SITE_TOKEN
+    powerdns-site:
+      class: octodns.provider.powerdns.PowerDnsProvider
+      host: "internal-dns.site.github.foo"
+      api_key: env/POWERDNS_SITE_API_KEY
+
+  zones:
+    hosts.site.github.foo.:
+      sources:
+        - gpanel-site
+      targets:
+        - powerdns-site

docs/source/index.rst

@@ -0,0 +1,512 @@
+DNS as Code
+===========
+
+Tools for managing DNS across multiple providers
+------------------------------------------------
+
+In the vein of infrastructure as code octoDNS provides a set of tools &
+patterns that make it easy to manage your DNS records across multiple
+providers. The resulting config can live in a repository and be deployed just
+like the rest of your code, maintaining a clear history and using your existing
+review & workflow.
+
+The architecture is pluggable and the tooling is flexible to make it applicable
+to a wide variety of use-cases. Effort has been made to make adding new
+providers as easy as possible. In the simple case that involves writing of a
+single class and a couple hundred lines of code, most of which is translating
+between the provider's schema and octoDNS's.
+
+Documentation
+-------------
+
+.. toctree::
+   :maxdepth: 1
+
+   getting-started.rst
+   examples/basic/README.md
+   examples/migrating-to-octodns/README.md
+
+``lenient``
+...........
+
+``lenient`` mostly focuses on the details of ``Record``s and standards
+compliance.  When set to ``true`` octoDNS will allow non-compliant
+configurations & values where possible. For example CNAME values that don't end
+with a ``.``, label length restrictions, and invalid geo codes on ``dynamic``
+records. When in lenient mode octoDNS will log validation problems at
+``WARNING`` and try and continue with the configuration or source data as it
+exists. See Lenience_ for more information on the concept and how it can be
+configured.
+
+.. _Lenience: records.rst#lenience
+
+``strict_supports``
+...................
+
+``strict_supports`` is a ``Provider`` level parameter that comes into play when
+a provider has been asked to create a record that it is unable to support. The
+simplest case of this would be record type, e.g. ``SSHFP`` not being supported
+by ``AzureProvider``. If such a record is passed to an ``AzureProvider`` as a
+target the provider will take action based on the ``strict_supports``. When
+``true`` it will throw an exception saying that it's unable to create the
+record, when set to ``false`` it will log at ``WARNING`` with information about
+what it's unable to do and how it is attempting to work around it. Other
+examples of things that cannot be supported would be ``dynamic`` records on a
+provider that only supports simple or the lack of support for specific geos in
+a provider, e.g.  Route53Provider does not support ``NA-CA-*``.
+
+It is worth noting that these errors will happen during the plan phase of
+things so that problems will be visible without having to make changes.
+
+As of octoDNS 1.x ``strict_supports`` is on by default. You have the choice to
+set ``strict_supports=false`` on a per provider basis to request that things warn
+and continue in a best-effort fashion.
+
+Configuring ``strict_supports``
+*******************************
+
+The ``strict_supports`` parameter is available on all providers and can be
+configured in YAML as follows::
+
+  providers:
+    someprovider:
+      class: whatever.TheProvider
+      ...
+      strict_supports: true
+
+Automatic PTR generation
+........................
+
+octoDNS supports automatically generating PTR records from the ``A``/``AAAA``
+records it manages. For more information see the `auto-arpa`_ documentation.
+
+Providers
+---------
+
+The table below lists the providers octoDNS supports. They are maintained in
+their own repositories and released as independent modules.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Provider
+     - Module
+     - Notes
+   * - /etc/hosts
+     - `octodns_etchosts`_
+     -
+   * - `Akamai Edge DNS`_
+     - `octodns_edgedns`_
+     -
+   * - `Amazon Route 53`_
+     - `octodns_route53`_
+     -
+   * - `AutoDNS`_
+     - `octodns_autodns`_
+     -
+   * - `Azion DNS`_
+     - `octodns_azion`_
+     -
+   * - `Azure DNS`_
+     - `octodns_azure`_
+     -
+   * - `BIND, AXFR, RFC-2136`_
+     - `octodns_bind`_
+     -
+   * - `Bunny DNS`_
+     - `octodns_bunny`_
+     -
+   * - `Cloudflare DNS`_
+     - `octodns_cloudflare`_
+     -
+   * - `ClouDNS`_
+     - `octodns_cloudns`_
+     -
+   * - `Constellix`_
+     - `octodns_constellix`_
+     -
+   * - `deSEC`_
+     - `octodns_desec`_
+     -
+   * - `DigitalOcean`_
+     - `octodns_digitalocean`_
+     -
+   * - `DNS Made Easy`_
+     - `octodns_dnsmadeeasy`_
+     -
+   * - `DNSimple`_
+     - `octodns_dnsimple`_
+     -
+   * - `Dyn`_ [deprecated]
+     - `octodns_dyn`_
+     -
+   * - `easyDNS`_
+     - `octodns_easydns`_
+     -
+   * - `EdgeCenter DNS`_
+     - `octodns_edgecenter`_
+     -
+   * - `Fastly`_
+     - `Financial-Times/octodns-fastly`_
+     -
+   * - `G-Core Labs DNS`_
+     - `octodns_gcore`_
+     -
+   * - `Gandi`_
+     - `octodns_gandi`_
+     -
+   * - `Google Cloud DNS`_
+     - `octodns_googlecloud`_
+     -
+   * - `Hetzner DNS`_
+     - `octodns_hetzner`_
+     -
+   * - `Infoblox`_
+     - `asyncon/octoblox`_
+     -
+   * - `Infomaniak`_
+     - `octodns_infomaniak`_
+     -
+   * - `Lexicon`_
+     - `dns-lexicon/dns-lexicon`_
+     -
+   * - `Mythic Beasts DNS`_
+     - `octodns_mythicbeasts`_
+     -
+   * - `NetBox-DNS Plugin`_
+     - `olofvndrhr/octodns-netbox-dns`_
+     -
+   * - `NS1`_
+     - `octodns_ns1`_
+     -
+   * - `OVHcloud DNS`_
+     - `octodns_ovh`_
+     -
+   * - `Pi-hole`_
+     - `jvoss/octodns-pihole`_
+     -
+   * - `PowerDNS`_
+     - `octodns_powerdns`_
+     -
+   * - `Rackspace`_
+     - `octodns_rackspace`_
+     -
+   * - `Scaleway`_
+     - `octodns_scaleway`_
+     -
+   * - `Selectel`_
+     - `octodns_selectel`_
+     -
+   * - `SPF Value Management`_
+     - `octodns_spf`_
+     -
+   * - `TransIP`_
+     - `octodns_transip`_
+     -
+   * - `UltraDNS`_
+     - `octodns_ultra`_
+     -
+   * - `YamlProvider`_
+     - built-in
+     - Supports all record types and core functionality
+   * - Zonefile
+     - `kompetenzbolzen/octodns-custom-provider`_
+     -
+
+.. _octodns_etchosts: https://github.com/octodns/octodns-etchosts/
+.. _Akamai Edge DNS: https://www.akamai.com/products/edge-dns
+.. _octodns_edgedns: https://github.com/octodns/octodns-edgedns/
+.. _Amazon Route 53: https://aws.amazon.com/route53/
+.. _octodns_route53: https://github.com/octodns/octodns-route53
+.. _AutoDNS: https://www.internetx.com/autodns/
+.. _octodns_autodns: https://github.com/octodns/octodns-autodns
+.. _Azion DNS: https://www.azion.com/en/products/edge-dns/
+.. _octodns_azion: https://github.com/aziontech/octodns-azion/
+.. _Azure DNS: https://azure.microsoft.com/en-us/services/dns/
+.. _octodns_azure: https://github.com/octodns/octodns-azure/
+.. _BIND, AXFR, RFC-2136: https://www.isc.org/bind/
+.. _octodns_bind: https://github.com/octodns/octodns-bind/
+.. _Bunny DNS: https://bunny.net/dns/
+.. _octodns_bunny: https://github.com/Relkian/octodns-bunny
+.. _Cloudflare DNS: https://www.cloudflare.com/dns/
+.. _octodns_cloudflare: https://github.com/octodns/octodns-cloudflare/
+.. _ClouDNS: https://www.cloudns.net/
+.. _octodns_cloudns: https://github.com/ClouDNS/octodns_cloudns
+.. _Constellix: https://constellix.com/
+.. _octodns_constellix: https://github.com/octodns/octodns-constellix/
+.. _deSEC: https://desec.io/
+.. _octodns_desec: https://github.com/rootshell-labs/octodns-desec
+.. _DigitalOcean: https://docs.digitalocean.com/products/networking/dns/
+.. _octodns_digitalocean: https://github.com/octodns/octodns-digitalocean/
+.. _DNS Made Easy: https://dnsmadeeasy.com/
+.. _octodns_dnsmadeeasy: https://github.com/octodns/octodns-dnsmadeeasy/
+.. _DNSimple: https://dnsimple.com/
+.. _octodns_dnsimple: https://github.com/octodns/octodns-dnsimple/
+.. _Dyn:  https://www.oracle.com/cloud/networking/dns/
+.. _octodns_dyn: https://github.com/octodns/octodns-dyn/
+.. _easyDNS: https://easydns.com/
+.. _octodns_easydns: https://github.com/octodns/octodns-easydns/
+.. _EdgeCenter DNS: https://edgecenter.ru/dns/
+.. _octodns_edgecenter: https://github.com/octodns/octodns-edgecenter/
+.. _Fastly: https://www.fastly.com/de/
+.. _Financial-Times/octodns-fastly: https://github.com/Financial-Times/octodns-fastly
+.. _G-Core Labs DNS: https://gcorelabs.com/dns/
+.. _octodns_gcore: https://github.com/octodns/octodns-gcore/
+.. _Gandi: https://www.gandi.net/en-US/domain/dns
+.. _octodns_gandi: https://github.com/octodns/octodns-gandi/
+.. _Google Cloud DNS: https://cloud.google.com/dns
+.. _octodns_googlecloud: https://github.com/octodns/octodns-googlecloud/
+.. _Hetzner DNS: https://www.hetzner.com/dns-console
+.. _octodns_hetzner: https://github.com/octodns/octodns-hetzner/
+.. _Infoblox: https://www.infoblox.com/
+.. _asyncon/octoblox: https://github.com/asyncon/octoblox
+.. _Infomaniak: https://www.infomaniak.com/
+.. _octodns_infomaniak: https://github.com/M0NsTeRRR/octodns-infomaniak
+.. _Lexicon: https://dns-lexicon.github.io/dns-lexicon/#
+.. _dns-lexicon/dns-lexicon: https://github.com/dns-lexicon/dns-lexicon
+.. _Mythic Beasts DNS: https://www.mythic-beasts.com/support/hosting/dns
+.. _octodns_mythicbeasts: https://github.com/octodns/octodns-mythicbeasts/
+.. _NetBox-DNS Plugin: https://github.com/peteeckel/netbox-plugin-dns
+.. _olofvndrhr/octodns-netbox-dns: https://github.com/olofvndrhr/octodns-netbox-dns
+.. _NS1: https://ns1.com/products/managed-dns
+.. _octodns_ns1: https://github.com/octodns/octodns-ns1/
+.. _OVHcloud DNS: https://www.ovhcloud.com/en/domains/dns-subdomain/
+.. _octodns_ovh: https://github.com/octodns/octodns-ovh/
+.. _Pi-hole: https://pi-hole.net/
+.. _jvoss/octodns-pihole: https://github.com/jvoss/octodns-pihole
+.. _PowerDNS: https://www.powerdns.com/
+.. _octodns_powerdns: https://github.com/octodns/octodns-powerdns/
+.. _Rackspace: https://www.rackspace.com/library/what-is-dns
+.. _octodns_rackspace: https://github.com/octodns/octodns-rackspace/
+.. _Scaleway: https://www.scaleway.com/en/dns/
+.. _octodns_scaleway: https://github.com/scaleway/octodns-scaleway
+.. _Selectel: https://selectel.ru/en/services/additional/dns/
+.. _octodns_selectel: https://github.com/octodns/octodns-selectel/
+.. _SPF Value Management: https://github.com/octodns/octodns-spf
+.. _octodns_spf: https://github.com/octodns/octodns-spf/
+.. _TransIP: https://www.transip.eu/knowledgebase/entry/155-dns-and-nameservers/
+.. _octodns_transip: https://github.com/octodns/octodns-transip/
+.. _UltraDNS: https://vercara.com/authoritative-dns
+.. _octodns_ultra: https://github.com/octodns/octodns-ultra/
+.. _YamlProvider: /octodns/provider/yaml.py
+.. _kompetenzbolzen/octodns-custom-provider: https://github.com/kompetenzbolzen/octodns-custom-provider
+
+Sources
+-------
+
+Similar to providers, but can only serve to populate records into a zone,
+cannot be synced to.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Source/Module
+     - Notes
+   * - `AxfrSource (BIND)`_
+     -
+   * - `DDNS Source`_
+     -
+   * - `EnvVarSource`_
+     - read-only environment variable injection
+   * - `Lexicon Source`_
+     -
+   * - `Netbox Source`_
+     -
+   * - `PHPIPAM Source`_
+     -
+   * - `TinyDnsFileSource`_
+     -
+   * - `ZoneFileSource`_
+     -
+
+.. _AxfrSource (BIND): https://github.com/octodns/octodns-bind/
+.. _DDNS Source: https://github.com/octodns/octodns-ddns
+.. _EnvVarSource: /octodns/source/envvar.py
+.. _Lexicon Source: https://github.com/doddo/octodns-lexicon
+.. _Netbox Source: https://github.com/sukiyaki/octodns-netbox
+.. _PHPIPAM Source: https://github.com/kompetenzbolzen/octodns-custom-provider
+.. _TinyDnsFileSource: /octodns/source/tinydns.py
+.. _ZoneFileSource: https://github.com/octodns/octodns-bind/
+
+Processors
+----------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Processor
+     - Description
+   * - `AcmeManagingProcessor`_
+     - Useful when processes external to octoDNS are managing acme challenge DNS records, e.g. LetsEncrypt
+   * - `AutoArpa`_
+     - See `Automatic PTR generation`_
+   * - `EnsureTrailingDots`_
+     - Processor that ensures ALIAS, CNAME, DNAME, MX, NS, PTR, and SRVs have trailing dots
+   * - `ExcludeRootNsChanges`_
+     - Filter that errors or warns on planned root/APEX NS records changes.
+   * - `IgnoreRootNsFilter`_
+     - Filter that IGNORES root/APEX NS records and prevents octoDNS from trying to manage them (where supported.)
+   * - `MetaProcessor`_
+     - Adds a special meta record with timing, UUID, providers, and/or version to aid in debugging and monitoring.
+   * - `NameAllowlistFilter`_
+     - Filter that ONLY manages records that match specified naming patterns, all others will be ignored
+   * - `NameRejectlistFilter`_
+     - Filter that IGNORES records that match specified naming patterns, all others will be managed
+   * - `ValueAllowlistFilter`_
+     - Filter that ONLY manages records that match specified value patterns based on `rdata_text`, all others will be ignored
+   * - `ValueRejectlistFilter`_
+     - Filter that IGNORES records that match specified value patterns based on `rdata_text`, all others will be managed
+   * - `OwnershipProcessor`_
+     - Processor that implements ownership in octoDNS so that it can manage only the records in a zone in sources and will ignore all others.
+   * - `SpfDnsLookupProcessor`_
+     - Processor that checks SPF values for violations of DNS query limits
+   * - `TtlRestrictionFilter`_
+     - Processor that restricts the allow TTL values to a specified range or list of specific values
+   * - `TypeAllowlistFilter`_
+     - Filter that ONLY manages records of specified types, all others will be ignored
+   * - `TypeRejectlistFilter`_
+     - Filter that IGNORES records of specified types, all others will be managed
+   * - `octodns-spf`_
+     - SPF Value Management for octoDNS
+
+.. _AcmeManagingProcessor: https://github.com/octodns/octodns/tree/main/octodns/processor/acme.py)
+.. _AutoArpa: https://github.com/octodns/octodns/tree/main/octodns/processor/arpa.py)
+.. _EnsureTrailingDots: https://github.com/octodns/octodns/tree/main/octodns/processor/trailing_dots.py)
+.. _ExcludeRootNsChanges: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _IgnoreRootNsFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _MetaProcessor: https://github.com/octodns/octodns/tree/main/octodns/processor/meta.py)
+.. _NameAllowlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _NameRejectlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _ValueAllowlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _ValueRejectlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _OwnershipProcessor: https://github.com/octodns/octodns/tree/main/octodns/processor/ownership.py)
+.. _SpfDnsLookupProcessor: https://github.com/octodns/octodns/tree/main/octodns/processor/spf.py)
+.. _TtlRestrictionFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/restrict.py)
+.. _TypeAllowlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _TypeRejectlistFilter: https://github.com/octodns/octodns/tree/main/octodns/processor/filter.py)
+.. _octodns-spf: https://github.com/octodns/octodns/tree/main//github.com/octodns/octodns-spf)
+
+Custom Sources and Providers
+----------------------------
+
+You can check out the source_ and provider_ directories to see what's currently
+supported.  Sources act as a source of record information. AxfrSource and
+TinyDnsFileSource are currently the only OSS sources, though we have several
+others internally that are specific to our environment. These include something
+to pull host data from gPanel_ and a similar provider that sources information
+about our network gear to create both ``A`` & ``PTR`` records for their
+interfaces. Things that might make good OSS sources might include an
+``ElbSource`` that pulls information about `AWS Elastic Load Balancers`_ and
+dynamically creates ``CNAME``s for them, or ``Ec2Source`` that pulls instance
+information so that records can be created for hosts similar to how our
+``GPanelProvider`` works.
+
+.. _source: https://github.com/octodns/octodns/tree/main/octodns/source/
+.. _provider: https://github.com/octodns/octodns/tree/main/octodns/provider/
+.. _gPanel: https://githubengineering.com/githubs-metal-cloud/
+.. _AWS Elastic Load Balancers: https://aws.amazon.com/elasticloadbalancing/
+
+Most of the things included in octoDNS are providers, the obvious difference
+being that they can serve as both sources and targets of data. We'd really like
+to see this list grow over time so if you use an unsupported provider then PRs
+are welcome. The existing providers should serve as reasonable examples. Those
+that have no GeoDNS support are relatively straightforward. Unfortunately most
+of the APIs involved to do GeoDNS style traffic management are complex and
+somewhat inconsistent so adding support for that function would be nice, but is
+optional and best done in a separate pass.
+
+The ``class`` key in the providers config section can be used to point to
+arbitrary classes in the python path so internal or 3rd party providers can
+easily be included with no coordination beyond getting them into
+``PYTHONPATH``, most likely installed into the virtualenv with octoDNS.
+
+For examples of building third-party sources and providers, see `Related
+Projects and Resources`_
+
+API Documentation
+-----------------
+
+.. toctree::
+   :maxdepth: 2
+
+   api
+
+Contributing
+------------
+
+Please see our contributing_ document if you would like to participate!
+
+.. _contributing: https://github.com/octodns/octodns/tree/main/CONTRIBUTING.md
+
+Getting help
+------------
+
+If you have a problem or suggestion, please `open an issue`_ in this repository, and
+we will do our best to help.
+
+Please note that this project adheres to the `Contributor Covenant Code of
+Conduct`_.
+
+.. _open an issue: https://github.com/octodns/octodns/issues/new
+.. _Contributor Covenant Code of Conduct: https://github.com/octodns/octodns/tree/main/CODE_OF_CONDUCT.md
+
+Related Projects and Resources
+------------------------------
+
+* GitHub Action: `octoDNS-Sync`_
+* NixOS Integration: `NixOS-DNS`_
+* Sample Implementations, see how others are using it
+
+  - `hackclub/dns`_
+  - `kubernetes/k8s.io:/dns`_
+  - `g0v-network/domains`_
+  - `jekyll/dns`_
+
+* Resources
+
+  - Article: `Visualising DNS records with Neo4j`_ + code
+  - Video: `FOSDEM 2019 - DNS as code with octodns`_
+  - GitHub Blog: `Enabling DNS split authority with octoDNS`_
+  - Tutorial: `How To Deploy and Manage Your DNS using octoDNS on Ubuntu 18.04`_
+  - Cloudflare Blog: `Improving the Resiliency of Our Infrastructure DNS Zone`_
+
+.. _octoDNS-Sync: https://github.com/marketplace/actions/octodns-sync
+.. _NixOS-DNS: https://github.com/Janik-Haag/nixos-dns/
+.. _hackclub/dns: https://github.com/hackclub/dns)
+.. _`kubernetes/k8s.io:/dns`: https://github.com/kubernetes/k8s.io/tree/main/dns)
+.. _g0v-network/domains: https://github.com/g0v-network/domains)
+.. _jekyll/dns: https://github.com/jekyll/dns)
+.. _Visualising DNS records with Neo4j: https://medium.com/@costask/querying-and-visualising-octodns-records-with-neo4j-f4f72ab2d474
+.. _FOSDEM 2019 - DNS as code with octodns: https://archive.fosdem.org/2019/schedule/event/dns_octodns/
+.. _Enabling DNS split authority with octoDNS: https://github.blog/2017-04-27-enabling-split-authority-dns-with-octodns/
+.. _How To Deploy and Manage Your DNS using octoDNS on Ubuntu 18.04: https://www.digitalocean.com/community/tutorials/how-to-deploy-and-manage-your-dns-using-octodns-on-ubuntu-18-04
+.. _Improving the Resiliency of Our Infrastructure DNS Zone: https://blog.cloudflare.com/improving-the-resiliency-of-our-infrastructure-dns-zone/
+
+If you know of any other resources, please do let us know!
+
+License
+-------
+
+octoDNS is licensed under the `MIT license`_.
+
+.. _MIT license: https://github.com/octodns/octodns/tree/mainLICENSE
+
+The MIT license grant is not for GitHub's trademarks, which include the logo
+designs. GitHub reserves all trademark and copyright rights in and to all
+GitHub trademarks. GitHub's logos include, for instance, the stylized designs
+that include "logo" in the file title in the following folder:
+https://github.com/octodns/octodns/tree/main/docs/logos/
+
+GitHub® and its stylized versions and the Invertocat mark are GitHub's
+Trademarks or registered Trademarks. When using GitHub's logos, be sure to
+follow the GitHub logo guidelines.
+
+Authors
+-------
+
+octoDNS was designed and authored by `Ross McFarland`_ and `Joe Williams`_. See
+https://github.com/octodns/octodns/graphs/contributors for a complete list of
+people who've contributed.
+
+.. _Ross McFarland: https://github.com/ross
+.. _Joe Williams: https://github.com/joewilliams

script/format

@@ -2,7 +2,7 @@
 
 set -e
 
-SOURCES="$(find *.py octodns tests -name '*.py') $(grep --files-with-matches '^#!.*python' script/*)"
+SOURCES="$(find *.py octodns tests docs -name '*.py') $(grep --files-with-matches '^#!.*python' script/*)"
 
 . env/bin/activate
 

script/generate-docs

@@ -1,23 +1,30 @@
-#!/bin/bash
-
+#!/bin/sh
 set -e
 
 cd "$(dirname "$0")/.."
 ROOT=$(pwd)
-cd "$ROOT"
 
-rm -rf ./docs/_build ./docs/modules
+if [ -z "$VENV_NAME" ]; then
+    VENV_NAME="env"
+fi
+
+ACTIVATE="$VENV_NAME/bin/activate"
+if [ ! -f "$ACTIVATE" ]; then
+    echo "$ACTIVATE does not exist, run ./script/bootstrap" >&2
+    exit 1
+fi
+. "$ACTIVATE"
+
+cd docs
+
+builder=$1
+if [ -n "$builder" ]; then
+    shift
+else
+    builder=html
+fi
 
-for f in $(find octodns -name "*.py" | grep -v __init__.py); do
-    module_name=$(echo $f | sed -e 's#/#.#g' -e 's/.py//')
-    outdir="docs/modules$(dirname $f | sed -e 's#octodns##')"
-    mkdir -p "$outdir"
-    cat <<EOL > "${outdir}/${module_name}.rst"
-====================================
-``${module_name}``
-====================================
-.. automodule:: ${module_name}
-EOL
-done
+build="_build/${builder}"
+rm -rf "$build"
 
-sphinx-build -M html "${ROOT}/docs" "${ROOT}/docs/_build" "$@"
+sphinx-build --builder "$builder" --conf-dir . "source/" "$build" "$@"

script/lint

@@ -15,6 +15,6 @@ if [ ! -f "$ACTIVATE" ]; then
 fi
 . "$ACTIVATE"
 
-SOURCES="$(find *.py octodns tests -name '*.py') $(grep --files-with-matches '^#!.*python' script/*)"
+SOURCES="$(find *.py octodns tests docs -name '*.py') $(grep --files-with-matches '^#!.*python' script/*)"
 
 pyflakes $SOURCES