Merge pull request #1294 from octodns/custom-example

Minor improvements to examples docker compose file

.changelog/5557cbfa56bb40bfbb97511af4659996.md

@@ -0,0 +1,4 @@
+---
+type: none
+---
+Add a custom source example to the doc

docs/examples/README.rst

@@ -45,3 +45,4 @@ and running.
 
    basic/README.md
    migrating-to-octodns/README.md
+   custom/index.rst

docs/examples/custom/config/dns.math.yaml

@@ -0,0 +1,5 @@
+---
+'':
+  # just going to put something in here for the hell of it
+  - type: TXT
+    value: Try querying for TXT records named fibonacci-N where N is an integer 0-25

docs/examples/custom/config/octodns.yaml

@@ -0,0 +1,32 @@
+---
+# see the basic example for detailed infomation on this file
+providers:
+  config:
+    class: octodns.provider.yaml.YamlProvider
+    directory: ./config
+
+  powerdns:
+    class: octodns_powerdns.PowerDnsProvider
+    host: 127.0.0.1
+    port: 8081
+    api_key: env/POWERDNS_API_KEY
+    timeout: 10
+
+  # So this is our custom/local source, see fibonacci.py. It works just like
+  # any other souce/provider. When sync is run the directory where that file
+  # lives will need to be included in your PYTHONPATH, likely ., so that
+  # octoDNS can find it.
+  fibonacci:
+    class: fibonacci.FibonacciProvider
+    n: 25
+
+zones:
+
+  dns.math.:
+    sources:
+      # Statically condifured records (yaml) will be included in this zone
+      - config
+      # as well as records generated by our simple custom provider
+      - fibonacci
+    targets:
+      - powerdns

docs/examples/custom/fibonacci.py

@@ -0,0 +1,77 @@
+#
+#
+#
+
+from logging import getLogger
+
+from octodns.record import Record
+from octodns.source.base import BaseSource
+
+
+# Based on https://stackoverflow.com/a/3954407
+def _fibonacci(n):
+    a, b = 0, 1
+    for _ in range(n):
+        yield a
+        a, b = b, a + b
+
+
+class FibonacciProvider(BaseSource):
+    # Here only TXT records are supported
+    SUPPORTS = ('TXT',)
+    # Do-not support (deprecated) GEO records
+    SUPPORTS_GEO = False
+
+    #  This is pretty much boilerplate, create a logger with a standard name,
+    #  call up to the parent class BaseSource's __init__, store our
+    #  attributes, and then calculate our fibonacci sequence
+    def __init__(self, id, n, ttl=3600):
+        klass = self.__class__.__name__
+        self.log = getLogger(f'{klass}[{id}]')
+        self.log.debug(
+            '__init__: id=%s, variable=%s, name=%s, ttl=%d', id, n, ttl
+        )
+        super().__init__(id)
+        self.n = n
+        self.ttl = ttl
+
+        # calculate the requested number of values
+        self.values = list(_fibonacci(n))
+
+    def populate(self, zone, target=False, lenient=False):
+        # This is the method adding records to the zone. For a source it's the
+        # only thing that needs to be implemented. Again there's some best
+        # practices wrapping our custom logic, mostly for logging/debug
+        # purposes.
+        self.log.debug(
+            'populate: name=%s, target=%s, lenient=%s',
+            zone.name,
+            target,
+            lenient,
+        )
+
+        before = len(zone.records)
+
+        # This is where the logic of the source lives. Here it's simply
+        # translated the previously calculated fibonacci sequence into
+        # corresponding TXT records. It could be anything: reading data from
+        # disk, calling APIs, ...
+        for i, value in enumerate(self.values):
+            # If there's a chance the generated record may result in validation
+            # errors it would make sense to pass lenient to new
+            txt = Record.new(
+                zone,
+                f'fibonacci-{i}',
+                {'value': str(value), 'ttl': self.ttl, 'type': 'TXT'},
+            )
+            # lenient should always be passed to add_record.  If there's a
+            # chance the source will create records that conflict with
+            # something previously added by another provider it may make sense
+            # to include `replace` here, possibly with an additional provider
+            # parameter `populate_should_replace`
+            zone.add_record(txt, lenient=lenient)
+
+        self.log.info(
+            'populate:   found %s records, exists=False',
+            len(zone.records) - before,
+        )

docs/examples/custom/index.rst

@@ -0,0 +1,142 @@
+Writing a Custom Source
+=======================
+
+Introduction
+------------
+
+Creating a custom source of record data for octoDNS is pretty simple and
+involves a bit of boilerplate and then filling in a single method,
+:py:meth:`octodns.source.base.BaseSource.populate`, with any logic required
+to fetch or create the desired records. In this example records will be created
+for the first 25 elements of the `Fibonacci Sequence`_. While contrived it
+should illustrate the process and requirements.
+
+.. _Fibonacci Sequence: https://en.wikipedia.org/wiki/Fibonacci_sequence
+
+Some relevant documentation for this example is in comments in the YAML
+configuration files and python code.
+
+* :download:`config/octodns.yaml`
+* :download:`config/dns.math.yaml`
+* :download:`fibonacci.py`
+
+From here on this README focuses on the custom source and the process of
+running octoDNS with access to it.
+
+Checking out the code and setting up the environment
+----------------------------------------------------
+
+You would not normally need to check out octoDNS itself, you instead would have
+a git repo with only your configuration files. Here we're cloning the repo only
+to get a copy of the example files::
+
+  $ git clone https://github.com/octodns/octodns.git
+  $ cd octodns/examples/basic/
+  $ python3 -mvenv env
+  $ source ../env.sh
+  $ source env/bin/activate
+  (env) $ pip install -r requirements.txt
+
+Finally check out :ref:`Running PowerDNS` to get a local instance of PowerDNS
+up and going before continuing.
+
+Running octoDNS sync
+--------------------
+
+Once you have your custom source, configuration files, and octoDNS installed
+you're ready to run the sync command to get it to plan an initial set of
+changes. The main difference here compared to the :ref:`basic-setup` is setting
+``PYTHONPATH`` so the source file can be located::
+
+  (env) $ export PYTHONPATH=.
+  (env) $ octodns-sync --config-file config/octodns.yaml
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.13.0)
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager _config_executor: max_workers=1
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager _config_include_meta: include_meta=False
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager _config_enable_checksum: enable_checksum=False
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager _config_auto_arpa: auto_arpa=False
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: global_processors=[]
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: global_post_processors=[]
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: provider=config (octodns.provider.yaml 1.13.0)
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: provider=powerdns (octodns_powerdns 1.0.0)
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager __init__: provider=fibonacci (fibonacci n/a)
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager sync: eligible_zones=[], eligible_targets=[], dry_run=True, force=False, plan_output_fh=<stdout>, checksum=None
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager sync:   zone=dns.math.
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager sync:     sources=['config', 'fibonacci']
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager sync:     processors=[]
+  2025-08-17T17:14:58  [140224800168896] INFO  Manager sync:     targets=['powerdns']
+  2025-08-17T17:14:58  [140224800168896] INFO  YamlProvider[config] populate:   found 1 records, exists=True
+  2025-08-17T17:14:58  [140224800168896] INFO  FibonacciProvider[fibonacci] populate:   found 25 records, exists=False
+  2025-08-17T17:14:58  [140224800168896] INFO  PowerDnsProvider[powerdns] plan: desired=dns.math.
+  2025-08-17T17:14:58  [140224800168896] INFO  PowerDnsProvider[powerdns] populate:   found 1 records, exists=True
+  2025-08-17T17:14:58  [140224800168896] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for dns.math.
+  2025-08-17T17:14:58  [140224800168896] INFO  PowerDnsProvider[powerdns] plan:   Creates=25, Updates=0, Deletes=0, Existing=1, Meta=False
+  2025-08-17T17:14:58  [140224800168896] INFO  Plan
+  ********************************************************************************
+  * dns.math.
+  ********************************************************************************
+  * powerdns (PowerDnsProvider)
+  *   Create <TxtRecord TXT 3600, fibonacci-0.dns.math., ['0']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-1.dns.math., ['1']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-10.dns.math., ['55']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-11.dns.math., ['89']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-12.dns.math., ['144']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-13.dns.math., ['233']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-14.dns.math., ['377']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-15.dns.math., ['610']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-16.dns.math., ['987']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-17.dns.math., ['1597']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-18.dns.math., ['2584']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-19.dns.math., ['4181']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-2.dns.math., ['1']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-20.dns.math., ['6765']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-21.dns.math., ['10946']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-22.dns.math., ['17711']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-23.dns.math., ['28657']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-24.dns.math., ['46368']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-3.dns.math., ['2']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-4.dns.math., ['3']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-5.dns.math., ['5']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-6.dns.math., ['8']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-7.dns.math., ['13']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-8.dns.math., ['21']> ()
+  *   Create <TxtRecord TXT 3600, fibonacci-9.dns.math., ['34']> ()
+  *   Summary: Creates=25, Updates=0, Deletes=0, Existing=1, Meta=False
+  ********************************************************************************
+
+The log output
+..............
+
+Everything here matches the output and meaning of the first run in
+:ref:`basic-setup`, with the important difference that both the statically
+configured and dynamically generated records are listed as planned changes.
+From here a ``--doit`` run can be executed to create the records in the
+PowerDNS server, which can then be queried::
+
+Viewing the results
+-------------------
+
+``dig`` can now be run to query for the records::
+
+  $ dig +short TXT dns.math.
+  "Try querying for TXT records named fibonacci-N where N is an integer 0-25"
+  $ dig +short TXT fibonacci-0.dns.math.
+  "0"
+  $ dig +short TXT fibonacci-23.dns.math.
+  "28657"
+  $ dig +short TXT fibonacci-99.dns.math.
+
+Next Steps
+----------
+
+If the source will only be used in a single octoDNS setup and you're OK with it
+living alongside your config, as was done in this example the
+:download:`fibonacci.py` file can be used as a starting point. It is also
+possible to create custom processors and providers that work in the same
+manner, though the details of the code involved are outside the scope of this
+example.
+
+If the provider will be used more widely or published for others to use, see
+`octodns-template`_.
+
+.. _octodns-template: https://github.com/octodns/octodns-template

docs/examples/custom/pyproject.toml

@@ -0,0 +1 @@
+../../../pyproject.toml

docs/examples/custom/requirements.txt

@@ -0,0 +1,2 @@
+octodns>=1.0.1
+octodns_powerdns>=0.0.5

docs/examples/docker-compose.yml

@@ -1,5 +1,3 @@
-version: "3.9"  # optional since v1.27.0
-
 volumes:
   powerdns-db-data:
     name: powerdns-db-data
@@ -16,8 +14,8 @@ services:
   powerdns:
     image: psitrax/powerdns
     ports:
-      - "53:53/tcp"
-      - "53:53/udp"
+      - "${PORT:-53}:53/tcp"
+      - "${PORT:-53}:53/udp"
       - "8081:8081"
     environment:
       MYSQL_HOST: db
@@ -26,4 +24,4 @@ services:
       MYSQL_PASS: l3tmein
     depends_on:
       - db
-    command: --api=yes --api-key=its@secret --loglevel=99 --webserver=yes --webserver-address=0.0.0.0 --webserver-allow-from=0.0.0.0/0 --webserver-password=its@secret --webserver-port=8081 --enable-lua-records=shared --edns-subnet-processing=yes
+    command: --api=yes --api-key=its@secret --loglevel=99 --log-dns-queries=yes --webserver=yes --webserver-address=0.0.0.0 --webserver-allow-from=0.0.0.0/0 --webserver-password=its@secret --webserver-port=8081 --enable-lua-records=shared --edns-subnet-processing=yes

docs/examples/env.sh

@@ -1 +1,3 @@
 export POWERDNS_API_KEY="its@secret"
+# you may need to change this to something else if there's already a DNS server running on your system
+export PORT=53