The journal is not currently unit tested. We need to write tests to confirm objects are logged to the journal as we expect them to be.

Environment

• Nautobot version: 2.x
• design-builder version: next

Proposed Functionality

Building on issue #52.

A specific job for the decommissioning (or rollback) of a service should be implemented. Ideally, this should be a generic job that takes as input the design instance and using its Journal is able to remove/update the objects/attributes. This operation should account for dependencies, both in order of deletion and also about protection.

• Note: The load_jobs helper may become unnecessary with Nautobot 2.x, since Jobs become importable Python modules in that version.
• The field__<relatedfield> action tag (is it supposed to have a leading !? There isn't one in the docs) seems a bit limiting, in that it requires that the related object be uniquely identifiable by a single field. Would it make sense to use a nested dict instead so as to permit multiple values, something like:

  !field:
      related_field_1: value_1
      related_field_2: value_2

• I wonder if the !ref action tag should be renamed to something like !store_ref or !save_ref for clarity and perhaps to make it less ambiguous with the !ref value tag?
• pyproject.toml identifies the minimum Nautobot version as 1.2.0 but DesignBuilderConfig has it as 1.2.7. These must be consistent.
• Consider renaming base.py to something like design_job.py for clarity
• Docstring for DesignJob.post_implementation method is fairly vague - what is this for and why would subclasses want/need to override it?
• The while cls is not DesignJob logic in DesignJob.render() is a bit opaque, some comments might be a good idea.
• Seems a bit odd to have a try...except block that does nothing on an exception except reraise the exception - remove?
• The 2.0 logic around dryrun seems backwards to me - shouldn't it be commit = **not** kwargs.pop("dryrun", False)?
• Is there a reason the Python module is named design_builder rather than nautobot_design_builder following our convention for other Nautobot apps?
• #74
• The Journal.log() docstring suggests incorrectly that created is a function parameter, when actually it appears to be an attribute of the provided model object?
• The docstring and typing of Journal.created_objects suggest that it returns a list, but it appears to actually return a dict?
• #75
• Do ModelInstance and Builder actually rely on all models being BaseModel subclasses, or should they be generalized to use django.db.models.Model instead?
• Would renaming ext.py to extensions.py be clearer?
• Would it be better to split Extension into two abstract classes, an AttributeExtension (requires attribute_tag and attribute() definitions) and a ValueExtension (requires value_tag and value() definitions)?
• TODO: Need to write unit tests for this on Extension.commit and Extension.roll_back
• What's the intent of the contrib directory? What makes these extensions "different"?
• Consider renaming invoke sample_data to something like invoke populate_sample_data or similar verb?

As ...
Austin, Network Automation Engineer

I want ...
Easily understand my design mistakes , without troubleshooting python and going through tracebacks

So that ...
I can quickly develop my designs

I know this is done when...
I can read errors easily

For example:
  File "/source/design_builder/object_creator.py", line 478, in _create_objects
    CreatorObject(self, model_cls, creator_object)
  File "/source/design_builder/object_creator.py", line 169, in __init__
    self._update_fields()
  File "/source/design_builder/object_creator.py", line 254, in _update_fields
    self.attributes[field_name] = extn.value(arg).instance
  File "/source/design_builder/ext.py", line 139, in value
    creator_object = self._env[key]
KeyError: 'branch'

Means, I did not saved my branch object with !ref statement

Environment

• Python version: 3.11
• Nautobot version: n/a
• design-builder version: 2023-07-31

Expected Behavior

Expected to be able to execute inv build

Observed Behavior

❯ inv build
Building Nautobot with Python 3.8...
Running docker-compose command "build"
/bin/bash: line 1: docker-compose: command not found

Steps to Reproduce

1. Have machine
2. Run inv build once into a poetry environment
3. See error

Environment

• Python version:
• Nautobot version:
• nautobot-design-builder version:

Expected Behavior

Run job based on dry_run setting that is already defined on the job.

Observed Behavior

dry_run must be explicitly called, but I think it should should not be needed to run

Steps to Reproduce

1. Go to Nautobot Shell
2. job_model = Job.objects.get(name="Initial Data")
3. job_result = JobResult.execute_job(job_model, admin_user)

Should be able to put other logic here:

Environment

• Python version: 3.7
• Nautobot version: 1.5.14
• design-builder version: 0.1.0

Expected Behavior

Whe executing a Nautobot Job that uses the following J2 template, I would expect that:

• the !ref succesfully renders to the referenced value.
• the !create_or_update directive takes care of idempotency.

# Secrets & Secrets Groups
secrets:
- "!create_or_update:name": "Device username"
  "description": "Username for network devices"
  "provider": "environment-variable"
  "parameters": {"variable": "NAUTOBOT_NAPALM_USERNAME"}
  "!ref": "device_username"
- "!create_or_update:name": "Device password"
  "description": "Password for network devices"
  "provider": "environment-variable"
  "parameters": {"variable": "NAUTOBOT_NAPALM_PASSWORD"}
  "!ref": "device_password"

secrets_groups:
- "!create_or_update:name": "Device credentials"
  "!ref": "device_credentials"

# FIX (nsk): Secrets Group Asssociations with 'create_or_update'
secrets_group_associations:
- "!create_or_update:group": "!ref:device_credentials"
  "!create_or_update:secret": "!ref:device_username"
  "access_type": "Generic"
  "secret_type": "username"
- "!create_or_update:group": "!ref:device_credentials"
  "!create_or_update:secret": "!ref:device_password"
  "access_type": "Generic"
  "secret_type": "password"

Observed Behavior

An exception is being thrown:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 2434, in to_python
    return uuid.UUID(**{input_form: value})
  File "/usr/local/lib/python3.7/uuid.py", line 157, in __init__
    hex = hex.replace('urn:', '').replace('uuid:', '')
AttributeError: 'ModelInstance' object has no attribute 'replace'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/jobs.py", line 1214, in _run_job
    output = job.run(data=data, commit=commit)
  File "/source/design_builder/base.py", line 182, in run
    self.implement_design(context, design_file)
  File "/source/design_builder/base.py", line 159, in implement_design
    raise ex
  File "/source/design_builder/base.py", line 156, in implement_design
    self.creator.implement_design(design)
  File "/source/design_builder/design.py", line 428, in implement_design
    self._create_objects(self.model_map[key], value)
  File "/source/design_builder/design.py", line 476, in _create_objects
    ModelInstance(self, model_cls, creator_object)
  File "/source/design_builder/design.py", line 160, in __init__
    self._load_instance()
  File "/source/design_builder/design.py", line 215, in _load_instance
    self.instance = self.model_class.objects.get(**query_filter)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 351, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 424, in get
    clone = self._chain() if self.query.combinator else self.filter(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 941, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 961, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 968, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1416, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1438, in _add_q
    split_subq=split_subq, check_filterable=check_filterable,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1370, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1216, in build_lookup
    lookup = lookup_class(lhs, rhs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/lookups.py", line 25, in __init__
    self.rhs = self.get_prep_lookup()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_lookups.py", line 117, in get_prep_lookup
    self.rhs = target_field.get_prep_value(self.rhs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 2418, in get_prep_value
    return self.to_python(value)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 2439, in to_python
    params={'value': value},
django.core.exceptions.ValidationError: ['“” is not a valid UUID.']

Steps to Reproduce

1. Use the provided J2 template in a design file.
2. Sync the relevant repo.
3. Execute the respective Job et voila!

Expected Behavior

Job result should update as the builder is running.

Observed Behavior

Job result seems to only update once everything has completed.

Nautobot 2.0 dropped slug fields nautobot/nautobot#3439

This issue is to check and track if design-builder plugin is impacted by #3439 change in Nautobot by using or referencing to any of the .slug fields.

Environment

• Nautobot version: 2.x
• design-builder version: next

Proposed Functionality

Building on issue #52.

At some point, when there will be many services defined, it could be necessary to establish a hierarchical design relationship, so a high-level design could be composed of lower-level ones. So, one design job could call several minor design jobs under one umbrella. Moreover, there could be a need to establish dependencies between the services, so one technical service could not be implemented until its dependencies are in place. For instance, a l3vpn requires the network management service in a device.

Environment

• Python version: all
• Nautobot version: all
• design-builder version: all

Expected Behavior

The to_yaml filter should render YAML for Design Builder Context objects.

Observed Behavior

The pyyaml dumper goes into an infinite recursion and causes a stack overflow.

Steps to Reproduce

1. Call to_yaml and pass it a Context object.

Environment

• Python version: 1.5
• Nautobot version: 3.9
• design-builder version: 0.1.0

Expected Behavior

If an exception occurs after rendering a portion of the list of design_files then all changes should be rolled back.

Observed Behavior

Changes that occurred from design files prior to the exception are not rolled back.

Steps to Reproduce

1. Create a design that utilizes multiple design files
2. Cause an error in the middle of the list
3. Observe that the previous changes are not rolled back

Environment

• Nautobot version: 1.6.0
• design-builder version: 0.4.4

Observed Behavior

When misspelling the DesignJob's Meta class (meta vs Meta), the job is not registered as a valid design. I expected a log to inform me that there was an issue with the DesignJob.

The failing check is isabstract in util.py line 154:
return issubclass(obj, DesignJob) and obj is not DesignJob and not inspect.isabstract(obj)

A log for when this check fails would be helpful.

Steps to Reproduce

1. Create a job with an improperly named Meta method and try to register it as a DesignJob:

class AristaObjectsDesign(DesignJob):
    """Create arista objects Job."""

    manufacturer = StringVar(
        label="Manufacturer Name",
    )

    class meta:
        name = "Arista Objects Design"
        commit_default = False
        design_file = "designs/0001_manufacturer.yaml.j2"
        context_class = AristaObjectsContext

Update as per nautobot/nautobot#3720

As ...
Austin, Network Automation Engineer

I want ...
Develop and test my designs quickly and efficiently

So that ...
I can quickly render and destroy my network design

I know this is done when...
I need only single cli command to test a design

Environment

• Nautobot version: 2.0 and 1.5
• design-builder version: 1.0

Proposed Functionality

This application should work with both Nautobot 1.5 and 2.0

Environment

• Nautobot version: 2.0
• design-builder version: next

Proposed Functionality

Add concrete models (Design Instance) to capture executions of the various Designs.

>>> Design

The DesignJob is the base Job definition for all the designs. In order to support some extra features required by this HLD, some additional fields would be required.

>>> Design Instance

This model should contain the following information/attributes:

>>> Journal

It is a many-to-many relationship to the AssociatedDesignObject table which contains:

This table is used for two purposes:

• From a design point of view, check which are the objects impacted.
• From an object perspective, check which are the design related.

Use Case

As Randy the Random User, I'd like to be able to control the whole lifecycle of a service:

• create a new service instance using a design created by my team's architects
• update resources related to the service instance using the same design
• decommission the service instance when it's no longer necessary

Environment

• Python version:
• Nautobot version:
• design-builder version:

Expected Behavior

Design builder works with Nautobot 2.0

Observed Behavior

Design builder failds to load on Nautobot 2.0

Steps to Reproduce

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 257, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 39, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/source/design_builder/management/commands/render_backbone_designs.py", line 10, in <module>
    from design_builder.designs.backbone import CreateBackbone
  File "/source/design_builder/designs/backbone.py", line 1, in <module>
    from design_builder.design_builder import DesignJob
  File "/source/design_builder/design_builder.py", line 23, in <module>
    from design_builder.logging import LoggingMixin
  File "/source/design_builder/logging.py", line 11, in <module>
    logging.ERROR: LogLevelChoices.LOG_FAILURE,
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/choices.py", line 21, in __getattribute__
    value = super().__getattribute__(attr)
AttributeError: type object 'LogLevelChoices' has no attribute 'LOG_FAILURE'

Environment

• Python version:
• Nautobot version:
• design-builder version:

Expected Behavior

IP Address creation works

Observed Behavior

IP Address creation is not possbile with Nautobot 2.0 namespaces

Steps to Reproduce

Goal:
Enable new user with design builder
Ensure new users can easily follow examples
Ensure new users understand data variable process/management (from the user perspective, not the tool designer perspective)
Ensure new users understand dependencies between YAML, UI Inputs and Context classes

Scope:

• migrate salsa docs
• link the knowledge sharing videos
• knowledge sharing diagrams / presentation to be included
• diagram showing how data is passed into design files

-> job UI INPUT \
                 \
                 ---> ContexClass.py (self.properties) ------> design files (YAML)
                /
-> YAML files  /

Environment

• Nautobot version: 1.x and 2.x
• nautobot-design-builder version:

Proposed Functionality

This is a parent issue to track different features that could enable Design Builder to support a complete lifecycle of design instances/implementations.

This is a list of features (in feature_delices branch):

• Persist in the DB the implementation of a design, with a reference to all the objects and changes associated
• Allow to decommission all the changes done from an existing instance of a design, understanding design instances dependencies
• Allow to update a design instance by rerunning the design implementation with new input data
• Allow to use of different versions of the design related to a design instance to allow concurrent versions with different flavors
• Protect DB objects that are managed completely or partially by a design instance
• Allow customization of data mappings from DB objects related to a design instance to be exposed via APIs

Use Case

Allow for implementing a complete design-oriented source of truth management.

Proposed Functionality

Add a jinja2 filter that will consume a variable and produce appropriate YAML

Use Case

This should allow for things like devices profiles to be represented in the context and rendered in the design making design templates more concise.

Environment

• Python version:
• Nautobot version:
• design-builder version:

Expected Behavior

Creation of Git Repository fails

Observed Behavior

Creation of Git Repository is possible

Steps to Reproduce

1. Create Git Repository2.

git_repositories:
  - name: backups
    remote_url: "https://github.com/nautobot/demo-gc-backups"
    branch: main

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/source/design_builder/management/commands/test_render_networktocode_designs.py", line 69, in handle
    job.run(data=None, commit=True)
  File "/source/design_builder/design_builder.py", line 460, in run
    self.implement_design(context, design_file)
  File "/source/design_builder/design_builder.py", line 437, in implement_design
    raise ex
  File "/source/design_builder/design_builder.py", line 434, in implement_design
    self.creator.implement_design(design)
  File "/source/design_builder/object_creator.py", line 430, in implement_design
    self._create_objects(self.model_map[key], value)
  File "/source/design_builder/object_creator.py", line 478, in _create_objects
    CreatorObject(self, model_cls, creator_object)
  File "/source/design_builder/object_creator.py", line 169, in __init__
    self._update_fields()
  File "/source/design_builder/object_creator.py", line 278, in _update_fields
    self.creator.save_model(self)
  File "/source/design_builder/object_creator.py", line 506, in save_model
    model.instance.save()
  File "/usr/local/lib/python3.8/site-packages/nautobot/extras/models/datasources.py", line 125, in save
    transaction.on_commit(on_commit_callback)
  File "/usr/local/lib/python3.8/site-packages/django/db/transaction.py", line 128, in on_commit
    get_connection(using).on_commit(func)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 645, in on_commit
    func()
  File "/usr/local/lib/python3.8/site-packages/nautobot/extras/models/datasources.py", line 109, in on_commit_callback
    assert self.request is not None, "No HTTP request associated with this update!"
AssertionError: No HTTP request associated with this update!

Environment

• Nautobot version: 2.1
• design-builder version: Christian's video on Slack

Proposed Functionality

During the presentation there was a data validation that came after the field was updated. Would be good to have a visual indicator that the field is going to be validated by Design Builder and what the design is. Maybe even what the design requirements are for the field

Use Case

Give the feedback before feedback that there may be something that will be errored if typed wrong.

Maybe a slightly grey background on the fields??

Apparently the get tag is not documented (see documentation here). We need to add some documentation on that tag.

Environment

• Nautobot version: 1.5.x
• design-builder version: latest

Proposed Functionality

Provide the ability to run Golden Config as a hook and/or chain it to happen after it.

Use Case

Making it easier for someone to generate a device, site, etc. data and then automatically generate the configurations for the asset(s) that you have just created. This get's us closer to full day 0/1 configuration deployment/ztp/process.

Environment

• Python version: 3.11
• Nautobot version: n/a
• design-builder version: 2023-07-31 version

Expected Behavior

Expected to be able to get into the poetry environment

Observed Behavior

❯ poetry shell
The currently activated Python version 3.11.2 is not supported by the project (>=3.8,<3.11).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

Steps to Reproduce

1. Get into a dev host that has Python3.11
2. Run poetry shell to try to get to the env
3. Get error

Environment

• Nautobot version:
• nautobot-design-builder version:

Proposed Functionality

Allow to customize protection at attribute level to allow different use cases. By default, it will default to protected. The information would be then populated into the JournalEntry to provide understanding of the attributes that are NOT protected.

Use Case

For example, when a design creates a Device with Status Planned, we would like to update the Status later to Active. By default, if data protection is enforce, that won't be possible (if not bypassed as superuser). To allow more rich workflows, some attributes of the objects created by a design should be "unprotected" to allow changes.

Environment

• Nautobot version: >= 1.5.x
• design-builder version: 0.1.0

The provided example design repository should be built in such a way as to provide a complete development environment without the need for a specific IDE. The current example is really geared towards users of Visual Studio Code's devcontainer feature.

Proposed Functionality

It should be very simple for a design developer to have a complete application stack ready for testing and development of standalone design repositories. This functionality should be provided using Python Invoke to align with other NTC tools. The example repository should be used as a template for creating new design repos where a simple clone of the repo and invoke debug will spin up all the necessary components for testing designs.

Environment

• Python version:
• Nautobot version:
• design-builder version:

Expected Behavior

Job render works correctly on Nautobot 2.0

Observed Behavior

Job render fails under Nautobot 2.0

Steps to Reproduce

Running docker-compose command "exec nautobot bash"
root@8de856dc6f9c:/source# nautobot-server test_render_branch_designs
Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/source/design_builder/management/commands/test_render_branch_designs.py", line 87, in handle
    job = CreateBase()
  File "/source/design_builder/design_builder.py", line 324, in __init__
    self.request = None
AttributeError: can't set attribute

Dev Standards

Once this section is addressed, we can ask the apps team nicely to use the drift manager tool and bring this repo up to the current standard.

• The development environment addition of the git server container makes it non-standard, is this done just to have a simple local github alternative? I'm wondering if it's not going to add more complexity in the long run instead of people using their own github/gitlab/internal vcs directly.
  • Case in point, the docs are not updated to talk about this container so it's a bit of a black box. With just the standard devenv we'd be 100% accurate from the get-go and it is also easier to maintain in the long run (via the drift manager tool and DevStandards).
• Minimum Nautobot version should be 1.6 LTM unless there's a good reason to also support 1.5 (and then it should be 1.5.2 minimum)
• To prepare the app for a Nautobot 2.0 compatible release, all imports should be from nautobot.apps where possible (https://docs.nautobot.com/projects/core/en/v1.6.4/plugins/development/)
• What is the plan for Nautobot 2.0 compatibility? I think the datasets it can produce are going to be very useful to have for testing/development with 2.0 now out.
• Update .cookiecutter.json to reflect the renamed module and latest cookie standard - suggested change below, but please double check. This is so we can apply the drift manager tool on this repo and bring it to the current standard. Also changed github references to the public nautobot org where this will be located once public.

{
  "cookiecutter": {
    "codeowner_github_usernames": "@abates @mzbroch",
    "full_name": "Network to Code, LLC",
    "email": "[email protected]",
    "github_org": "nautobot",
    "plugin_name": "nautobot_design_builder",
    "verbose_name": "Nautobot Design Builder",
    "plugin_slug": "nautobot-design-builder",
    "project_slug": "nautobot-plugin-design-builder",
    "repo_url": "https://github.com/nautobot/nautobot-plugin-design-builder",
    "base_url": "design-builder",
    "min_nautobot_version": "1.6.0",
    "max_nautobot_version": "1.9999",
    "camel_name": "NautobotDesignBuilder",
    "project_short_description": "A Nautobot App that uses design templates to easily create data objects in Nautobot with minimal input from a user.",
    "model_class_name": "None",
    "open_source_license": "Apache-2.0",
    "docs_base_url": "https://docs.nautobot.com",
    "docs_app_url": "https://docs.nautobot.com/projects/design-builder/en/latest",
  }
}

Documentation

Some very nice write-ups in there! There are some sections which need some love:

• app_overview.md -> Description is empty
• app_getting_started.md -> need an empty line before ![Jobs list](../images/screenshots/sample-design-jobs-list.png)
• app_use_cases.md incomplete - not sure how much you can add here, but for sure some more screenshots / examples of results and datasets could work
• design_quickstart references at the end invoke create-local-repo which does not exist (see below about keeping the devenv standard before adding it though :)
• Admin Guide is vanilla - please fill in as necessary.
• All Developer Note - Remove Me! admonitions in the docs must be addressed and removed
  • One exception we plan to push via the cookiecutter drift management is dev/contributing.md so leave it as-is for now

I very much like the addition of abbreviations, I'll look to add it into the template, as I think a good part of this list should come from the cookicutter (as standard).

Environment

• Python version:
• Nautobot version:
• design-builder version:

Expected Behavior

Observed Behavior

Report fails:

root@6be83fc03aaa:/source# nautobot-server render_backbone_designs
Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/source/design_builder/management/commands/render_backbone_designs.py", line 32, in handle
    job.run(data=None, commit=True)
  File "/usr/local/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/source/design_builder/base.py", line 220, in run
    raise ex
  File "/source/design_builder/base.py", line 206, in run
    self.results["report"] = self.render_report(context, self.creator.journal)
AttributeError: 'CreateBackbone' object has no attribute 'results'

Steps to Reproduce

Environment

• Python version: all
• Nautobot version: all
• design-builder version: 0.1.0

Expected Behavior

The rendered design file is supposed to be included in the job results Additional Data.

Observed Behavior

This seems to be working when the job completes successfully, but not when the job fails. I believe it is likely related to the lack of a transaction referred to in #1

Steps to Reproduce

1. Run a design that will fail after the design has been rendered.
2. Observe that the design is not included in the additional data

Environment

• Nautobot version: latest
• design-builder version: current

Proposed Functionality

I believe anything can be done, since it is just Python, but I am suggesting to create a documented standard of what should be done.

Currently you may end up with a folder structure like this

├── __init__.py
├── backbone.py
├── branch.py
├── datacenter.py
├── design_files
│   ├── __init__.py
│   ├── backbone_context.py
│   ├── backbone_context.yml
│   ├── branch_context.py
│   ├── branch_context.yml
│   ├── device_profiles.yaml
│   └── templates
│       ├── backbone
│       │   ├── 100_backbone_base_elements.yaml.j2
│       │   ├── 200_backbone_cables.yaml.j2
│       │   └── pops_report.md.j2
│       └── branch
│           ├── 100_branch_base_elements.yaml.j2
│           ├── 200_branch.yaml.j2
│           └── 300_branch_access_switch.yaml.j2

It would be cleaner imho to have a folder with all designs, here are two options

.
├── __init__.py
├── backbone
│   ├── 100_base_elements.yaml.j2
│   ├── 200_cables.yaml.j2
│   ├── __init__.py
│   ├── context.py
│   ├── context.yml
│   └── pops_report.md.j2
└── branch
    ├── 100_branch_base_elements.yaml.j2
    ├── 200_branch.yaml.j2
    ├── 300_branch_access_switch.yaml.j2
    ├── __init__.py
    ├── branch_context.py
    └── branch_context.yml

You can still import the Jobs in a central location

# _init__.py output

from .branch import BranchJob
from .backbone import BackboneJob

So one of:

{design}/{file}
{design}/{design}_{file}

Use Case

A folder structure that reenforces the design.

Environment

• Nautobot version: 2.x
• design-builder version: next

Proposed Functionality

Building on issue #52.

For all the data managed by a service, defined in the journal attribute, we should avoid that other mechanisms can change this data outside of the design builder. This enforcement should happen in the related objects, at different levels (object or attribute or key in an attribute), so an option could be using Custom Validators which will take into account the references to designs. For instance, using a relationship towards the design at object level, it could check the related journals (using the reverse key to AssociatedDesignObject), to see if the data is protected or not. In more detail, the process could look like:

1. When an object is going to call the validation, it would check the Custom Validator
2. If there is a reference from this object to one or many designs, it would extract the Journals
3. It would check if the change requested is compatible with the Journals, and if not, it would reject the change. For instance, if it’s trying to change a key in a config context which is owned by a design, it should say that this key can’t be overwritten because a design owns this data.

Use Case

As Bob the Design Builder, I'd like to have the capability to "lock" data that are solely managed by my designs and prevent user manipulation. For instance:

{
  “description”: "this is an important device”,
 “config_context”: {“lldp”: true}
}

For the device config context, because we are only referencing the “lldp” key, this is the only key that is owned by the design. Other keys in the config_context can be modified outside of the control of this design.

Environment

• Nautobot version: 1.5.x
• design-builder version: 0.1.0

Proposed Functionality

The CI/CD should include a nightly build that publishes the .whl file to github. In order to stand up a testing environment for designs the plugin needs to be installed in Nautobot. Having a wheel file that is privately available would make developing design repos easier, at least until this project is open sourced.

Environment

• Python version: >= 3.8
• Nautobot version: Any
• nautobot-design-builder version: 1.0

Expected Behavior

I expect a context that has @property decorators can actually access the properties.

Observed Behavior

The property attributes are not found and an AttributeError is raised when trying to access them.

Steps to Reproduce

1. Create a property in a context class.
2. Attempt to access that property from an instance of the context class.

Environment

• design-builder version: 0.4.4

Expected Behavior

I can use Design Builder example jobs to successfully import design jobs

Observed Behavior

I followed example and documentation, however I was not able to successfully import Design Jobs

Steps to Reproduce

For the 2.0, following code allowed me to successfully import design jobs

$PLUGIN/jobs.py:

from nautobot.apps.jobs import Job, register_jobs
from nautobot_gizmo_designs.designs.networktocode import CreateNetworkToCode

register_jobs(CreateNetworkToCode)

Environment

• Nautobot version:
• nautobot-design-builder version:

Proposed Functionality

The Design Instance name is a mandatory attribute that today is required to be entered by the user.

The proposed functionality will allow the Design to include logic to auto-calculate the name leveraging input data (and other context). This would be dynamically generated when the input data changes (only for creation).

There could be an option to disable the user edition if enforced by design.

Use Case

Environment

• Nautobot version: Any
• design-builder version: latest

Proposed Functionality

It is a very common pattern to have simple queries that are almost always utilizing a natural key lookup. For instance, the Status model is heavily used throughout Nautobot and is required on many models (devices, interfaces, sites, etc). This leads to quite a bit of repetition in Design Builder like so:

sites:
- name: "New Site"
  status__name: "Active"

In this example, the Status object is being assigned by its name Active. It would be nice if design builder could auto-detect when the intent is to lookup by natural key so that the above code becomes:

sites:
- name: "New Site"
  status: "Active"

Environment

• Nautobot version:
• nautobot-design-builder version: 1.1.0

Proposed Functionality

Improve docs by:

• Better understanding of the Context file usage
• More examples of how to create Jinja2 files and use the tags. Easy to copy and paste to get started.
• Easier visualization of Nautobot object attributes to translate into designs. For example, how do I know the field of the object X in Nautobot ORM? (naming can differ from UI and API)

Use Case

Environment

• Nautobot version:
• design-builder version:

Proposed Functionality

As an organization, I may define my remote sites all come out of the IP block say, 10.192.0.0/15. I want to have a method to define this supernet block for a design, and be able to just get the next available block, say /24 out of that block of addresses, without having to define what the supernet is. I just want it in configuration type thing. And maybe support multiple blocks, that when 10.192.0.0/15 is full, I move on to 10.224.0.0/15 type thing.

Use Case

Least number of inputs, most number of outputs.

Environment

• Nautobot version: 1.x/2.x
• design-builder version: future

Proposed Functionality

Some of the proposed functionality requires different behavior and code depending on the major version of Nautobot. We need to develop a way to load the correct version of an extension based on the Nautobot version that is running.

Use Case

See #9

Environment

• Nautobot version:
• design-builder version:

Proposed Functionality

At the Design Instance level, I would like to see a diff of the specific changes that will be applied

Use Case

This way, I have more clear understanding of the impact of the design implementation

Proposed Functionality

The builder should wrap the build process in a transaction so that failures are rolled back even when run outside of a Nautobot job context.

Environment

• Nautobot version: 2.x
• design-builder version: next

Proposed Functionality

Building on issue #52, add the capability to have Design Revisions (versioning).

Designs may evolve over time. Once a design is implemented in the database, it may be necessary to migrate to a newer revision. The DesignJob would start providing reference to the version, and it will be taken by the design implementation. How to migrate from one revision to another could be implemented in different ways (not analyzed in detail here), but it could look similar to what Django database migrations do: from one state to another, it would define how the data should be changed to match the new expectations. This is not a trivial feature, as the new revision may require extra input data, or there would be some data inconsistency to manage.

An easier alternative would be to create a brand new design, so to transition from one to the other, the older one should be decommissioned first. For example, we start with the technical service l3vpn_v1, and after a while, a new l3vpn_v2 is available. In order to deploy the new service to replace the old one, we could decommission the previous service, and then recreate a new one (this could also be done with one design using a revision to understand if the process of decommission and provision is needed). This would be simpler, but it would imply a downtime of the service.

Environment

• Nautobot version: 2.0.0
• nautobot-design-builder version: any

Only applies to v2

Atomic transactions come at a considerable cost when the number of objects are measured in the thousands, this was why atomic transactions were removed in nautobot v2 as they are way to problematic compared to the value they add for most use cases. Design Builder has a valid use case but the total time to generate ~5k objects on generic compute can take 18 hours to run the job and on CPU optimized compute can take over 6 hours for the same job. When comparing the same job to using the API the job is substantially slower than using a separate script with the API.

Proposed Functionality

V2 only job class that removes the atomic transaction

Use Case