1.2.0 release (#144)

* Updated lib mapping docs (#113)

* Updated lib mapping docs

* Updated per review suggestions

* Pinned mistune for m2r2 (#114)

* Switch to poetry-core

* Update banner parsing for EOS (#121)

Update banner parsing for EOS

* Fixes encrypt type7 (#122)

* fixes encrypt type7

* Document support on duplicate lines (#128)

* Few updates to PR of 125 (#129)

* initial pass at type hinting and getting mypy to run

* update tasks and ci to do mypy as well

* rm unused import

* fix contributing end-line number

* fix contributing end-line number

* Add py.typed file to signify availability of type hints to third parties.

* Updates to type hinting.

* Add EOS _build_banner type hints.

* Address feedback.

* Linting.

* Remove .idea from commit and ignore in .gitignore.

* Remove remnants of merge conflict.

* PR feedback.

* PR comments.

* Ignore pylint errors for abstract-method.

* Black.

* add examples and better exception messaging

Co-authored-by: Leo Kirchner <[email protected]>

* Adding nxos_ssh to Napalm mapper (#138)

* Update CODEOWNERS (#134)

adding codeowner

* prep for 1.2.0 release (#143)

* prep for 1.2.0 release

* Update CHANGELOG.md

Co-authored-by: Jeff Kala <[email protected]>

Co-authored-by: Jeff Kala <[email protected]>

Co-authored-by: Adam Byczkowski <[email protected]>
Co-authored-by: Fabian Affolter <[email protected]>
Co-authored-by: Ken Celenza <[email protected]>
Co-authored-by: Dr. X <[email protected]>
Co-authored-by: Leo Kirchner <[email protected]>
Co-authored-by: Joe Wesch <[email protected]>
Co-authored-by: Andrew Bates <[email protected]>

Author: 8 people

.github/CODEOWNERS

@@ -1,2 +1,2 @@
 # Default owner(s) of all files in this repository
-* @itdependsnetworks @jeffkala @qduk
+* @itdependsnetworks @jeffkala @qduk @abates

.github/workflows/ci.yml

@@ -32,6 +32,19 @@ jobs:
         run: "poetry run invoke bandit"
     needs:
       - "black"
+  mypy:
+    runs-on: "ubuntu-20.04"
+    env:
+      INVOKE_LOCAL: "True"
+    steps:
+      - name: "Check out repository code"
+        uses: "actions/checkout@v2"
+      - name: "Setup environment"
+        uses: "networktocode/gh-action-setup-poetry-environment@v2"
+      - name: "Type-Hints: mypy"
+        run: "poetry run invoke mypy"
+    needs:
+      - "black"
   pydocstyle:
     runs-on: "ubuntu-20.04"
     env:
@@ -103,6 +116,7 @@ jobs:
             PYTHON_VER=${{ env.PYTHON_VER }}
     needs:
       - "bandit"
+      - "mypy"
       - "pydocstyle"
       - "flake8"
       - "yamllint"

.gitignore

@@ -192,48 +192,11 @@ $RECYCLE.BIN/
 # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
 # Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
 
-# User-specific stuff
-.idea/**/workspace.xml
-.idea/**/tasks.xml
-.idea/**/usage.statistics.xml
-.idea/**/dictionaries
-.idea/**/shelf
-
-# Generated files
-.idea/**/contentModel.xml
-
-# Sensitive or high-churn files
-.idea/**/dataSources/
-.idea/**/dataSources.ids
-.idea/**/dataSources.local.xml
-.idea/**/sqlDataSources.xml
-.idea/**/dynamic.xml
-.idea/**/uiDesigner.xml
-.idea/**/dbnavigator.xml
-
-# Gradle
-.idea/**/gradle.xml
-.idea/**/libraries
-
-# Gradle and Maven with auto-import
-# When using Gradle or Maven with auto-import, you should exclude module files,
-# since they will be recreated, and may cause churn.  Uncomment if using
-# auto-import.
-# .idea/artifacts
-# .idea/compiler.xml
-# .idea/jarRepositories.xml
-# .idea/modules.xml
-# .idea/*.iml
-# .idea/modules
-# *.iml
-# *.ipr
+.idea/
 
 # CMake
 cmake-build-*/
 
-# Mongo Explorer plugin
-.idea/**/mongoSettings.xml
-
 # File-based project format
 *.iws
 
@@ -246,51 +209,12 @@ out/
 # JIRA plugin
 atlassian-ide-plugin.xml
 
-# Cursive Clojure plugin
-.idea/replstate.xml
-
 # Crashlytics plugin (for Android Studio and IntelliJ)
 com_crashlytics_export_strings.xml
 crashlytics.properties
 crashlytics-build.properties
 fabric.properties
 
-# Editor-based Rest Client
-.idea/httpRequests
-
-# Android studio 3.1+ serialized cache file
-.idea/caches/build_file_checksums.ser
-
-### PyCharm Patch ###
-# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
-
-# *.iml
-# modules.xml
-# .idea/misc.xml
-# *.ipr
-
-# Sonarlint plugin
-# https://plugins.jetbrains.com/plugin/7973-sonarlint
-.idea/**/sonarlint/
-
-# SonarQube Plugin
-# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin
-.idea/**/sonarIssues.xml
-
-# Markdown Navigator plugin
-# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced
-.idea/**/markdown-navigator.xml
-.idea/**/markdown-navigator-enh.xml
-.idea/**/markdown-navigator/
-
-# Cache file creation bug
-# See https://youtrack.jetbrains.com/issue/JBR-2257
-.idea/$CACHE_FILE$
-
-# CodeStream plugin
-# https://plugins.jetbrains.com/plugin/12206-codestream
-.idea/codestream.xml
-
 ### vscode ###
 .vscode/*
 *.code-workspace

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## v1.2.0 - 2022-08
+
+### Added
+
+- #128 Documentation for Nokia SROS duplicate lines/duplicate line detection
+- #138 Added nxos_ssh to Napalm mapper
+
+### Changed
+
+- #113 Updated lib mapping docs
+- #115 Switched build backend to poetry-core
+- #121 Update banner parsing for EOS
+- #129 Add type hints to the whole project and mypy testing setup and CI
+- #134 Updated CODEOWNERS
+
+### Fixed
+
+- #122 Fixed encrypt type7
+
 ## v1.1.0 - 2022-04
 
 ### Added
@@ -23,7 +42,7 @@
 
 ### Added
 
-- #69 Normalise banner demiliter for IOS to ^C & support parsing delimiter ^
+- #69 Normalise banner delimiter for IOS to ^C & support parsing delimiter ^
 
 ### Fixed
 
@@ -113,7 +132,7 @@
 
 ### Fixed
 
-- Enable docsting tests
+- Enable docstring tests
 - Fix docstring tests
 - Fix wording and links on README
 

README.md

@@ -24,7 +24,7 @@ Functions are grouped with like functions, such as IP or MAC address based funct
   * Compliance - Provides the ability to compare two configurations to sanely understand the differences.
   * Parsing - Provides the ability to parse configuration for the minor differences that are there.
 * DNS - Provides the ability to work with DNS, such as validating that a FQDN is resolvable.
-* Interface - Provides the ability to work with interface names, expanding, abbreviating, and spliting the names.
+* Interface - Provides the ability to work with interface names, expanding, abbreviating, and splitting the names.
 * IP Address - Provides the ability to work with IP addresses, primarily exposing Python `ipaddress` functionality.
 * Library Mapper - Provides mappings in expected vendor names between Netmiko, NAPALM, pyntc, ntc-templates, pyats, and scrapli.
 * MAC Address - Provides the ability to work with MAC addresses such as validating or converting to integer.
@@ -152,15 +152,15 @@ Except for unit tests, testing is only supported on Python 3.7.
 The project is packaged with a light development environment based on `docker-compose` to help with the local development of the project and to run tests within TravisCI.
 
 The project is following Network to Code software development guidelines and are leveraging the following:
-- Black, Pylint, Bandit, flake8, and pydocstyle for Python linting and formatting.
+- Black, Pylint, Bandit, Mypy, flake8, and pydocstyle for Python linting and formatting.
 - pytest, coverage, and unittest for unit tests.
 
 There are a number of things that are required in order to have a successful PR.
 
 - All new functions must contain at least 1 example in their docstrings.
 - Docstrings must conform to the google docstring [convention](https://google.github.io/styleguide/pyguide.html#381-docstrings).
 - Unit test for newly added functions are required.
-- If applicable, tests related to config parsing and compliuance must be added.
+- If applicable, tests related to config parsing and compliance must be added.
 - Update the jinja2 filter (netutils.utils.jinja2_convenience_function) for any new functions (see below for details).
 - If you create a new file in the `netutils` folder, you must create a new folder and `index.rst` in the docs folder (see below for details).
 - Your PR must not introduce any required dependencies. You can introduce optional or development dependencies.
@@ -219,6 +219,7 @@ Each command can be executed with `invoke <command>`. Each command also has its
   black              Run black to check that Python files adhere to its style standards.
   coverage           Run the coverage report against pytest.
   flake8             Run flake8 to check that Python files adhere to its style standards.
+  mypy               Run mypy to validate typing-hints.
   pylint             Run pylint code analysis.
   pydocstyle         Run pydocstyle to validate docstring formatting adheres to NTC defined standards.
   pytest             Run pytest for the specified name and Python version.

docs/requirements.txt

@@ -1,4 +1,5 @@
 m2r2==0.2.7
+mistune==0.8.4
 Sphinx
 toml
 sphinx-rtd-theme

docs/source/conf.py

@@ -13,12 +13,16 @@
 import os
 import sys
 import toml
+from jinja2 import Environment, FileSystemLoader
+
 
 sys.path.insert(0, os.path.abspath("../.."))
+from netutils import lib_mapper  # noqa: E402
+
+DIR_PATH = os.path.dirname(os.path.realpath(__file__))
 sys.path.append(os.path.abspath("sphinxext"))
 toml_dict = toml.load("../../pyproject.toml")
 
-
 # -- Project information -----------------------------------------------------
 
 project = toml_dict["tool"]["poetry"]["name"]
@@ -63,3 +67,29 @@
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+
+
+def build_mapping_tables(app):
+    """Build the library mappings tables."""
+    env = Environment(loader=FileSystemLoader(f"{DIR_PATH}"))
+    template_file = env.get_template("table_template.j2")
+
+    LIST_OF_MAP_DICTS = []
+    for attr in dir(lib_mapper):
+        if (attr.endswith("MAPPER_REVERSE") or attr.endswith("_MAPPER")) and not (
+            attr.startswith("_") or attr.startswith("NETMIKO") or attr.startswith("MAIN")
+        ):
+            LIST_OF_MAP_DICTS.append(attr)
+
+    for dict_name in LIST_OF_MAP_DICTS:
+        lib_name = dict_name.split("_")[0]
+        filename = f"{lib_name}_reverse" if "REVERSE" in dict_name else lib_name
+        headers = ["NORMALIZED", lib_name] if "REVERSE" in dict_name else [lib_name, "NORMALIZED"]
+        rendered_template = template_file.render(lib_names=headers, mappings=getattr(lib_mapper, dict_name))
+        with open(f"{DIR_PATH}/netutils/lib_mapping/{filename}_table.rst", "w") as table_file:
+            table_file.write(rendered_template)
+
+
+def setup(app):
+    """Call methods during builder initiated."""
+    app.connect("builder-inited", build_mapping_tables)

docs/source/contributing/index.rst

@@ -4,5 +4,5 @@ Contributing
 
 .. mdinclude:: ../../../README.md
    :start-line: 148
-   :end-line: 233
+   :end-line: 234
 

docs/source/netutils/configs/index.rst

@@ -25,3 +25,25 @@ F5 Parser
 Nokia SROS Parser
 -----------------
 - The section banners have been simplified to extract the section header itself. This means that `echo "System Configuration"` will be converted to just "System Configuration".
+
+Duplicate Line Detection
+--------------------------
+In some circumstances replacing lines, such as secrets without uniqueness in the replacement, will result in duplicated lines that are invalid configuration, such as::
+
+    snmp-server community <<REPLACED>> RO SNMP_ACL_RO
+    snmp-server community <<REPLACED>> RO SNMP_ACL_RO
+
+There are some known use cases, such as the below that are considered::
+
+    router bgp 6500
+     bgp router-id 10.0.0.11
+     !
+     address-family ipv4 unicast
+      redistribute connected
+     exit-address-family <--- duplicated hierarchy
+     !
+     address-family l2vpn evpn
+      neighbor underlay activate
+     exit-address-family <--- duplicated hierarchy
+
+Documented use cases that are actual configuration on a network device are considered valid and should be opened for bug fixes. However, configuration that does not actually exist on the running config of network devices are out of scope for the parser.

docs/source/netutils/lib_mapping/ANSIBLE_reverse_table.rst

@@ -0,0 +1,78 @@
+.. raw:: html
+
+	<table border="1" class="docutils">
+		<th>NORMALIZED</th><th>ANSIBLE</th>
+		
+		<tr>
+			<td>arista_eos</td><td>arista.eos.eos</td>
+		</tr>
+		<tr>
+			<td>ciena_saos</td><td>ciena.saos6.saos6</td>
+		</tr>
+		<tr>
+			<td>cisco_asa</td><td>cisco.asa.asa</td>
+		</tr>
+		<tr>
+			<td>cisco_ios</td><td>cisco.ios.ios</td>
+		</tr>
+		<tr>
+			<td>cisco_xr</td><td>cisco.iosxr.iosxr</td>
+		</tr>
+		<tr>
+			<td>cisco_nxos</td><td>cisco.nxos.nxos</td>
+		</tr>
+		<tr>
+			<td>huawei</td><td>community.network.ce</td>
+		</tr>
+		<tr>
+			<td>dell_os6</td><td>dellemc.os6.os6</td>
+		</tr>
+		<tr>
+			<td>dell_os9</td><td>dellemc.os9.os9</td>
+		</tr>
+		<tr>
+			<td>dell_os10</td><td>dellemc.os10.0s10</td>
+		</tr>
+		<tr>
+			<td>ericsson_ipos</td><td>community.network.eric_eccli</td>
+		</tr>
+		<tr>
+			<td>extreme_exos</td><td>community.network.exos</td>
+		</tr>
+		<tr>
+			<td>extreme_netiron</td><td>community.network.ironware</td>
+		</tr>
+		<tr>
+			<td>extreme_nos</td><td>community.network.nos</td>
+		</tr>
+		<tr>
+			<td>extreme_slx</td><td>community.network.slxos</td>
+		</tr>
+		<tr>
+			<td>extreme_vsp</td><td>community.network.voss</td>
+		</tr>
+		<tr>
+			<td>juniper_junos</td><td>junipernetworks.junos.junos</td>
+		</tr>
+		<tr>
+			<td>lenovo_cnos</td><td>community.network.cnos</td>
+		</tr>
+		<tr>
+			<td>lenovo_enos</td><td>community.network.enos</td>
+		</tr>
+		<tr>
+			<td>mikrotik_routeros</td><td>community.network.routeros</td>
+		</tr>
+		<tr>
+			<td>nokia_sros</td><td>community.network.sros</td>
+		</tr>
+		<tr>
+			<td>pluribus</td><td>community.network.netvisor</td>
+		</tr>
+		<tr>
+			<td>ruckus_icx</td><td>community.network.icx</td>
+		</tr>
+		<tr>
+			<td>vyos</td><td>vyos.vyos.vyos</td>
+		</tr>
+	</table>