feat: 295 monorepo directory structure design proposal (#389)

* initial directory structure organization * feat: initial work on config and common options * chore: add openssf scorecard workflow (#359) Signed-off-by: Jennifer Power <[email protected]> * build(deps): bump compliance-trestle from 3.4.0 to 3.5.0 (#380) Bumps [compliance-trestle](https://github.com/oscal-compass/compliance-trestle) from 3.4.0 to 3.5.0. - [Release notes](https://github.com/oscal-compass/compliance-trestle/releases) - [Changelog](https://github.com/oscal-compass/compliance-trestle/blob/develop/CHANGELOG.md) - [Commits](oscal-compass/compliance-trestle@v3.4.0...v3.5.0) --- updated-dependencies: - dependency-name: compliance-trestle dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: adds logic to load yaml config into click context to set defautl values. improves config error handling. * feat: adds debug logging statements * feat: add markdown directory creation and call to compliance trestle init * feat: simplify directory creation and better error handling for invalid configs * feat: initial work on autosync * Initial create command for click cli * Initial create command for click cli * adding unit test for config module * adding unit test for config module * Update autosync command * Update autosync command * feat: add ssp index option * feat: add ssp index option * add unit tests for init command * add unit tests for init command * feat: root call create and logging replacement * feat: root call create and logging replacement * feat: add upstream commands, fix common options decorators, expand config * feat: add upstream commands, fix common options decorators, expand config * Update autosync options and add tests * Update autosync options and add tests * docs: adr-001 cli implementation (#347) * docs: adding draft of CLI decision record * docs: adding details around config file * docs: refactor wording for clarity * docs: update config example * expand content for default behaviors around oscal-model * feat: add logic to make_config for nested upstream model and update related tests * feat: add logic to make_config for nested upstream model and update related tests * feat: create command logic for compdef and ssp * feat: create command logic for compdef and ssp * feat: create command updates to prompts and logger messages * feat: create command updates to prompts and logger messages * feat: add default git info to init prompts and config * feat: add default git info to init prompts and config * fix hidden keep file creation * fix hidden keep file creation * Add rule-transform command and unit test * Add rule-transform command and unit test * feat: create command logic and adding unit tests * feat: create command logic and adding unit tests * Fix AttributeError, some misc updates AttributeError: 'NoneType' object has no attribute 'encode' * Fix AttributeError, some misc updates AttributeError: 'NoneType' object has no attribute 'encode' * feat: unit tests added for create command * feat: unit tests added for create command * refactor sync upstreams and autosync to match existing entrypoint syntax * refactor sync upstreams and autosync to match existing entrypoint syntax * Fix AttributeError, some misc updates AttributeError: 'NoneType' object has no attribute 'encode' * Fix AttributeError, some misc updates AttributeError: 'NoneType' object has no attribute 'encode' * feat: unit tests added for create command * feat: unit tests added for create command * fix: docstrings added for create command unit tests * fix: docstrings added for create command unit tests * add file pattern filter * add file pattern filter * fix: updated headers with license and copyright * fix: updated headers with license and copyright * fix: updated logger statements * fix: updated logger statements * fix: logger statements shortened * fix: logger statements shortened * fix: yaml default deletion * fix: yaml default deletion * docs: updates to reference the CLI commands in the README.md * docs: updates to reference the CLI commands in the README.md * feat: update for required ssp name * feat: update for required ssp name * Update trestlebot/cli/commands/init.py Co-authored-by: Jennifer Power <[email protected]> * Update trestlebot/cli/commands/init.py Co-authored-by: Jennifer Power <[email protected]> * Update trestlebot/cli/commands/init.py Co-authored-by: Jennifer Power <[email protected]> * Update trestlebot/cli/commands/init.py Co-authored-by: Jennifer Power <[email protected]> * fix typo in error msg * fix typo in error msg * fix help text for sync upstreams * fix help text for sync upstreams * fix: update for help text and testing location errors * fix: update for help text and testing location errors * fix: update for clarity on profile name for trestle workspace * fix: update for clarity on profile name for trestle workspace * Fix AssertionError, add missing register * Fix AssertionError, add missing register * fix: profile name prompting update * fix: profile name prompting update * feat: updating compdef list to required * feat: updating compdef list to required * docs: change of verbiage for readability * docs: change of verbiage for readability * docs: change to indicate trestle-bot as a cli tool * docs: change to indicate trestle-bot as a cli tool * feat: change to help description of create command * feat: change to help description of create command * docs: added high level folder structure for cli * docs: added high level folder structure for cli * fix: default value returned if no key in dictionary * fix: default value returned if no key in dictionary * feat: align skip-item option to skip-items * feat: align skip-item option to skip-items * fix: add missing git options in create command * fix: add missing git options in create command * fix: refactor testt and remove prompts * fix: refactor testt and remove prompts * fix: formatting issues and typos * fix: formatting issues and typos * chore: update poetry lock with latest dependencies Signed-off-by: George Vauter <[email protected]> * chore: update poetry lock with latest dependencies Signed-off-by: George Vauter <[email protected]> * fix: do not overwrite config path if set Signed-off-by: George Vauter <[email protected]> * fix: do not overwrite config path if set Signed-off-by: George Vauter <[email protected]> * fix: do not overwrite config path if set Signed-off-by: George Vauter <[email protected]> * fix: do not overwrite config path if set Signed-off-by: George Vauter <[email protected]> --------- Signed-off-by: Jennifer Power <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: George Vauter <[email protected]> Co-authored-by: Hannah Braswell <[email protected]> Co-authored-by: Jennifer Power <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Qingmin Duanmu <[email protected]>
complytime · Dec 18, 2024 · 0314389 · 0314389
1 parent 6cd4196
commit 0314389
Show file tree

Hide file tree

Showing 22 changed files with 1,948 additions and 16 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -73,7 +73,9 @@ For workflow diagrams, see the [diagrams](./docs/workflows/) under the `docs` fo
 #### Code structure
 
 - `actions` - Provides specific logic for `trestle-bot` tasks that are packaged as Actions. See [README.md](./actions/README.md) for more information.
-- `entrypoints` - Provides top level logic for specific user-facing tasks. These tasks are not necessarily related in any way so they are not organized into a hierarchical command structure, but they do inherit logic and flags from a base class.
+- `cli` - Provides top level logic for specific user-facing tasks. These tasks are not necessarily related so they are not organized into a hierarchical command structure, but they do share some common modules.
+- `cli/commands` - Provides top level logic for commands and their associated subcommands. The commands are accessed by the single entrypoint `root.py`.
+- `cli/options` - Provides command line options and arguments that are frequently used within `cli/commands`.
 - `provider.py, github.py, and gitlab.py` - Git provider abstract class and concrete implementations for interacting with the API.
 - `tasks` - Pre-tasks can be configured before the main git logic is run. Any task that does workspace management should go here.
 - `tasks/authored` - The `authored` package contains logic for managing authoring tasks for single instances of a top-level OSCAL model. These encapsulate logic from the `compliance-trestle` library and allows loose coupling between `tasks` and `authored` types.

diff --git a/README.md b/README.md
@@ -7,33 +7,33 @@
 
 
 
-trestle-bot assists users in leveraging [Compliance-Trestle](https://github.com/oscal-compass/compliance-trestle) in CI/CD workflows for [OSCAL](https://github.com/usnistgov/OSCAL) formatted compliance content management.
+trestle-bot is a CLI tool that assists users in leveraging [Compliance-Trestle](https://github.com/oscal-compass/compliance-trestle) in CI/CD workflows for [OSCAL](https://github.com/usnistgov/OSCAL) formatted compliance content management.
 
 > WARNING: This project is currently under initial development. APIs may be changed incompatibly from one commit to another.
 
 ## Getting Started
 
 ### Available Commands
 
-The `autosync` command will sync trestle-generated Markdown files to OSCAL JSON files in a trestle workspace. All content under the provided markdown directory when the action is run will be transformed. This action supports all top-level models [supported by compliance-trestle for authoring](https://oscal-compass.github.io/compliance-trestle/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring/).
+The `autosync` command will sync trestle-generated Markdown files to OSCAL JSON files in a trestle workspace. All content under the provided markdown directory will be transformed when the action is run. This action supports all top-level models [supported by compliance-trestle for authoring](https://oscal-compass.github.io/compliance-trestle/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring/).
 
 The `rules-transform` command can be used when managing [OSCAL Component Definitions](https://pages.nist.gov/OSCAL-Reference/models/v1.1.1/component-definition/json-outline/) in a trestle workspace. The action will transform rules defined in the rules YAML view to an OSCAL Component Definition JSON file.
 
-The `create-cd` command can be used to create a new [OSCAL Component Definition](https://pages.nist.gov/OSCAL-Reference/models/v1.1.1/component-definition/json-outline/) in a trestle workspace. The action will create a new Component Definition JSON file and corresponding directories that contain rules YAML files and trestle-generated Markdown files. This action prepares the workspace for use with the `rules-transform` and `autosync` actions.
+The `create compdef` command can be used to create a new [OSCAL Component Definition](https://pages.nist.gov/OSCAL-Reference/models/v1.1.1/component-definition/json-outline/) in a trestle workspace. The action will create a new Component Definition JSON file and corresponding directories that contain rules YAML files and trestle-generated Markdown files. This action prepares the workspace for use with the `rules-transform` and `autosync` actions.
 
-The `sync-upstreams` command can be used to sync and validate upstream OSCAL content stored in a git repository to a local trestle workspace. Which content is synced is determined by the `include_model_names` and `exclude_model_names` inputs.
+The `sync-upstreams` command can be used to sync and validate upstream OSCAL content stored in a git repository to a local trestle workspace. The inputs `include_models` and `exclude_models` determine which content is synced to the trestle workspace.
 
-The `create-ssp` command can be used to create a new [OSCAL System Security Plans](https://pages.nist.gov/OSCAL-Reference/models/v1.1.1/system-security-plan/json-outline/) (SSP) in a trestle workspace. The action will create a new SSP JSON file and corresponding directories that contain trestle-generated Markdown files. This action prepares the workspace for use with the `autosync` action by creating or updating the `ssp-index.json` file. The `ssp-index.json` file is used to track the relationships between the SSP and the other OSCAL content in the workspace for the `autosync` action.
+The `create ssp` command can be used to create a new [OSCAL System Security Plans](https://pages.nist.gov/OSCAL-Reference/models/v1.1.1/system-security-plan/json-outline/) (SSP) in a trestle workspace. The action will create a new SSP JSON file and corresponding directories that contain trestle-generated Markdown files. This action prepares the workspace for use with the `autosync` action by creating or updating the `ssp-index.json` file. The `ssp-index.json` file is used to track the relationships between the SSP and the other OSCAL content in the workspace for the `autosync` action.
 
 Below is a table of the available commands and their current availability as a GitHub Action:
 
-| Command            | Available as a GitHub Action |
-|--------------------|------------------------------|
-| `autosync`         | &#10003;                     |
-| `rules-transform`  | &#10003;                     |                   
-| `create-cd`        | &#10003;                     |
-| `sync-upstreams`   | &#10003;                     |
-| `create-ssp`       |                              |
+| Command           | Available as a GitHub Action |
+|-------------------|------------------------------|
+| `autosync`        | &#10003;                     |
+| `rules-transform` | &#10003;                     |                   
+| `create compdef`  | &#10003;                     |
+| `sync-upstreams`  | &#10003;                     |
+| `create ssp`      |                              |
 
 For detailed documentation on how to use each action, see the README.md in each folder under [actions](./actions/).
 
@@ -47,7 +47,7 @@ provider information is supported for GitHub Actions (GitHub) and GitLab CI (Git
 
 ### Run as a Container
 
-> Note: When running the commands in a container, all are prefixed with `trestlebot` (e.g. `trestlebot-autosync`). The default entrypoint for the container is the autosync command.
+> Note: When running the commands in a container, all are prefixed with `trestlebot` (e.g. `trestlebot autosync`). The default entrypoint for the container is the autosync command.
 
 Build and run the container locally:
 
@@ -72,4 +72,4 @@ This project is licensed under the Apache 2.0 License - see the [LICENSE.md](LIC
 
 ## Troubleshooting
 
-See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for troubleshooting tips.
+See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for troubleshooting tips.
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -48,6 +48,7 @@ flake8-print = "^5.0.0"
 pre-commit = "^3.4.0"
 mkdocs-material = "^9.5.43"
 markdown-include = "^0.8.1"
+types-pyyaml = "^6.0.12.20240917"
 
 [tool.poetry.group.tests]
 optional = true

diff --git a/tests/trestlebot/cli/test_autosync_cmd.py b/tests/trestlebot/cli/test_autosync_cmd.py
@@ -0,0 +1,94 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2024 Red Hat, Inc.
+
+
+"""Testing module for trestlebot autosync command"""
+import pathlib
+from typing import Tuple
+
+from click.testing import CliRunner
+from git import Repo
+
+from trestlebot.cli.commands.autosync import autosync_cmd
+from trestlebot.cli.config import TrestleBotConfig, write_to_file
+
+
+def test_invalid_oscal_model(tmp_repo: Tuple[str, Repo]) -> None:
+    """Test invalid OSCAl model option."""
+
+    repo_path, _ = tmp_repo
+    runner = CliRunner()
+    result = runner.invoke(
+        autosync_cmd,
+        [
+            "--oscal-model",
+            "invalid",
+            "--repo-path",
+            repo_path,
+            "--markdown-dir",
+            "markdown",
+            "--branch",
+            "main",
+            "--committer-name",
+            "Test User",
+            "--committer-email",
+            "[email protected]",
+        ],
+    )
+    assert "Invalid value for '--oscal-model'" in result.output
+    assert result.exit_code == 2
+
+
+def test_missing_ssp_index_file_option(tmp_repo: Tuple[str, Repo]) -> None:
+    """Test missing ssp_index_file option for autosync ssp."""
+    repo_path, _ = tmp_repo
+    runner = CliRunner()
+    cmd_options = [
+        "--oscal-model",
+        "ssp",
+        "--repo-path",
+        repo_path,
+        "--markdown-dir",
+        "markdown",
+        "--branch",
+        "main",
+        "--committer-name",
+        "Test User",
+        "--committer-email",
+        "[email protected]",
+    ]
+    result = runner.invoke(autosync_cmd, cmd_options)
+    assert result.exit_code == 1
+    assert "Missing option '--ssp-index-file'" in result.output
+
+
+def test_missing_markdown_dir_option(tmp_repo: Tuple[str, Repo]) -> None:
+    # When no markdown_dir setting in trestlebot config file.
+    repo_path, _ = tmp_repo
+    runner = CliRunner()
+    filepath = pathlib.Path(repo_path).joinpath("config.yml")
+    config_obj = TrestleBotConfig(repo_path=repo_path)
+    write_to_file(config_obj, filepath)
+    cmd_options = [
+        "--oscal-model",
+        "compdef",
+        "--repo-path",
+        repo_path,
+        "--branch",
+        "main",
+        "--committer-name",
+        "Test User",
+        "--committer-email",
+        "[email protected]",
+        "--config",
+        str(filepath),
+    ]
+    result = runner.invoke(autosync_cmd, cmd_options)
+    assert result.exit_code == 2
+    assert "Error: Missing option '--markdown-dir'" in result.output
+
+    # With 'markdown_dir' setting in config.yml
+    config_obj = TrestleBotConfig(markdown_dir="markdown")
+    write_to_file(config_obj, filepath)
+    result = runner.invoke(autosync_cmd, cmd_options)
+    assert result.exit_code == 0
diff --git a/tests/trestlebot/cli/test_config.py b/tests/trestlebot/cli/test_config.py
@@ -0,0 +1,80 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2024 Red Hat, Inc.
+
+
+"""Unit tests for CLI config module"""
+import pathlib
+
+import pytest
+import yaml
+
+from trestlebot.cli.config import (
+    TrestleBotConfig,
+    TrestleBotConfigError,
+    UpstreamsConfig,
+    load_from_file,
+    make_config,
+    write_to_file,
+)
+
+
+@pytest.fixture
+def config_obj() -> TrestleBotConfig:
+    return TrestleBotConfig(
+        repo_path="/tmp",
+        markdown_dir="markdown",
+        upstreams=UpstreamsConfig(sources=["repo@main"]),
+    )
+
+
+def test_invalid_config_raises_errors() -> None:
+    """Test create config with invalid directory to raise error."""
+
+    with pytest.raises(TrestleBotConfigError) as ex:
+        _ = make_config(dict(repo_path="0"))
+
+    assert (
+        str(ex.value)
+        == "Invalid config value for repo_path. Path does not point to a directory."
+    )
+
+
+def test_make_config_raises_no_errors(tmp_init_dir: str) -> None:
+    """Test create a valid config object."""
+    values = {
+        "repo_path": tmp_init_dir,
+        "markdown_dir": "markdown",
+        "committer_name": "committer-name",
+        "committer_email": "committer-email",
+        "upstreams": {"sources": ["https://test@main"], "skip_validation": True},
+    }
+    config = make_config(values)
+    assert isinstance(config, TrestleBotConfig)
+    assert config.upstreams is not None
+    assert config.upstreams.sources == ["https://test@main"]
+    assert config.upstreams.skip_validation is True
+    assert config.repo_path == pathlib.Path(tmp_init_dir)
+    assert config.markdown_dir == values["markdown_dir"]
+    assert config.committer_name == values["committer_name"]
+    assert config.committer_email == values["committer_email"]
+
+
+def test_config_write_to_file(config_obj: TrestleBotConfig, tmp_init_dir: str) -> None:
+    """Test config is written to yaml file."""
+    filepath = pathlib.Path(tmp_init_dir).joinpath("config.yml")
+    write_to_file(config_obj, filepath)
+    with open(filepath, "r") as f:
+        yaml_data = yaml.safe_load(f)
+
+    assert yaml_data == config_obj.to_yaml_dict()
+
+
+def test_config_load_from_file(config_obj: TrestleBotConfig, tmp_init_dir: str) -> None:
+    """Test config is read from yaml file into config object."""
+    filepath = pathlib.Path(tmp_init_dir).joinpath("config.yml")
+    with filepath.open("w") as config_file:
+        yaml.dump(config_obj.to_yaml_dict(), config_file)
+
+    config = load_from_file(filepath)
+    assert isinstance(config, TrestleBotConfig)
+    assert config == config_obj