Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor big5, eventdata, http_logs workloads #538

Merged
merged 1 commit into from
Jan 1, 2025

Conversation

OVI3D0
Copy link
Member

@OVI3D0 OVI3D0 commented Dec 18, 2024

Description

Refactors big5, eventdata, and http_logs workloads to make use of the common_operations directory.

Issues Resolved

#489

Testing

  • New functionality includes testing

Tested by refactoring each test procedure and ensuring it runs as it did before refactoring.

Backport to Branches:

  • 6
  • 7
  • 1
  • 2
  • 3

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@OVI3D0 OVI3D0 added backport 2 Backport to the "2" branch backport 1 backport 3 Backport to the "3" branch backport 7 Backport to the "7" branch labels Dec 18, 2024
@rishabh6788
Copy link
Collaborator

rishabh6788 commented Dec 18, 2024

Do not get the intent and benefit of this change, I would prefer all the operations related to a workload to be grouped inside the workload dir. Each can be easily customized limiting to a workload. For e.g., I see that cluster health operation has been refactored to

{
    "name": "check-cluster-health",
    "operation": {
        "operation-type": "cluster-health",
        "index": "{{ index_name }}",
        "request-params": {
            "wait_for_status": "{{cluster_health | default('green')}}",
            "wait_for_no_relocating_shards": "true"
        },
        "retry-until-success": true
    }
}

This will now require user to make sure they add another workload param, why?
I believe we are reducing the flexibility and increasing the burden on the end user.

@OVI3D0
Copy link
Member Author

OVI3D0 commented Dec 19, 2024

This will now require user to make sure they add another workload param, why? I believe we are reducing the flexibility and increasing the burden on the end user.

Not necessarily, index_name is now parameterized but the index name value is provided by the workload.json file. There's no additional burden on the user unless they want to customize the index name. If you test the changes you'll notice that the index name is still provided for each workload without needing to pass in this value.

Do not get the intent and benefit of this change, I would prefer all the operations related to a workload to be grouped inside the workload dir. Each can be easily customized limiting to a workload.

The intent of these changes is to group up all of the common operations so they can be pulled from one single directory. If we want to make changes to one of these operations, we only need to do it once and the changes will be reflected across all workloads. I think if we keep these related operations inside each workloads directory then we're housing a lot of redundancy across the OSB workloads since they are the exact same operations

@IanHoang
Copy link
Collaborator

@rishabh6788 Consolidating the common basic operations from all workloads to a single shared directory should also reduce boilerplate code and allow users to focus on characteristics (operations and test procedures) that are unique to that workload. This would also simplify the experience for end users who are both experienced with and new to OSB.

Open to discussing this more in #489?

@gkamat gkamat merged commit 4a4a5f2 into opensearch-project:main Jan 1, 2025
6 checks passed
@opensearch-trigger-bot
Copy link

The backport to 1 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1 1
# Navigate to the new working tree
pushd ../.worktrees/backport-1
# Create a new branch
git switch --create backport/backport-538-to-1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4a4a5f2ceb9af420951ccce0c82b762c628bee83
# Push it to GitHub
git push --set-upstream origin backport/backport-538-to-1
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1

Then, create a pull request where the base branch is 1 and the compare/head branch is backport/backport-538-to-1.

@opensearch-trigger-bot
Copy link

The backport to 3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-3 3
# Navigate to the new working tree
pushd ../.worktrees/backport-3
# Create a new branch
git switch --create backport/backport-538-to-3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4a4a5f2ceb9af420951ccce0c82b762c628bee83
# Push it to GitHub
git push --set-upstream origin backport/backport-538-to-3
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-3

Then, create a pull request where the base branch is 3 and the compare/head branch is backport/backport-538-to-3.

opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 1, 2025
Signed-off-by: Michael Oviedo <[email protected]>
(cherry picked from commit 4a4a5f2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot
Copy link

The backport to 7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-7 7
# Navigate to the new working tree
pushd ../.worktrees/backport-7
# Create a new branch
git switch --create backport/backport-538-to-7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4a4a5f2ceb9af420951ccce0c82b762c628bee83
# Push it to GitHub
git push --set-upstream origin backport/backport-538-to-7
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-7

Then, create a pull request where the base branch is 7 and the compare/head branch is backport/backport-538-to-7.

IanHoang pushed a commit that referenced this pull request Jan 7, 2025
(cherry picked from commit 4a4a5f2)

Signed-off-by: Michael Oviedo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1 backport 2 Backport to the "2" branch backport 3 Backport to the "3" branch backport 7 Backport to the "7" branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants