[Feature] Kuberay MultiKueue adapter #3892

mszadkow · 2024-12-19T10:53:06Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

We want to be bale to run Kuberay workloads (RayJob, RayCluster) in MultiKueue.

Which issue(s) this PR fixes:

Relates to #3822

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2024-12-19T10:53:09Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2024-12-19T10:53:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mszadkow
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mszadkow · 2024-12-19T10:53:18Z

/ok-to-test

netlify · 2024-12-19T10:53:40Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`ad4df0d`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6763fd8be472e400089303f9

mszadkow · 2024-12-19T10:53:58Z

Makefile-deps.mk

+		chmod -R u+w "$(EXTERNAL_CRDS_DIR)/ray-operator" && \
+		rm -rf "$(EXTERNAL_CRDS_DIR)/ray-operator"; \
+	fi
+	mkdir -p "${EXTERNAL_CRDS_DIR}/ray-operator"; \


Yes, I need all of those folders to be able to properly install it.

mszadkow · 2024-12-19T10:55:36Z

Makefile-test.mk

@@ -70,6 +70,7 @@ IMAGE_TAG ?= $(IMAGE_REPO):$(GIT_TAG)
 JOBSET_VERSION = $(shell $(GO_CMD) list -m -f "{{.Version}}" sigs.k8s.io/jobset)
 KUBEFLOW_VERSION = $(shell $(GO_CMD) list -m -f "{{.Version}}" github.com/kubeflow/training-operator)
 KUBEFLOW_MPI_VERSION = $(shell $(GO_CMD) list -m -f "{{.Version}}" github.com/kubeflow/mpi-operator)
+KUBERAY_VERSION = $(shell $(GO_CMD) list -m -f "{{.Version}}" github.com/ray-project/kuberay/ray-operator)


So far it's the version without managedBy.
We don't have a ray-operator image that supports managedBy, that would have to be a custom one...
latest -> 1.2.2, which is the last release

mszadkow · 2024-12-19T10:55:52Z

config/components/manager/kustomization.yaml

@@ -14,4 +14,4 @@ kind: Kustomization
 images:
 - name: controller
  newName: us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue
-  newTag: main
+  newTag: v0.10.0-rc.3-53-g215d32ad-dirty


ups...will clean it

mszadkow · 2024-12-19T10:58:00Z

hack/e2e-common.sh

+    export KUBERAY_MANIFEST="${ROOT_DIR}/dep-crds/ray-operator/default/"
+    export KUBERAY_IMAGE=bitnami/kuberay-operator:${KUBERAY_VERSION/#v}
+    export KUBERAY_RAY_IMAGE=rayproject/ray:2.9.0
+    export KUBERAY_RAY_IMAGE_ARM=rayproject/ray:2.9.0-aarch64


this one is for us people working on macOS, it's vital for development, not so much for prod - I think it should stay

mszadkow · 2024-12-19T10:58:48Z

hack/e2e-common.sh

@@ -136,6 +155,22 @@ function install_mpi {
    kubectl apply --server-side -f "${KUBEFLOW_MPI_MANIFEST}"
 }

+#$1 - cluster name
+function install_kuberay {
+    # Extra e2e images required for Kuberay, 


that one gives us the speed on test execution, but takes time here...
I am about to measure how much

mszadkow · 2024-12-19T10:59:08Z

pkg/controller/jobs/raycluster/raycluster_multikueue_adapter.go

+	remoteJob.Labels[kueue.MultiKueueOriginLabel] = origin
+
+	// clear the managedBy enables the controller to take over
+	// remoteJob.Spec.ManagedBy = nil


This can be removed np

mszadkow · 2024-12-19T10:59:12Z

pkg/controller/jobs/raycluster/raycluster_multikueue_adapter.go

+}
+
+func (b *multikueueAdapter) IsJobManagedByKueue(ctx context.Context, c client.Client, key types.NamespacedName) (bool, string, error) {
+	// job := rayv1.RayCluster{}


This can be removed np

mszadkow · 2024-12-19T11:02:16Z

In the current stage I need to add RayCluster e2e test

k8s-ci-robot · 2024-12-19T11:03:00Z

@mszadkow: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-test-integration-main	`d27f12c`	link	true	`/test pull-kueue-test-integration-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

mbobrovskyi · 2024-12-19T11:20:57Z

pkg/controller/jobs/rayjob/rayjob_multikueue_adapter.go

+	}
+
+	// if the remote exists, just copy the status
+	if err == nil {


Please also wait for unsuspend. See #3730.

mbobrovskyi · 2024-12-19T11:21:51Z

pkg/controller/jobs/raycluster/raycluster_multikueue_adapter.go

+	}
+
+	// if the remote exists, just copy the status
+	if err == nil {


mbobrovskyi · 2024-12-19T11:36:37Z

test/e2e/multikueue/e2e_test.go

+			if runtime.GOOS == "darwin" {
+				E2eKuberayTestImage = "rayproject/ray:2.9.0-py39-aarch64"
+			}
+			fmt.Println(runtime.GOOS, E2eKuberayTestImage)


Suggested change

fmt.Println(runtime.GOOS, E2eKuberayTestImage)

ginkgo.GinkgoLogr.Info(runtime.GOOS, E2eKuberayTestImage)

Or maybe remove it?

yes, I will remove it, it was for debugging purposes

mbobrovskyi · 2024-12-19T11:43:32Z

test/integration/multikueue/multikueue_test.go


-var _ = ginkgo.Describe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {
+var _ = ginkgo.FDescribe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {


Suggested change

var _ = ginkgo.FDescribe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {

var _ = ginkgo.Describe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {

mbobrovskyi · 2024-12-19T12:02:13Z

pkg/controller/jobs/rayjob/rayjob_controller.go

@@ -87,7 +88,7 @@ func (j *RayJob) IsSuspended() bool {
 }

 func (j *RayJob) IsActive() bool {
-	return j.Status.JobDeploymentStatus != rayv1.JobDeploymentStatusSuspended
+	return j.Status.JobDeploymentStatus != rayv1.JobDeploymentStatusNew


Do we need this changes?

well, that was the most important change actually
Because JobDeploymentStatusSuspended is not a initial status and the workload couldn't be created at all
Maybe it should be both like not new and not suspended, but definitively not the suspended only

I am not even entirely sure if this should watch the JobDeploymentStatus, as the Job has its own Status.

I think you right, it should be both.

kueue/pkg/controller/jobframework/interface.go

Lines 57 to 58 in cea42f2

// IsActive returns true if there are any running pods.

IsActive() bool

cc: @andrewsykim

mszadkow added 2 commits December 19, 2024 11:49

Add the rayjob multikueue adapter

4985129

Add the raycluster multikueue adapter

3b5b50f

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 19, 2024

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 19, 2024

k8s-ci-robot requested review from mimowo and PBundyra December 19, 2024 10:53

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 19, 2024

mszadkow commented Dec 19, 2024

View reviewed changes

mszadkow added 2 commits December 19, 2024 11:57

Introduce Kuberay Multikueue e2e tests

2fa0210

Introduce Kuberay Multikueue integration tests

ad4df0d

mszadkow commented Dec 19, 2024

View reviewed changes

mszadkow force-pushed the feature/kuberay-multikueue-adapter branch from d27f12c to ad4df0d Compare December 19, 2024 11:03

mbobrovskyi reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Kuberay MultiKueue adapter #3892

[Feature] Kuberay MultiKueue adapter #3892

mszadkow commented Dec 19, 2024

k8s-ci-robot commented Dec 19, 2024

k8s-ci-robot commented Dec 19, 2024

mszadkow commented Dec 19, 2024

netlify bot commented Dec 19, 2024 •

edited

Loading

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow commented Dec 19, 2024

k8s-ci-robot commented Dec 19, 2024

mbobrovskyi Dec 19, 2024 •

edited

Loading

mszadkow Dec 19, 2024

mbobrovskyi Dec 19, 2024

mbobrovskyi Dec 19, 2024

mszadkow Dec 19, 2024

mbobrovskyi Dec 19, 2024

mszadkow Dec 19, 2024

mbobrovskyi Dec 19, 2024

mszadkow Dec 19, 2024

mszadkow Dec 19, 2024

mbobrovskyi Dec 19, 2024

mbobrovskyi Dec 20, 2024

	fmt.Println(runtime.GOOS, E2eKuberayTestImage)
	ginkgo.GinkgoLogr.Info(runtime.GOOS, E2eKuberayTestImage)


		var _ = ginkgo.Describe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {
		var _ = ginkgo.FDescribe("Multikueue", ginkgo.Ordered, ginkgo.ContinueOnFailure, func() {

	// IsActive returns true if there are any running pods.
	IsActive() bool

[Feature] Kuberay MultiKueue adapter #3892

Are you sure you want to change the base?

[Feature] Kuberay MultiKueue adapter #3892

Conversation

mszadkow commented Dec 19, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Dec 19, 2024

k8s-ci-robot commented Dec 19, 2024

mszadkow commented Dec 19, 2024

netlify bot commented Dec 19, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mszadkow commented Dec 19, 2024

k8s-ci-robot commented Dec 19, 2024

mbobrovskyi Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Dec 19, 2024 •

edited

Loading

mbobrovskyi Dec 19, 2024 •

edited

Loading