Skip to content

Releases: kubeflow/training-operator

v1.8.1 release

10 Sep 15:14
@andreyvelich andreyvelich
v1.8.1
0f8735f
Compare
Choose a tag to compare
v1.8.1 release Latest
Latest

This is the Training Operator v1.8.1 release.

Bug Fixes

  • [Bug] Finish CleanupJob early if the job is suspended (#2243 by @mszadkow)
  • [SDK] Fix trainer error: Update the version of base image and add "num_labels" for downloading pretrained models (#2230 by @helenxie-bit)
  • Update huggingface_hub Version in the storage initializer to fix ImportError (#2180 by @helenxie-bit)

New Contributors

Contributors

mszadkow and helenxie-bit
Assets 2
Loading

v1.8.0 release

23 Jul 18:10
@andreyvelich andreyvelich
v1.8.0
f8687ca
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.8.0 release

This is the Training Operator v1.8.0 release.

This release introduces a new Python API for LLMs Fine-Tuning that simplifies the ability to fine-tune foundational models using distributed PyTorch nodes.

Install the Kubeflow Training SDK as follows to try it:

pip install -U "kubeflow-training[huggingface]"

LLMs Fine-Tuning API

Breaking Changes

New Features

Control Plane Updates

SDK Improvements

Bug Fixes

Misc

Read more

Contributors

garymm, rpemsel, and 23 other contributors
Assets 2
Loading

v1.8.0-rc.0 release

28 Apr 18:37
@johnugeorge johnugeorge
v1.8.0-rc.0
643af3d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.8.0-rc.0 release Pre-release
Pre-release

New features

Bug fixes

Misc

Assets 2
Loading

v1.7.0 release

01 Nov 07:49
@johnugeorge johnugeorge
v1.7.0
5525468
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.7.0 release

Breaking Changes

  • Make scheduler-plugins the default gang scheduler. #1747 (Syulin7)
  • Upgrade the kubernetes dependencies to v1.27 #1834 (tenzen-y)

New features

Bug fixes

  • Fix a bug that XGBoostJob's running condition isn't updated when the job is resumed #1866 (tenzen-y)
  • Set a Running condition when the XGBoostJob is completed and doesn't have a Running condition #1789 (tenzen-y)
  • Avoid to depend on local env when installing the code-generators #1810 (tenzen-y)

Misc

Assets 2
Loading

v1.7.0-rc.0 release

07 Aug 13:00
@johnugeorge johnugeorge
v1.7.0-rc.0
434cef7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.7.0-rc.0 release Pre-release
Pre-release

Breaking Changes

  • Make scheduler-plugins the default gang scheduler. #1747 (Syulin7)
  • Upgrade the kubernetes dependencies to v1.27 #1834 (tenzen-y)

New features

Bug fixes

  • Fix a bug that XGBoostJob's running condition isn't updated when the job is resumed #1866 (tenzen-y)
  • Set a Running condition when the XGBoostJob is completed and doesn't have a Running condition #1789 (tenzen-y)
  • Avoid to depend on local env when installing the code-generators #1810 (tenzen-y)

Misc

Assets 2
Loading

v1.6.0 release

21 Mar 19:37
@johnugeorge johnugeorge
v1.6.0
66aa635
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.6.0 release

Note: Since scheduler-plugins has changed API from sigs.k8s.io with the x-k8s.io, future releases of training operator(v1.7+) will not support scheduler-plugins v0.24.x or lower. Related: #1773

Note: Latest Python SDK 1.6 version does not support earlier training operator versions. The minimum training operator version required is v1.6.0 release. Related: #1702

New Features

Bug fixes

Misc

Closed issues:

  • The default value for CleanPodPolicy is inconsistent. #1753
  • HPA support for PyTorch Elastic #1751
  • Bug: allowance of non DNS-1035 compliant PyTorchJob names results in service creation failures and missing state [#1745](https://github.com/kubeflow/t...
Read more
Assets 2
Loading

v1.6.0-rc.1 release

14 Feb 09:05
@johnugeorge johnugeorge
v1.6.0-rc.1
27e5499
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.6.0-rc.1 release Pre-release
Pre-release

Note: Since scheduler-plugins has changed API from sigs.k8s.io with the x-k8s.io, future releases of training operator(v1.7+) will not support scheduler-plugins v0.24.x or lower

Merged pull requests:

Closed issues:

  • The default value for CleanPodPolicy is inconsistent. #1753
  • HPA support for PyTorch Elastic #1751
  • Bug: allowance of non DNS-1035 compliant PyTorchJob names results in service creation failures and missing state #1745
  • paddle-operator can not get podgroup status(inqueue) with volcano when enable gang #1729
  • *job API(master) cannot compatible with old job [#1725](https://github.com/kubeflow/training-opera...
Read more
Assets 2
Loading

v1.6.0-rc.0 release

26 Jan 13:32
@johnugeorge johnugeorge
v1.6.0-rc.0
b8004ae
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.6.0-rc.0 release Pre-release
Pre-release

v1.6.0-rc.0 release

Assets 2
Loading

v1.5.0 release

19 Aug 13:56
@johnugeorge johnugeorge
v1.5.0
1435d57
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.0 release

Full Changelog

New Features

Bug Fixes

Misc

Assets 2
Loading

v1.5.0-rc.0 release

28 Jun 18:31
@johnugeorge johnugeorge
v1.5.0-rc.0
8c6eab2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.0-rc.0 release Pre-release
Pre-release

Full Changelog

Closed issues:

  • MPIJob worker still running when NotEnoughResources with enable-gang-scheduling==true? #1617
  • unable to fetch TFJob when I use client.go run tfjob #1612
  • Pytorchjob dist-mnist no training logs #1601
  • kubectl get tfjob -o yaml, but not status output #1598
  • missing image in tf_job_design_doc.md #1590
  • Labels in Python client are out of date #1587
  • PyTorchJob Pods "Not Ready" After Completing Training #1577
  • cannot use "github.com/go-openapi/spec".Schema{...} (type "github.com/go-openapi/spec".Schema) as type "k8s.io/kube-openapi/pkg/validation/spec".Schema in field value #1576
  • PyTorchJob: OnFailure Policy won't handle pod failure gracefully #1570
  • pytorchjob doesn't have status.startTIme. #1566
  • Optional-test-infra Deprecation Notice - Training #1561
  • Should we update MPIJob unit test CleanPodPolicy field? #1555
  • --enable-gang-scheduling=true doesn't work for MPIJob #1548
  • PyTorchJob fails when creating a task with a different namespace but the same name #1543
  • Reconcile PyTorchJob error: PyTorchJob.status.replicaStatuses: Invalid value: "null" after enable-gang-scheduling #1538
  • Job TTLs not working #1533
  • Support PodGroup in scheduler-plugins/coscheduling #1518
  • support elastic training #1515
  • Modified the configuration of RootLogger #1514
  • Add checking import order in CI #1510
  • Scale down of pytorchJob cause workers pod to restart #1509
  • Support label selector based success/failure conditions #1507
  • [feat] Support SuccessPolicy in PyTorchJob #1505
  • pytorch elastic scheduler error #1504
  • Could you add the example of MPIJob in this repository #1502
  • [Feature] Create a Informer/ClientSet for PyTorch Jobs #1499
  • [feature] Make init container injection logic availabel to all jobs #1498
  • Roadmaps for 1.4 release #1496
  • [bug] (MpiJob)Init container KubectlDeliveryImage should remain the ability that it can be specified from container parameters or environment variables. #1494
  • Reconcile PyTorch Job error Operation cannot be fulfilled on pytrochjobs.kubeflow.org #1492
  • Python PytorchJob: no attribute openapi_types for example code #1481
  • PyTorch DistributedDataParallel training with multi nodes #1475
  • Installing kubeflow-training breaks import for other kubeflow packages (katib, fairing, etc.) #1471
  • Deprecate ksonnet and use python/golang to submit jobs #1468
  • Help Wanted in ParameterServerStrategy Example. #1459
  • Bug: SomeTimes Coredumped using tfjob #1456
  • [question] PyTorchJob MNIST example training speed #1454
  • tfjob status not match when EnableDynamicWorker set true #1452
  • training-operator set scheduler error #1447
  • [sdk]: Replace TableLogger component in the SDK for better support with ipykernel>=6.x #1446
  • SDK: wait_for_job reports typeError #1445
  • Update prometheus monitoring doc #1443
  • Master branch should provide a nightly image #1433
  • Clean up test folder before testing #1429
  • Clean up TF specific docs #1424
  • [feature] Support SchedulingPolicy in PyTorchJob #1414
  • Hyperlinks in the "Overview" section is incorrect/not found #1411
  • add workqueue metric #1407
  • Validation fails for MXJob Tune example #1402
  • Rate exceeded for aws ecr image #1400
  • change layout to follow the standard of kubebuilder? #1397
  • [example] kubeflow/tf-dist-mnist-test:1.0 is missing in v1.2-branch examples/v1/dist-mnist #1393
  • Update kubeflow/website for 1.4 release #1392
  • Cut beta release of tf-operator for 1.4 release #1385
  • "invalid memory address or nil pointer dereference" #1382
  • some questions about job sync #1379
  • Provides a default Grafana dashboard #1376
  • [feature] Support different PS/worker types #1369
  • Need to copy all (mainly pytorch) framework's example dir to tf-operator/examples #1366
  • Add more CRD validations markers to block invalid job on client apply #1363
  • Update presubmit and post submit job triggers #1354
  • Optimize post submit jobs flow #1353
  • Enable leader election in controller manager using controllermanagerconfig #1350
  • Support mpi jobs in universal operator #1345
  • post-submit job failure in master branch #1343
  • Improve observability of universal operator #1340
  • Best practice to organize main.go and Dockerfile? #1333
  • Should training operator keep clientset in the same repository? #1332
  • Test image has incorrect tag? #1329
  • Prepare e2e tests for all frameworks #1323
  • Reduce e2e replica-restart-policy-tests running time #1319
  • Improve logs structure by consolidating libs from controller runtime and controllers #1313
  • Enable tests for all frameworks #1311
  • [bug] The pod wil be recreated until the expectation expires #1306
  • Upgrade CRDs to apiextensions.k8s.io/v1 #1304
  • Add role details as new columns to kubectl get jobs output for CRD. #1301
  • How to handle long pending pods in a TF-job? #1282
  • Could you release a new version of Python SDK #1279
  • Update swagger.json schema for TFJobSpec to include RunPolicy [#1278](https://github.com/kubeflow...
Read more
Assets 2
Loading