Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make test.py run tests using the inductor backend #3

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

gbeckenkamp
Copy link

@gbeckenkamp gbeckenkamp commented Dec 17, 2024

As it stands, test.py is being used on nightly and pull request workflows in eager mode only (the default for the Model class). As inductor is a valid working backend that is also supported by all platforms, we should also validate it via real models in torchbench, specially in PRs.
This should expose pre-existing issues with the current inductor implementation on main and improve the torch.compile coverage, while not extensively increasing the time that PRs take to run with new tests.
This change allows for the discovery of both eager and inductor backends for eval and train tests. As the current implementation of the skip mechanism only account for eval and train being implemented or not, and assuming that any operation that works on eager should also work via inductor, we only let the metadata skip the test if the eval or train steps are not implemented for a particular method.
Test collection for each model should look like this now:

pytest test.py -k "alexnet" --ignore_machine_config --co

  <Module test.py>
    <UnitTestCase TestBenchmark>
      <TestCaseFunction test_alexnet_check_device_cpu_eager>
      <TestCaseFunction test_alexnet_check_device_cpu_inductor>
      <TestCaseFunction test_alexnet_eval_cpu_eager>
      <TestCaseFunction test_alexnet_eval_cpu_inductor>
      <TestCaseFunction test_alexnet_example_cpu_eager>
      <TestCaseFunction test_alexnet_example_cpu_inductor>
      <TestCaseFunction test_alexnet_train_cpu_eager>
      <TestCaseFunction test_alexnet_train_cpu_inductor>

@gbeckenkamp gbeckenkamp marked this pull request as ready for review December 17, 2024 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant