Add A is B and B is A Eval

[mmtmn]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access be granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject it since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Also, please note that we're using Git LFS for storing the JSON files, so please make sure that you move the JSON file to Git LFS before submitting a PR. Details on how to use Git LFS are available here.

Eval details 📑

Eval name

ab

Eval description

This evaluation aims to assess the model's ability to correctly identify and understand the relationship between two entities, where A is a specific entity (which could be a chemical element, a painting, a bird species, a star, a mountain, a novel, a river, or a musical instrument) and B is a unique characteristic or fact about that entity. The model should be able to accurately interpret the user's query about the entity (A) and provide a relevant fact (B), and vice versa. This evaluation will help in fine-tuning the model's understanding of context, relation between entities, and its ability to provide accurate and relevant responses. The entities and their characteristics have been chosen to be specific and challenging.

What makes this a useful eval?

This evaluation is important for several reasons:

1. Contextual Understanding: It tests the model's ability to understand the context of a conversation, particularly the relationship between two related entities (A and B).

2. Accuracy: It assesses the model's ability to provide accurate and relevant information based on the user's query.

3. Relevance: It evaluates the model's ability to understand the relevance of A to B and vice versa, which is crucial in providing meaningful and coherent responses.

4. General Knowledge: This tests the model's knowledge about various specific entities and their unique characteristics or facts, which is a part of its general knowledge. The entities span a wide range of categories, including chemistry, art, ornithology, astronomy, geography, literature, and music, making this a comprehensive test of the model's general knowledge.

5. Versatility: This checks the model's ability to handle different types of queries, as the user can ask about the specific entity (A) or its unique characteristic or fact (B). This tests the model's flexibility in understanding and responding to a variety of query structures across diverse knowledge domains.

gpt-3.5-turbo
[2023-10-10 22:42:42,192] [oaieval.py:229] Final report:
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/A: 2
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/B: 11
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/D: 2
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/C: 1

gpt-4
[2023-10-10 22:43:37,306] [oaieval.py:229] Final report:
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/C: 7
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/B: 7
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/D: 2

class: modelgraded.classify:ModelBasedClassify
eval_type: cot_classify
modelgraded_spec: fact

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 15 high-quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

A popular paper has been going around called 'The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”'. I figured an Eval on this would have a unique value.

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your YAML is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the commits on the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgment

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and the high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.

Submit eval

• I have filled out all required fields of this form
• I have used Git LFS for the Eval JSON data
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that mypy, black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "The user will ask you a question about a specific chemical element that has the atomic number 79, please respond to the best of your abilities."}, {"role": "user", "content": "What is the name of the element with atomic number 79?"}], "ideal": "The element with atomic number 79 is Gold."}
{"input": [{"role": "system", "content": "The user will ask you a question about a chemical element that is often associated with jewelry and has the symbol Au, please respond to the best of your abilities."}, {"role": "user", "content": "Which element is this?"}], "ideal": "This element is Gold."}
{"input": [{"role": "system", "content": "The user will ask you a question about a specific painting that was stolen from the Isabella Stewart Gardner Museum in 1990, please respond to the best of your abilities."}, {"role": "user", "content": "What is the name of the Vermeer painting stolen from the Isabella Stewart Gardner Museum in 1990?"}], "ideal": "The name of the Vermeer painting stolen from the Isabella Stewart Gardner Museum in 1990 is 'The Concert'."}
{"input": [{"role": "system", "content": "The user will ask you a question about a Vermeer painting that is still missing after being stolen from a museum in Boston, please respond to the best of your abilities."}, {"role": "user", "content": "Which painting is this?"}], "ideal": "This painting is 'The Concert'."}
{"input": [{"role": "system", "content": "The user will ask you a question about a specific species of bird that is known for its elaborate courtship dance, please respond to the best of your abilities."}, {"role": "user", "content": "What is the name of the bird species known for its moonwalk-like courtship dance?"}], "ideal": "The bird species known for its moonwalk-like courtship dance is the Blue Manakin."}
{"input": [{"role": "system", "content": "The user will ask you a question about a bird species that is known for its blue color and unique courtship dance, please respond to the best of your abilities."}, {"role": "user", "content": "Which bird species is this?"}], "ideal": "This bird species is the Blue Manakin."}

[andrew-openai]

Thanks for the eval submission.

As it stands, I feel like there might be a few issues with this eval:

• questions like "What is [city] known for?" might have many correct answers, so the current grading will be noisy (most will be superset/subset) and it would be unclear to judge if those are good things.
• The facts used in your dataset are pretty common / well-known and might not fully demonstrate the A is B / B is A effect for large models.

[mmtmn]

@andrew-openai, thanks for the eval revision.

I've updated the data. If possible, please let me know if this update addresses the issues or if there are more issues to be addressed. I've updated the Eval JSON data in the opening pull request comment as well.

Here are the updated results:

gpt-3.5-turbo
[2023-10-10 22:42:42,192] [oaieval.py:229] Final report:
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/A: 2
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/B: 11
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/D: 2
[2023-10-10 22:42:42,192] [oaieval.py:231] counts/C: 1

gpt-4
[2023-10-10 22:43:37,306] [oaieval.py:229] Final report:
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/C: 7
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/B: 7
[2023-10-10 22:43:37,306] [oaieval.py:231] counts/D: 2

Edit: I've also noticed the eval description needed to be updated with the data changes, as well as some parts of the pull request opening comment, and so I did.

[usama-openai]

Thanks for submitting this eval! This PR looks good. I'm approving this PR.

[andrew-openai]

This looks great, approving + merging