This repository contains the extended versions of prompts used in the research paper titled "Challenging LLMs Beyond Information Retrieval: Reasoning Degradation with Long Context Windows." The study explores how Large Language Models (LLMs) handle reasoning tasks as input size increases, and documents how their performance degrades as the context window lengthens.
The paper presents three additional tests:
- Highlight Inefficient Code
- Decrypting Cryptography from a Clue
- Unlock $100.00
These tests were designed to challenge LLMs in different reasoning tasks, demonstrating that while the model performs well with shorter prompts, its accuracy diminishes as the length of the prompt increases.
The primary goal of this repository is to provide transparency and reproducibility for researchers and practitioners interested in the study of LLM performance. It includes the long versions of the prompts used in these additional tests, which were not fully presented in the published paper due to space constraints.
The repository is organized as follows:
/prompts/: Contains the text files for each of the three tests. Each file has the long version of the prompt used in the paper.
-
Clone the Repository:
git clone https://github.com/natanaelwf/LLM_AdditionalTests_LongPrompts.git
-
Explore the Prompts: Navigate to the
/prompts/directory to review the different prompts.
This repository is licensed under the MIT License.