Autoformalization of Game Descriptions using Large Language Models

Overview

This is a framework for autoformalising natural language game-theoretical scenarios into their Prolog specification using Large Language Models. The framework and its evaluation are described in more detail in this paper.

Installation

To install the required dependencies run:

pip install -r requirements.txt

The framework requires also SWI-Prolog to be installed.

Usage

To run the sample experiment, use the following command in your terminal:

python experiment.py

You can modify the parameters of the experiment by modifying params.ini. To use GPT-4 used by default in the experiment, the API key has to be stored in an environment variable.

Project Structure

The structure of the project is as follows:

.
├── CONFIG/
├── DATA/
├── GAMES/
│   └── 2x2/
│   └── GENERALISATION/
|   └── INCOMPLETE/
├── OUTPUT/
│   └── axioms/
│   └── logs/
|   └── prompts/
├── llms/
│   ├── gpt4.py
├── src/
│   ├── GameStatus.py
│   ├── base_llm.py
│   ├── game_formaliser.py
│   ├── setup_logger.py
│   ├── solver.pl
│   ├── solver.py
│   ├── utils.py
├── experiment.py

Authors

Agnieszka Mensfelt
Kostas Stathis
Vince Trencsenyi

Citing This Work

@inproceedings{mensfelt24a,
  author       = {Agnieszka Mensfelt and
                  Kostas Stathis and
                  Vince Tencsenyi},
  title        = {Autoformalization of {G}ame {D}escriptions Using {L}arge {L}anguage {M}odels},
  booktitle    = {1st {I}nternational {W}orkshop on {N}ext-{G}eneration {L}anguage {M}odels for {K}nowledge {R}epresentation and {R}easoning},
  address      = {Hanoi, Vietnam},
  publisher    = {ArXiv},
  year         = {2024},
}