Racing Car Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "671593a2",
   "metadata": {},
   "source": [
    "# Racing Car Project"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8fd2e6c",
   "metadata": {},
   "source": [
    "In this project we'll be seeing how to train a neural network with reinforcement learning in order to successfully drive a car in a virtual game environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "70f79e17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in links: https://download.pytorch.org/whl/cu113/torch_stable.html\n",
      "Requirement already satisfied: torch==1.10.0+cu113 in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (1.10.0+cu113)\n",
      "Requirement already satisfied: torchvision==0.11.1+cu113 in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (0.11.1+cu113)\n",
      "Requirement already satisfied: torchaudio===0.10.0+cu113 in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (0.10.0+cu113)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (from torch==1.10.0+cu113) (4.0.1)\n",
      "Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (from torchvision==0.11.1+cu113) (8.3.1)\n",
      "Requirement already satisfied: numpy in c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages (from torchvision==0.11.1+cu113) (1.21.4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 20.2.3; however, version 22.2.2 is available.\n",
      "You should consider upgrading via the 'c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "!pip install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fd31107c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "from stable_baselines3 import PPO\n",
    "from stable_baselines3.common.vec_env import  DummyVecEnv\n",
    "from stable_baselines3.common.evaluation import evaluate_policy\n",
    "import os\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5952bdfa",
   "metadata": {},
   "source": [
    "## Test environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c1bfc17",
   "metadata": {},
   "source": [
    "First of all we create the test environment and render it, after that we start the game without first training the model. Here we expect to have a low score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5f0f5805",
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(\"CarRacing-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4edbc79c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\giuse\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\pyglet\\image\\codecs\\wic.py:289: UserWarning: [WinError -2147417850] Cannot change thread mode after it is set\n",
      "  warnings.warn(str(err))\n"
     ]
    }
   ],
   "source": [
    "env.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e89eaa1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 1140..1429 -> 289-tiles track\n",
      "Episode:1 Score:-34.02777777777822\n"
     ]
    }
   ],
   "source": [
    "episodes = 1\n",
    "for episode in range(1, episodes+1):\n",
    "    obs = env.reset()\n",
    "    done = False\n",
    "    score = 0\n",
    "    \n",
    "    while not done:\n",
    "        env.render()\n",
    "        action = env.action_space.sample()\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        score+=reward\n",
    "    print(\"Episode:{} Score:{}\".format(episode, score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "379e284b",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05a6e7f6",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1e0db7",
   "metadata": {},
   "source": [
    "Here we start training the model using Torch. (Disclaimer, running the following lines of code on your computer might not work if you don't have the right graphic card)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e0b36ab4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__CUDNN VERSION: 8200\n",
      "__Number CUDA Devices: 1\n",
      "__CUDA Device Name: NVIDIA GeForce RTX 2060\n",
      "__CUDA Device Total Memory [GB]: 6.442123264\n"
     ]
    }
   ],
   "source": [
    "use_cuda = torch.cuda.is_available()\n",
    "if use_cuda:\n",
    "    print('__CUDNN VERSION:', torch.backends.cudnn.version())\n",
    "    print('__Number CUDA Devices:', torch.cuda.device_count())\n",
    "    print('__CUDA Device Name:',torch.cuda.get_device_name(0))\n",
    "    print('__CUDA Device Total Memory [GB]:',torch.cuda.get_device_properties(0).total_memory/1e9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fcf6554b",
   "metadata": {},
   "outputs": [],
   "source": [
    "env = DummyVecEnv([lambda: env])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a2dc7789",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda device\n",
      "Wrapping the env in a VecTransposeImage.\n"
     ]
    }
   ],
   "source": [
    "model = PPO(\"CnnPolicy\", env, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa71e771",
   "metadata": {},
   "source": [
    "The cell below is where we train our model, the training is gonna take some a few minutes (depending on your pc specs). \n",
    "The longer we train our model, the higher the score will be"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "18d416a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 1075..1348 -> 273-tiles track\n",
      "Track generation: 1145..1443 -> 298-tiles track\n",
      "Track generation: 1163..1458 -> 295-tiles track\n",
      "-----------------------------\n",
      "| time/              |      |\n",
      "|    fps             | 113  |\n",
      "|    iterations      | 1    |\n",
      "|    time_elapsed    | 17   |\n",
      "|    total_timesteps | 2048 |\n",
      "-----------------------------\n",
      "Track generation: 1136..1424 -> 288-tiles track\n",
      "Track generation: 1078..1358 -> 280-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 104         |\n",
      "|    iterations           | 2           |\n",
      "|    time_elapsed         | 39          |\n",
      "|    total_timesteps      | 4096        |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.009157797 |\n",
      "|    clip_fraction        | 0.0938      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.25       |\n",
      "|    explained_variance   | 0.181       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.172       |\n",
      "|    n_updates            | 20          |\n",
      "|    policy_gradient_loss | -0.00964    |\n",
      "|    std                  | 0.993       |\n",
      "|    value_loss           | 0.596       |\n",
      "-----------------------------------------\n",
      "Track generation: 1088..1371 -> 283-tiles track\n",
      "Track generation: 1139..1428 -> 289-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 102         |\n",
      "|    iterations           | 3           |\n",
      "|    time_elapsed         | 59          |\n",
      "|    total_timesteps      | 6144        |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.011040331 |\n",
      "|    clip_fraction        | 0.117       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.22       |\n",
      "|    explained_variance   | 0.0897      |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.136       |\n",
      "|    n_updates            | 30          |\n",
      "|    policy_gradient_loss | -0.0171     |\n",
      "|    std                  | 0.983       |\n",
      "|    value_loss           | 0.474       |\n",
      "-----------------------------------------\n",
      "Track generation: 1225..1535 -> 310-tiles track\n",
      "Track generation: 1035..1304 -> 269-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 101         |\n",
      "|    iterations           | 4           |\n",
      "|    time_elapsed         | 80          |\n",
      "|    total_timesteps      | 8192        |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.010180868 |\n",
      "|    clip_fraction        | 0.148       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.19       |\n",
      "|    explained_variance   | 0.184       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.109       |\n",
      "|    n_updates            | 40          |\n",
      "|    policy_gradient_loss | -0.016      |\n",
      "|    std                  | 0.972       |\n",
      "|    value_loss           | 0.604       |\n",
      "-----------------------------------------\n",
      "Track generation: 1105..1386 -> 281-tiles track\n",
      "Track generation: 995..1248 -> 253-tiles track\n",
      "---------------------------------------\n",
      "| time/                   |           |\n",
      "|    fps                  | 99        |\n",
      "|    iterations           | 5         |\n",
      "|    time_elapsed         | 102       |\n",
      "|    total_timesteps      | 10240     |\n",
      "| train/                  |           |\n",
      "|    approx_kl            | 0.0150411 |\n",
      "|    clip_fraction        | 0.15      |\n",
      "|    clip_range           | 0.2       |\n",
      "|    entropy_loss         | -4.16     |\n",
      "|    explained_variance   | 0.219     |\n",
      "|    learning_rate        | 0.0003    |\n",
      "|    loss                 | 0.103     |\n",
      "|    n_updates            | 50        |\n",
      "|    policy_gradient_loss | -0.0213   |\n",
      "|    std                  | 0.966     |\n",
      "|    value_loss           | 0.463     |\n",
      "---------------------------------------\n",
      "Track generation: 1224..1534 -> 310-tiles track\n",
      "Track generation: 1019..1283 -> 264-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 99          |\n",
      "|    iterations           | 6           |\n",
      "|    time_elapsed         | 124         |\n",
      "|    total_timesteps      | 12288       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.012085228 |\n",
      "|    clip_fraction        | 0.122       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.15       |\n",
      "|    explained_variance   | 0.0797      |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0914      |\n",
      "|    n_updates            | 60          |\n",
      "|    policy_gradient_loss | -0.0174     |\n",
      "|    std                  | 0.965       |\n",
      "|    value_loss           | 0.449       |\n",
      "-----------------------------------------\n",
      "Track generation: 1156..1449 -> 293-tiles track\n",
      "Track generation: 1175..1473 -> 298-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 98          |\n",
      "|    iterations           | 7           |\n",
      "|    time_elapsed         | 145         |\n",
      "|    total_timesteps      | 14336       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.020013088 |\n",
      "|    clip_fraction        | 0.202       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.12       |\n",
      "|    explained_variance   | 0.133       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0105      |\n",
      "|    n_updates            | 70          |\n",
      "|    policy_gradient_loss | -0.0321     |\n",
      "|    std                  | 0.95        |\n",
      "|    value_loss           | 0.35        |\n",
      "-----------------------------------------\n",
      "Track generation: 990..1247 -> 257-tiles track\n",
      "Track generation: 1156..1449 -> 293-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 98          |\n",
      "|    iterations           | 8           |\n",
      "|    time_elapsed         | 166         |\n",
      "|    total_timesteps      | 16384       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.018477932 |\n",
      "|    clip_fraction        | 0.195       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.09       |\n",
      "|    explained_variance   | -0.00574    |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0365      |\n",
      "|    n_updates            | 80          |\n",
      "|    policy_gradient_loss | -0.025      |\n",
      "|    std                  | 0.94        |\n",
      "|    value_loss           | 0.292       |\n",
      "-----------------------------------------\n",
      "Track generation: 1163..1458 -> 295-tiles track\n",
      "Track generation: 1034..1303 -> 269-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 98          |\n",
      "|    iterations           | 9           |\n",
      "|    time_elapsed         | 187         |\n",
      "|    total_timesteps      | 18432       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.023108505 |\n",
      "|    clip_fraction        | 0.215       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.06       |\n",
      "|    explained_variance   | 0.354       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0373      |\n",
      "|    n_updates            | 90          |\n",
      "|    policy_gradient_loss | -0.0263     |\n",
      "|    std                  | 0.934       |\n",
      "|    value_loss           | 0.392       |\n",
      "-----------------------------------------\n",
      "Track generation: 1148..1439 -> 291-tiles track\n",
      "Track generation: 1105..1390 -> 285-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 98          |\n",
      "|    iterations           | 10          |\n",
      "|    time_elapsed         | 208         |\n",
      "|    total_timesteps      | 20480       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.026610361 |\n",
      "|    clip_fraction        | 0.236       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -4.03       |\n",
      "|    explained_variance   | 0.13        |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | -0.00565    |\n",
      "|    n_updates            | 100         |\n",
      "|    policy_gradient_loss | -0.04       |\n",
      "|    std                  | 0.925       |\n",
      "|    value_loss           | 0.318       |\n",
      "-----------------------------------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 1047..1323 -> 276-tiles track\n",
      "Track generation: 1094..1372 -> 278-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 98         |\n",
      "|    iterations           | 11         |\n",
      "|    time_elapsed         | 229        |\n",
      "|    total_timesteps      | 22528      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.02131917 |\n",
      "|    clip_fraction        | 0.23       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -4         |\n",
      "|    explained_variance   | 0.348      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.0165     |\n",
      "|    n_updates            | 110        |\n",
      "|    policy_gradient_loss | -0.0324    |\n",
      "|    std                  | 0.915      |\n",
      "|    value_loss           | 0.305      |\n",
      "----------------------------------------\n",
      "Track generation: 1192..1494 -> 302-tiles track\n",
      "Track generation: 1133..1427 -> 294-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 97          |\n",
      "|    iterations           | 12          |\n",
      "|    time_elapsed         | 251         |\n",
      "|    total_timesteps      | 24576       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.028928243 |\n",
      "|    clip_fraction        | 0.243       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.96       |\n",
      "|    explained_variance   | 0.354       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0383      |\n",
      "|    n_updates            | 120         |\n",
      "|    policy_gradient_loss | -0.0336     |\n",
      "|    std                  | 0.902       |\n",
      "|    value_loss           | 0.471       |\n",
      "-----------------------------------------\n",
      "Track generation: 1185..1485 -> 300-tiles track\n",
      "Track generation: 1295..1623 -> 328-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 97          |\n",
      "|    iterations           | 13          |\n",
      "|    time_elapsed         | 273         |\n",
      "|    total_timesteps      | 26624       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.032231405 |\n",
      "|    clip_fraction        | 0.245       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.91       |\n",
      "|    explained_variance   | 0.345       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0934      |\n",
      "|    n_updates            | 130         |\n",
      "|    policy_gradient_loss | -0.0322     |\n",
      "|    std                  | 0.885       |\n",
      "|    value_loss           | 0.476       |\n",
      "-----------------------------------------\n",
      "Track generation: 1069..1346 -> 277-tiles track\n",
      "Track generation: 1244..1559 -> 315-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 97          |\n",
      "|    iterations           | 14          |\n",
      "|    time_elapsed         | 294         |\n",
      "|    total_timesteps      | 28672       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.059215635 |\n",
      "|    clip_fraction        | 0.361       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.86       |\n",
      "|    explained_variance   | 0.611       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | -0.0154     |\n",
      "|    n_updates            | 140         |\n",
      "|    policy_gradient_loss | -0.0523     |\n",
      "|    std                  | 0.87        |\n",
      "|    value_loss           | 0.282       |\n",
      "-----------------------------------------\n",
      "Track generation: 1324..1659 -> 335-tiles track\n",
      "Track generation: 1420..1779 -> 359-tiles track\n",
      "retry to generate track (normal if there are not manyinstances of this message)\n",
      "Track generation: 1230..1542 -> 312-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 97         |\n",
      "|    iterations           | 15         |\n",
      "|    time_elapsed         | 316        |\n",
      "|    total_timesteps      | 30720      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.03833337 |\n",
      "|    clip_fraction        | 0.309      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.82      |\n",
      "|    explained_variance   | 0.773      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.00679    |\n",
      "|    n_updates            | 150        |\n",
      "|    policy_gradient_loss | -0.0403    |\n",
      "|    std                  | 0.862      |\n",
      "|    value_loss           | 0.356      |\n",
      "----------------------------------------\n",
      "Track generation: 1220..1529 -> 309-tiles track\n",
      "Track generation: 1247..1563 -> 316-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 97          |\n",
      "|    iterations           | 16          |\n",
      "|    time_elapsed         | 337         |\n",
      "|    total_timesteps      | 32768       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.059249412 |\n",
      "|    clip_fraction        | 0.319       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.78       |\n",
      "|    explained_variance   | 0.827       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | -0.0374     |\n",
      "|    n_updates            | 160         |\n",
      "|    policy_gradient_loss | -0.0437     |\n",
      "|    std                  | 0.849       |\n",
      "|    value_loss           | 0.142       |\n",
      "-----------------------------------------\n",
      "Track generation: 1081..1355 -> 274-tiles track\n",
      "Track generation: 1104..1384 -> 280-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 96          |\n",
      "|    iterations           | 17          |\n",
      "|    time_elapsed         | 358         |\n",
      "|    total_timesteps      | 34816       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.040025014 |\n",
      "|    clip_fraction        | 0.332       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.74       |\n",
      "|    explained_variance   | 0.883       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0115      |\n",
      "|    n_updates            | 170         |\n",
      "|    policy_gradient_loss | -0.0432     |\n",
      "|    std                  | 0.839       |\n",
      "|    value_loss           | 0.347       |\n",
      "-----------------------------------------\n",
      "Track generation: 1143..1433 -> 290-tiles track\n",
      "Track generation: 1162..1457 -> 295-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 96         |\n",
      "|    iterations           | 18         |\n",
      "|    time_elapsed         | 381        |\n",
      "|    total_timesteps      | 36864      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.04504414 |\n",
      "|    clip_fraction        | 0.333      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.71      |\n",
      "|    explained_variance   | 0.843      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | -0.0105    |\n",
      "|    n_updates            | 180        |\n",
      "|    policy_gradient_loss | -0.0366    |\n",
      "|    std                  | 0.833      |\n",
      "|    value_loss           | 0.503      |\n",
      "----------------------------------------\n",
      "Track generation: 1072..1349 -> 277-tiles track\n",
      "Track generation: 1115..1398 -> 283-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 96          |\n",
      "|    iterations           | 19          |\n",
      "|    time_elapsed         | 403         |\n",
      "|    total_timesteps      | 38912       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.049691074 |\n",
      "|    clip_fraction        | 0.339       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.68       |\n",
      "|    explained_variance   | 0.893       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | -0.0137     |\n",
      "|    n_updates            | 190         |\n",
      "|    policy_gradient_loss | -0.0359     |\n",
      "|    std                  | 0.822       |\n",
      "|    value_loss           | 0.309       |\n",
      "-----------------------------------------\n",
      "Track generation: 1191..1493 -> 302-tiles track\n",
      "Track generation: 1199..1503 -> 304-tiles track\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 96         |\n",
      "|    iterations           | 20         |\n",
      "|    time_elapsed         | 426        |\n",
      "|    total_timesteps      | 40960      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06382162 |\n",
      "|    clip_fraction        | 0.344      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.64      |\n",
      "|    explained_variance   | 0.919      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.0571     |\n",
      "|    n_updates            | 200        |\n",
      "|    policy_gradient_loss | -0.0367    |\n",
      "|    std                  | 0.813      |\n",
      "|    value_loss           | 0.471      |\n",
      "----------------------------------------\n",
      "Track generation: 1119..1403 -> 284-tiles track\n",
      "Track generation: 1395..1751 -> 356-tiles track\n",
      "Track generation: 1023..1287 -> 264-tiles track\n",
      "retry to generate track (normal if there are not manyinstances of this message)\n",
      "Track generation: 1215..1523 -> 308-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 95          |\n",
      "|    iterations           | 21          |\n",
      "|    time_elapsed         | 449         |\n",
      "|    total_timesteps      | 43008       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.046703286 |\n",
      "|    clip_fraction        | 0.337       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.6        |\n",
      "|    explained_variance   | 0.942       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0337      |\n",
      "|    n_updates            | 210         |\n",
      "|    policy_gradient_loss | -0.0397     |\n",
      "|    std                  | 0.801       |\n",
      "|    value_loss           | 0.634       |\n",
      "-----------------------------------------\n",
      "Track generation: 1116..1399 -> 283-tiles track\n",
      "Track generation: 1116..1399 -> 283-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 95         |\n",
      "|    iterations           | 22         |\n",
      "|    time_elapsed         | 471        |\n",
      "|    total_timesteps      | 45056      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06110602 |\n",
      "|    clip_fraction        | 0.389      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.56      |\n",
      "|    explained_variance   | 0.964      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | -0.0685    |\n",
      "|    n_updates            | 220        |\n",
      "|    policy_gradient_loss | -0.0352    |\n",
      "|    std                  | 0.794      |\n",
      "|    value_loss           | 0.279      |\n",
      "----------------------------------------\n",
      "Track generation: 1211..1518 -> 307-tiles track\n",
      "Track generation: 1306..1637 -> 331-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 95          |\n",
      "|    iterations           | 23          |\n",
      "|    time_elapsed         | 493         |\n",
      "|    total_timesteps      | 47104       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.050754976 |\n",
      "|    clip_fraction        | 0.332       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.54       |\n",
      "|    explained_variance   | 0.947       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.00364     |\n",
      "|    n_updates            | 230         |\n",
      "|    policy_gradient_loss | -0.0237     |\n",
      "|    std                  | 0.792       |\n",
      "|    value_loss           | 0.417       |\n",
      "-----------------------------------------\n",
      "Track generation: 1152..1444 -> 292-tiles track\n",
      "Track generation: 1284..1609 -> 325-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 95          |\n",
      "|    iterations           | 24          |\n",
      "|    time_elapsed         | 515         |\n",
      "|    total_timesteps      | 49152       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.040190596 |\n",
      "|    clip_fraction        | 0.366       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.53       |\n",
      "|    explained_variance   | 0.702       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.351       |\n",
      "|    n_updates            | 240         |\n",
      "|    policy_gradient_loss | -0.0228     |\n",
      "|    std                  | 0.788       |\n",
      "|    value_loss           | 2.55        |\n",
      "-----------------------------------------\n",
      "Track generation: 1184..1484 -> 300-tiles track\n",
      "Track generation: 1189..1490 -> 301-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 95         |\n",
      "|    iterations           | 25         |\n",
      "|    time_elapsed         | 538        |\n",
      "|    total_timesteps      | 51200      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06658345 |\n",
      "|    clip_fraction        | 0.394      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.51      |\n",
      "|    explained_variance   | 0.941      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.117      |\n",
      "|    n_updates            | 250        |\n",
      "|    policy_gradient_loss | -0.0267    |\n",
      "|    std                  | 0.78       |\n",
      "|    value_loss           | 1.05       |\n",
      "----------------------------------------\n",
      "Track generation: 1132..1419 -> 287-tiles track\n",
      "Track generation: 1197..1501 -> 304-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 26         |\n",
      "|    time_elapsed         | 560        |\n",
      "|    total_timesteps      | 53248      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06613495 |\n",
      "|    clip_fraction        | 0.401      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.48      |\n",
      "|    explained_variance   | 0.853      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.557      |\n",
      "|    n_updates            | 260        |\n",
      "|    policy_gradient_loss | -0.0216    |\n",
      "|    std                  | 0.777      |\n",
      "|    value_loss           | 1.43       |\n",
      "----------------------------------------\n",
      "Track generation: 1128..1418 -> 290-tiles track\n",
      "Track generation: 1176..1474 -> 298-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 27         |\n",
      "|    time_elapsed         | 583        |\n",
      "|    total_timesteps      | 55296      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05487094 |\n",
      "|    clip_fraction        | 0.364      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.46      |\n",
      "|    explained_variance   | 0.945      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.172      |\n",
      "|    n_updates            | 270        |\n",
      "|    policy_gradient_loss | -0.0241    |\n",
      "|    std                  | 0.768      |\n",
      "|    value_loss           | 0.809      |\n",
      "----------------------------------------\n",
      "Track generation: 1171..1468 -> 297-tiles track\n",
      "Track generation: 1230..1542 -> 312-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 28         |\n",
      "|    time_elapsed         | 606        |\n",
      "|    total_timesteps      | 57344      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.04758959 |\n",
      "|    clip_fraction        | 0.37       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.43      |\n",
      "|    explained_variance   | 0.88       |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.253      |\n",
      "|    n_updates            | 280        |\n",
      "|    policy_gradient_loss | -0.0292    |\n",
      "|    std                  | 0.759      |\n",
      "|    value_loss           | 1.68       |\n",
      "----------------------------------------\n",
      "Track generation: 1137..1426 -> 289-tiles track\n",
      "Track generation: 1207..1513 -> 306-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 29          |\n",
      "|    time_elapsed         | 627         |\n",
      "|    total_timesteps      | 59392       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.057155747 |\n",
      "|    clip_fraction        | 0.415       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.4        |\n",
      "|    explained_variance   | 0.983       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | -0.0117     |\n",
      "|    n_updates            | 290         |\n",
      "|    policy_gradient_loss | -0.0249     |\n",
      "|    std                  | 0.753       |\n",
      "|    value_loss           | 0.487       |\n",
      "-----------------------------------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 994..1256 -> 262-tiles track\n",
      "Track generation: 1356..1699 -> 343-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 30          |\n",
      "|    time_elapsed         | 649         |\n",
      "|    total_timesteps      | 61440       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.040270522 |\n",
      "|    clip_fraction        | 0.327       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.38       |\n",
      "|    explained_variance   | 0.973       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.096       |\n",
      "|    n_updates            | 300         |\n",
      "|    policy_gradient_loss | -0.0259     |\n",
      "|    std                  | 0.751       |\n",
      "|    value_loss           | 0.824       |\n",
      "-----------------------------------------\n",
      "Track generation: 1227..1538 -> 311-tiles track\n",
      "Track generation: 1242..1556 -> 314-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 31          |\n",
      "|    time_elapsed         | 670         |\n",
      "|    total_timesteps      | 63488       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.030613849 |\n",
      "|    clip_fraction        | 0.299       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.35       |\n",
      "|    explained_variance   | 0.967       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.133       |\n",
      "|    n_updates            | 310         |\n",
      "|    policy_gradient_loss | -0.0251     |\n",
      "|    std                  | 0.741       |\n",
      "|    value_loss           | 1.01        |\n",
      "-----------------------------------------\n",
      "Track generation: 1127..1415 -> 288-tiles track\n",
      "retry to generate track (normal if there are not manyinstances of this message)\n",
      "Track generation: 1147..1438 -> 291-tiles track\n",
      "Track generation: 1110..1398 -> 288-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 32         |\n",
      "|    time_elapsed         | 692        |\n",
      "|    total_timesteps      | 65536      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05400733 |\n",
      "|    clip_fraction        | 0.362      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.31      |\n",
      "|    explained_variance   | 0.887      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.169      |\n",
      "|    n_updates            | 320        |\n",
      "|    policy_gradient_loss | -0.0231    |\n",
      "|    std                  | 0.732      |\n",
      "|    value_loss           | 1.46       |\n",
      "----------------------------------------\n",
      "Track generation: 1049..1315 -> 266-tiles track\n",
      "Track generation: 1207..1517 -> 310-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 33          |\n",
      "|    time_elapsed         | 714         |\n",
      "|    total_timesteps      | 67584       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.052844867 |\n",
      "|    clip_fraction        | 0.391       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.27       |\n",
      "|    explained_variance   | 0.967       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.236       |\n",
      "|    n_updates            | 330         |\n",
      "|    policy_gradient_loss | -0.0298     |\n",
      "|    std                  | 0.72        |\n",
      "|    value_loss           | 0.869       |\n",
      "-----------------------------------------\n",
      "Track generation: 1135..1423 -> 288-tiles track\n",
      "Track generation: 1223..1533 -> 310-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 34         |\n",
      "|    time_elapsed         | 735        |\n",
      "|    total_timesteps      | 69632      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.07580136 |\n",
      "|    clip_fraction        | 0.399      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.23      |\n",
      "|    explained_variance   | 0.976      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.0358     |\n",
      "|    n_updates            | 340        |\n",
      "|    policy_gradient_loss | -0.0145    |\n",
      "|    std                  | 0.716      |\n",
      "|    value_loss           | 0.348      |\n",
      "----------------------------------------\n",
      "Track generation: 1255..1573 -> 318-tiles track\n",
      "Track generation: 1204..1509 -> 305-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 35          |\n",
      "|    time_elapsed         | 757         |\n",
      "|    total_timesteps      | 71680       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.038369983 |\n",
      "|    clip_fraction        | 0.35        |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.21       |\n",
      "|    explained_variance   | 0.956       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.599       |\n",
      "|    n_updates            | 350         |\n",
      "|    policy_gradient_loss | -0.0216     |\n",
      "|    std                  | 0.713       |\n",
      "|    value_loss           | 1.36        |\n",
      "-----------------------------------------\n",
      "Track generation: 1045..1310 -> 265-tiles track\n",
      "Track generation: 1148..1439 -> 291-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 36         |\n",
      "|    time_elapsed         | 778        |\n",
      "|    total_timesteps      | 73728      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06680203 |\n",
      "|    clip_fraction        | 0.39       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.2       |\n",
      "|    explained_variance   | 0.914      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.193      |\n",
      "|    n_updates            | 360        |\n",
      "|    policy_gradient_loss | -0.0241    |\n",
      "|    std                  | 0.709      |\n",
      "|    value_loss           | 1.25       |\n",
      "----------------------------------------\n",
      "Track generation: 1284..1619 -> 335-tiles track\n",
      "Track generation: 1141..1430 -> 289-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 37          |\n",
      "|    time_elapsed         | 801         |\n",
      "|    total_timesteps      | 75776       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.057959944 |\n",
      "|    clip_fraction        | 0.357       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.18       |\n",
      "|    explained_variance   | 0.949       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.155       |\n",
      "|    n_updates            | 370         |\n",
      "|    policy_gradient_loss | -0.0279     |\n",
      "|    std                  | 0.706       |\n",
      "|    value_loss           | 1.15        |\n",
      "-----------------------------------------\n",
      "Track generation: 1168..1464 -> 296-tiles track\n",
      "Track generation: 1103..1383 -> 280-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 38         |\n",
      "|    time_elapsed         | 824        |\n",
      "|    total_timesteps      | 77824      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.08736109 |\n",
      "|    clip_fraction        | 0.383      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.16      |\n",
      "|    explained_variance   | 0.974      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.0768     |\n",
      "|    n_updates            | 380        |\n",
      "|    policy_gradient_loss | -0.0277    |\n",
      "|    std                  | 0.697      |\n",
      "|    value_loss           | 0.672      |\n",
      "----------------------------------------\n",
      "Track generation: 1090..1367 -> 277-tiles track\n",
      "Track generation: 988..1245 -> 257-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 39         |\n",
      "|    time_elapsed         | 846        |\n",
      "|    total_timesteps      | 79872      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.07890025 |\n",
      "|    clip_fraction        | 0.43       |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.12      |\n",
      "|    explained_variance   | 0.943      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.304      |\n",
      "|    n_updates            | 390        |\n",
      "|    policy_gradient_loss | -0.0295    |\n",
      "|    std                  | 0.692      |\n",
      "|    value_loss           | 1.29       |\n",
      "----------------------------------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 1272..1594 -> 322-tiles track\n",
      "Track generation: 868..1095 -> 227-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 40         |\n",
      "|    time_elapsed         | 869        |\n",
      "|    total_timesteps      | 81920      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.07894936 |\n",
      "|    clip_fraction        | 0.408      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.1       |\n",
      "|    explained_variance   | 0.94       |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.0713     |\n",
      "|    n_updates            | 400        |\n",
      "|    policy_gradient_loss | -0.0145    |\n",
      "|    std                  | 0.691      |\n",
      "|    value_loss           | 1.04       |\n",
      "----------------------------------------\n",
      "Track generation: 1097..1376 -> 279-tiles track\n",
      "Track generation: 1180..1479 -> 299-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 94         |\n",
      "|    iterations           | 41         |\n",
      "|    time_elapsed         | 891        |\n",
      "|    total_timesteps      | 83968      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.07120307 |\n",
      "|    clip_fraction        | 0.438      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.09      |\n",
      "|    explained_variance   | 0.938      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.384      |\n",
      "|    n_updates            | 410        |\n",
      "|    policy_gradient_loss | -0.0103    |\n",
      "|    std                  | 0.688      |\n",
      "|    value_loss           | 2.84       |\n",
      "----------------------------------------\n",
      "Track generation: 1074..1351 -> 277-tiles track\n",
      "Track generation: 1232..1544 -> 312-tiles track\n",
      "Track generation: 1132..1419 -> 287-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 42          |\n",
      "|    time_elapsed         | 914         |\n",
      "|    total_timesteps      | 86016       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.078685895 |\n",
      "|    clip_fraction        | 0.431       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.08       |\n",
      "|    explained_variance   | 0.974       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.0338      |\n",
      "|    n_updates            | 420         |\n",
      "|    policy_gradient_loss | -0.0189     |\n",
      "|    std                  | 0.686       |\n",
      "|    value_loss           | 0.651       |\n",
      "-----------------------------------------\n",
      "Track generation: 1056..1324 -> 268-tiles track\n",
      "Track generation: 1302..1632 -> 330-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 93         |\n",
      "|    iterations           | 43         |\n",
      "|    time_elapsed         | 936        |\n",
      "|    total_timesteps      | 88064      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05847176 |\n",
      "|    clip_fraction        | 0.379      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -3.07      |\n",
      "|    explained_variance   | 0.87       |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.894      |\n",
      "|    n_updates            | 430        |\n",
      "|    policy_gradient_loss | -0.0124    |\n",
      "|    std                  | 0.684      |\n",
      "|    value_loss           | 6.41       |\n",
      "----------------------------------------\n",
      "Track generation: 1363..1708 -> 345-tiles track\n",
      "Track generation: 1144..1434 -> 290-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 93          |\n",
      "|    iterations           | 44          |\n",
      "|    time_elapsed         | 959         |\n",
      "|    total_timesteps      | 90112       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.058711722 |\n",
      "|    clip_fraction        | 0.396       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.06       |\n",
      "|    explained_variance   | 0.676       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.488       |\n",
      "|    n_updates            | 440         |\n",
      "|    policy_gradient_loss | -0.0195     |\n",
      "|    std                  | 0.682       |\n",
      "|    value_loss           | 3.96        |\n",
      "-----------------------------------------\n",
      "Track generation: 1062..1331 -> 269-tiles track\n",
      "Track generation: 1171..1468 -> 297-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 93          |\n",
      "|    iterations           | 45          |\n",
      "|    time_elapsed         | 982         |\n",
      "|    total_timesteps      | 92160       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.055831883 |\n",
      "|    clip_fraction        | 0.387       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.04       |\n",
      "|    explained_variance   | 0.964       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.187       |\n",
      "|    n_updates            | 450         |\n",
      "|    policy_gradient_loss | -0.0214     |\n",
      "|    std                  | 0.677       |\n",
      "|    value_loss           | 1.7         |\n",
      "-----------------------------------------\n",
      "Track generation: 1111..1398 -> 287-tiles track\n",
      "Track generation: 1151..1442 -> 291-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 93          |\n",
      "|    iterations           | 46          |\n",
      "|    time_elapsed         | 1005        |\n",
      "|    total_timesteps      | 94208       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.045335747 |\n",
      "|    clip_fraction        | 0.376       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.03       |\n",
      "|    explained_variance   | 0.901       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.52        |\n",
      "|    n_updates            | 460         |\n",
      "|    policy_gradient_loss | -0.00858    |\n",
      "|    std                  | 0.675       |\n",
      "|    value_loss           | 3.38        |\n",
      "-----------------------------------------\n",
      "Track generation: 1041..1314 -> 273-tiles track\n",
      "Track generation: 1032..1294 -> 262-tiles track\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 93          |\n",
      "|    iterations           | 47          |\n",
      "|    time_elapsed         | 1028        |\n",
      "|    total_timesteps      | 96256       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.067019194 |\n",
      "|    clip_fraction        | 0.418       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -3.01       |\n",
      "|    explained_variance   | 0.931       |\n",
      "|    learning_rate        | 0.0003      |\n",
      "|    loss                 | 0.215       |\n",
      "|    n_updates            | 470         |\n",
      "|    policy_gradient_loss | -0.0208     |\n",
      "|    std                  | 0.668       |\n",
      "|    value_loss           | 1.7         |\n",
      "-----------------------------------------\n",
      "Track generation: 1033..1296 -> 263-tiles track\n",
      "Track generation: 1031..1293 -> 262-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 93         |\n",
      "|    iterations           | 48         |\n",
      "|    time_elapsed         | 1050       |\n",
      "|    total_timesteps      | 98304      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.05955406 |\n",
      "|    clip_fraction        | 0.408      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -2.98      |\n",
      "|    explained_variance   | 0.965      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.264      |\n",
      "|    n_updates            | 480        |\n",
      "|    policy_gradient_loss | -0.0118    |\n",
      "|    std                  | 0.666      |\n",
      "|    value_loss           | 1.69       |\n",
      "----------------------------------------\n",
      "Track generation: 1051..1318 -> 267-tiles track\n",
      "Track generation: 1011..1276 -> 265-tiles track\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 93         |\n",
      "|    iterations           | 49         |\n",
      "|    time_elapsed         | 1072       |\n",
      "|    total_timesteps      | 100352     |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.06502902 |\n",
      "|    clip_fraction        | 0.371      |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -2.96      |\n",
      "|    explained_variance   | 0.911      |\n",
      "|    learning_rate        | 0.0003     |\n",
      "|    loss                 | 0.228      |\n",
      "|    n_updates            | 490        |\n",
      "|    policy_gradient_loss | -0.0225    |\n",
      "|    std                  | 0.658      |\n",
      "|    value_loss           | 2.26       |\n",
      "----------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines3.ppo.ppo.PPO at 0x19e1d006250>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.learn(total_timesteps=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cb3b23cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Track generation: 1267..1588 -> 321-tiles track\n",
      "Track generation: 1163..1458 -> 295-tiles track\n",
      "Track generation: 1145..1435 -> 290-tiles track\n",
      "Track generation: 1199..1503 -> 304-tiles track\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(182.5498843540748, 115.31087818476664)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " evaluate_policy(model, env, n_eval_episodes=3, render= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2bcc5dc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa702a4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b947b504",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}