Spanner Load Test App

This repository contains sample game API code, load test sample code and arroud middleware for distributed load testing:

• Game API server
• Locust load test code
• Setup make file via terraform and gcloud

Requirements

Highly recommend: you to use it on Cloud Shell, because of easing to prepare settings for this demo

Cloud Environment

We expected to set organization policy default

• Google Cloud Project
• Google Account
  • It needs roles/owner permission on your project

Of course, you can deploy user account, but we recommend you to use Service Account.

• Service Account
  • It needs roles/owner permission on your project
• Service Account Key file

# Example to create account and keyfile
export SA_NAME=terraformer
export PROJECT_ID=YOUR_PROJECT_ID

gcloud config set project $PROJECT_ID

# Create Service Account
gcloud iam service-accounts create $SA_NAME \
  --description="Operation service account for spanner stress demo" \
  --display-name=$SA_NAME
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/owner"
# Download a key file
# Set JSON as a key file type 
export KEY_FILE=terraformer.json
gcloud iam service-accounts keys create $KEY_FILE \
  --iam-account=${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

Deploy Tools

Cloud Shell has already installed followings:

• gcloud
• terraform (auther version: Terraform v1.2.9 on linux_amd64)
• make command
• envsubst

Development Tools

If you need to develop this repository, you should install followings:

• python (auther version: 3.9.2)
  • pip (auther version: 22.2)
  • pipenv (auther version: 2022.7.24)
• docker (auther version: 20.10.17, API version: 1.41)
• docker-compose (auther version: version 1.24.0)
• tbls (Optional)
• terraform-docs (Optional)
• act (Optional)

How to use this demo

Setup env settings

You should set own environment by followings:

git clone https://github.com/cloudspannerecosystem/spanner-stress-test-demospanner-stress-test-demo.git
cd spanner-stress-test-demo

# You should change .env_sample to .env for gcloud environment
# and edit .env file
cp .env_sample .env
vim .env

Spesifications of .env file

Key name	Description	Example value
GOOGLE_CLOUD_PROJECT	Google Cloud Project ID	test
ACCOUNTS	Account list for access to bastion	["serviceAccount:[email protected]", "serviceAccount:[email protected]"]
KEY_PATH	Credential key file path	~/workspace/terraformer.json
REGION	Google Cloud Region you hope	asia-northeast1
SERVICE_NAME	Cloud Run's service name	sample-game
INSTANCE_NAME	Cloud Spanner's instance name	spanner-demo
DATABASE_NAME	Cloud Spanner's database name for testing	sample-game
ENV	Env id, but we expected to use "production" when you deploy on Google Cloud.	production
LOG_LEVEL	Loglevel of app and Locust	INFO
POD_NUM	The number of Pod for load test containers	10
USERS	Peak number of concurrent Locust users	10
RUN_TIME	Stop after the specified amount of time	1h

Deploy on Google Cloud

# Create environment via terraform and gcloud
make create.cloud.environment

# Run load testing
make deploy.locust

Watch the result

You can check the result in real time by followings:

• Google Cloud's operations suite

  • Cloud Monitoring
  • Cloud Logging
  • Cloud Trace

• Locust plugin dashboard

How to connect to grafana dashboard

# SSH port-forwording to grafana
make open.grafana
output: 
  Grafana dashboard: https://3443-xxxxyyyzzzz

# Cloud Shell envronment(You can access via proxy)
open https://3443-xxxxyyyzzzz

# Other environment
open http://localhost:3443

Change Cloud Spanner cluster size

Note: If your spanner mark high CPU usage, you should change Cloud Spanner size manually

• How to change conpute capacity of Cloud Spanner

Delete all components after tests

make delete.cloud.environment

Features

Asis

• Rest API for dummy game
  • opperate Cloud Spanner(database schema)
• API docs of Swagger by FastAPI
• Distributed Stressor by Locust
• IaC by terraform and gcloud

Tobe

• Auto Scalling for Cloud Spanner, instead of manual scalling

Reference

Internal modules reference

• Sample game app documents
• Test senario documents by locust
• Terraform documents

# Terraform document was generated by the following command
terraform-docs markdown table terraform/ > terraform/README.md

3rd party reference

• FastAPI
• Locust
• Locust Plugins
• OpenTelemetry FastAPI Instrunentation

Contribution

Note: Understand how to check your PR before you send PR

Some of tips in CI testing

CI by github actions covered unit-testing in apps and integration test in this repo, and so basically you should check the result of test, but there are some caustions as followings:

• CI check just linting and validation to .tf files, and so please run terraform by own environment when you chenge .tf files
• This Integration test is not perfect, and so you should check following points by manually
  • A fails ratio about GET characters/$user_id under 30%, because test senario cause 4XX error, and so it was not 0% of fail ratio
  • Fails ratios in other endpoint are 0%

• Others
  • CONTRIBUTING docs

License

This is Apache 2.0 License

Note

This is not an officially supported Google product