How to ingest API Gateway logs using S3 as log source

[owenCCY]

We do not support API Gateway ingestion officially in CLO, but customer wants to create the ingestion pipeline from S3 bucket.

[owenCCY]

Step1. Create log config for API Gateway in CLO console

1. Create a new Log Config in CLO console
2. Choose Multi-line Text as the Log Type
3. Choose Custom as the Parser
4. Paste the following content into the config:

First Line Regular Expression:

(?<timestamp>\d+)\s\((?<id>.+)\)\s(?<message>.+)

Example PostgreSQL logs:

1705550539841 (13909cc9-b022-40f2-b846-cbab5035ec68) Request parameter overrides:
Add Authorization: Basic SmFja19DaGVuOkE0JVpodDYx
Add X-HIP-FlowName: DailySales
Add X-HIP-SiteId: ALL_CN
Add X-HIP-FlowId: E3511
Override Host: esb-test.eloccitane.com
Override Content-Type: application/json

Example:

[owenCCY]

Step2. Create ingestion pipeline for API Gateway Logs in CLO console

1. Go to application log section in CLO, create an application pipeline which takes Amazon S3 as log source

2. Create source settings, the S3 bucket and Prefix you choose should be the place you store your API Gateway logs

3. Select the log config we created in the first step

[owenCCY]

Step3. Change index templates

Go to Dev Tools in OpenSearch dashboard:

Use the following command to view existing indices:

GET _cat/indices?v

Please find out the index you created for API Gateway Logs, and copy the name of the index prefix.

Let's say you name it as example-api-gateway. Please substitute with the name of your own index name!

Delete the existing example-api-gateway index using the following command:

DELETE example-api-gateway*

Note: you need to substitute the index name example-api-gateway with the one you created.

Get the index template of the existing index template using the following command:

GET _index_template/example-api-gateway-template

Note: you need to substitute the index name example-api-gateway with the one you created.

Please copy the returned index template and modify the value of Timestamp field with the following value:

"Timestamp" : {
  "format" : "epoch_millis",
  "type" : "date"
}

Update the index template with the modified index template, the command should look like the following:

PUT _index_template/example-api-gateway-template
{
  "index_patterns" : [
    "example-api-gateway-*"
  ],
  "template" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "example-api-gateway"
          }
        }
      }
    },
    "mappings" : {
      "properties" : {
        "timestamp" : {
          "format" : "epoch_millis",
          "type" : "date"
        },
        "id" : {
          "type" : "text"
        },
        "message" : {
          "type" : "text"
        }
      }
    }
  },
  "composed_of" : [ ]
}

Note: you need to substitute the index name example-api-gateway with the one you created.

Create the alias mapping for example-api-gateway index

PUT %3Cexample-api-gateway-%7Bnow%7Byyyy-MM-dd%7D%7D-000001%3E
{
  "aliases": {
    "example-api-gateway": {
      "is_write_index": true
    }
  }
}

Note: you need to substitute the index name example-api-gateway with the one you created.

[owenCCY]

Step4. Create Index Pattern

1. Go to OpenSearch Dashboard and create index pattern
2. Type in the index pattern name as example-api-gateway*
3. Choose timestamp as the time field
4. Click Create