Praeco elastalert-server seems to ignore "verify_certs" option and always throws "Self-Signed Certificate in Chain".

[joshbasho]

Hello,

This is likely related to #213, but not entirely sure.

We are currently unable to use and Praeco version of Elastalert-Server. As in the issue linked, our issue was also solved by using daichi703n/elastalert:"0.2.1-dev". This image is based off jfcantu/elastalert, which was created to fix this issue in the ServerCentral repo. . It is also a few point releases behind and am unsure if there are any plans for it to be updated further.

Regardless of the setting chosen for verify_certs, we get this error after we attempt to access Praeco. Looks like elastalert base starts up fine. Issue only happens once an api call is made.

18:05:00.822Z  INFO elastalert-server: ProcessController:  Index create exited with code 0                                                                                                                                            
 18:05:00.823Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]                                                                                                                                   
 18:05:00.836Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 38)                                                                                                                                                
 18:05:00.838Z  INFO elastalert-server: Server:  Server listening on port 3030                                                                                                                                                          
 18:05:00.888Z  INFO elastalert-server: Server:  Websocket listening on port 3333                                                                                                                                                       
 18:05:00.888Z  INFO elastalert-server: Server:  Server started                                                                                                                                                                         
 18:06:17.005Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.                                                                                                                                              
 Elasticsearch ERROR: 2020-08-17T18:06:17Z                                                                                                                                                                                              
   Error: Request error, retrying                                                                                                                                                                                                       
   GET https://elasticsearch:9200/_cat/indices?h=index => self signed certificate in certificate chain                                                                                                     
       at Log.error (/opt/elastalert-server/node_modules/elasticsearch/src/lib/log.js:239:56)                                                                                                                                           
       at checkRespForFailure (/opt/elastalert-server/node_modules/elasticsearch/src/lib/transport.js:298:18)                                                                                                                           
       at HttpConnector.<anonymous> (/opt/elastalert-server/node_modules/elasticsearch/src/lib/connectors/http.js:171:7)                                                                                                                
       at ClientRequest.wrapper (/opt/elastalert-server/node_modules/lodash/lodash.js:4949:19)                                                                                                                                          
       at ClientRequest.emit (events.js:314:20)                                                                                                                                                                                         
       at TLSSocket.socketErrorListener (_http_client.js:469:9)                                                                                                                                                                         
       at TLSSocket.emit (events.js:314:20)                                                                                                                                                                                             
       at emitErrorNT (internal/streams/destroy.js:100:8)                                                                                                                                                                               
       at emitErrorCloseNT (internal/streams/destroy.js:68:3)                                                                                                                                                                           
       at processTicksAndRejections (internal/process/task_queues.js:80:21)                                                                                                                                                             
 Elasticsearch WARNING: 2020-08-17T18:06:17Z                                                                                                                                                                                            
   Unable to revive connection: https://elasticsearch:9200/                                                                                                                                                
 Elasticsearch WARNING: 2020-08-17T18:06:17Z                                                                                                                                                                                            
   No living connections       COPY loaders.py /opt/elastalert/elastalert/loaders.py

I saw an issue opened in the Bitsensor repo, but, as far as I know, Bitsensor is now defunct so I'm not sure any more updates will be made.

elastalert.yaml

      rules_folder: rules
      scan_subdirectories: false
      run_every:
        minutes: 1
      buffer_time:
        minutes: 1
      es_host: es-master
      es_port: 9200
      es_username: svc_elastalert
      es_password: 
      writeback_index: praeco_elastalert_status
      use_ssl: True
      verify_certs: True
      ca_certs: /opt/elastalert-server/ssl-ca-certs/elastic-certificate.pem
      alert_time_limit:
        minutes: 1440
      skip_invalid: True

api.config.json

      {
        "appName": "elastalert-server",
        "port": 3030,
        "wsport": 3333,
        "elastalertPath": "/opt/elastalert",
        "verbose": true,
        "es_debug": false,
        "debug": false,
        "rulesPath": {
          "relative": false,
          "path": "/opt/elastalert-data/rules"
        },
        "templatesPath": {
          "relative": false,
          "path": "/opt/elastalert-data/rule_templates"
        },
        "dataPath": {
          "relative": false,
          "path": "/opt/elastalert-data/data"
        },
        "es_host": "es-master",
        "es_port": 9200,
        "es_username": "svc_elastalert",
        "es_password": "",
        "es_ssl": true,
        "ca_certs": "/opt/elastalert-server/ssl-ca-certs/elastic-certificate.pem",
        "verify_certs": true,
        "writeback_index": "praeco_elastalert_status"
      }

[daichi703n]

@AddChickpeas That's strange... I've never fixed about connecting to Elasticserach. I used jfcantu's image for fixing output of Alert Log (jfcantu/elastalert-server@9e6c5dd).

By the way, I'm using es_ca_certs key in api.config.json with some reason. I forgot why...

  "es_ssl": true,
  "es_ca_certs": "/opt/elastalert-server/config/ca.crt",

Let me confirm how did I build the Image.

@nsano-rururu ブログコメントでお知らせいただきありがとうございます。Issue・Update追えておらずすみません... 証明書系は試行錯誤した記憶があるので、振り返って確認します。
既に確認済みかとは思いますが、このPRを見て上のパラメータ設定した気がします。 ServerCentral/elastalert-server#2

[nsano-rururu]

@daichi703n

情報ありがとうございます。

[nsano-rururu]

@AddChickpeas

I will post what I came up with about the points to check the operation.

I saw an issue opened in the Bitsensor repo, but, as far as I know, Bitsensor is now defunct so I'm not sure any more updates will be made.

Is the version of the problem that occurred in Bitsensor 3.0.0-beta.0 or 3.0.0-beta.1. Or I wonder if it was changed to ElastAlert 0.2.4.

Docker image name	tag	ElastAlert	Elasticsearch 7 Support	Remarks
bitsensor/elastalert	2.0.1	0.1.39	×
bitsensor/elastalert	lastet	0.1.39	×
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2	〇
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2	〇
servercentral/elastalert	latest	0.2.1	〇	bitsensor/elastalert fork Customize Python 3.6.8
daichi703n/elastalert	0.2.1-dev2	0.2.1	〇	servercentral/elastalert fork Customize bugfix
praecoapp/elastalert-server	latest	0.2.4	〇	servercentral/elastalert fork Customize bug fix Library Update Python 3.8

・Is it possible to solve it by lowering the Elasticsearch version with the npm library used by ElastAlert Server?
・Should I reduce ElastAlert from 0.2.4 to 0.2.1?
・Does the problem occur only with the self-signed SSL certificate?
・Is there a problem with the SSL certificate that was officially purchased?

・Does the first version of the Docker image updated to ElastAlert 0.2.4 give the same error?

johnsusek/elastalert-server
https://hub.docker.com/r/johnsusek/elastalert-server

I feel that it is necessary to check only the difference in ElastAlert version.
It is possible to upload a Docker image with ElastAlert downgraded to 0.2.1.

[daichi703n]

@nsano-rururu
I tested with praecoapp/elastalert-server latest d636242e4a84 8 days ago 517MB. As a result, es_ca_certs (NOT ca_certs) in api.config.json seems to be required for verifying es cert.

https://github.com/johnsusek/elastalert-server/blob/d996233759199437c93b2ffac17a35bbcb67a05c/src/common/elasticsearch_client.js#L82-L95

And I think verify_certs does not work too. When I used invalid key ca_certs, cert error occured.

elastalert_1     | Elasticsearch ERROR: 2020-08-18T17:23:12Z
elastalert_1     |   Error: Request error, retrying
elastalert_1     |   POST https://elasticsearch:9200/praeco_elastalert_status/_search?type=elastalert => unable to verify the first certificate
elastalert_1     |       at Log.error (/opt/elastalert-server/node_modules/elasticsearch/src/lib/log.js:239:56)
elastalert_1     |       at checkRespForFailure (/opt/elastalert-server/node_modules/elasticsearch/src/lib/transport.js:298:18)
elastalert_1     |       at HttpConnector.<anonymous> (/opt/elastalert-server/node_modules/elasticsearch/src/lib/connectors/http.js:171:7)
elastalert_1     |       at ClientRequest.wrapper (/opt/elastalert-server/node_modules/lodash/lodash.js:4949:19)
elastalert_1     |       at ClientRequest.emit (events.js:314:20)
elastalert_1     |       at TLSSocket.socketErrorListener (_http_client.js:469:9)
elastalert_1     |       at TLSSocket.emit (events.js:314:20)
elastalert_1     |       at emitErrorNT (internal/streams/destroy.js:100:8)
elastalert_1     |       at emitErrorCloseNT (internal/streams/destroy.js:68:3)
elastalert_1     |       at processTicksAndRejections (internal/process/task_queues.js:80:21)

I was able to skip verify cert by fixing like this.

export function getClient() {
  let scheme = 'http';
  let ssl_body = {};

  if (config.get('es_ssl')) {
    scheme = 'https';
-    ssl_body.rejectUnauthorized = true;
+    ssl_body.rejectUnauthorized = false;

Please refer to this.
https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/7.x/auth-reference.html#_ssl_configuration

To turn off certificate verification, you must specify an ssl object in the top level config and set rejectUnauthorized: false.

We have to make this option configurable by verify_certs (or other key name).

Node.jsに詳しくないのですが、ドキュメントと挙動を見る限り上記の通りかと思います。お手数をおかけしますが、確認・修正リリースをお願いします。

@AddChickpeas Would you please test this hotfix on your env?

> docker exec -it <name> sh
/opt/elastalert-server $ sed -i "s/ssl_body.rejectUnauthorized = true/ssl_body.rejectUnauthorized = false/" src/common/elasticsearch_client.js
/opt/elastalert-server $ exit
> docker restart <name>

[joshbasho]

@daichi703n

Changing it to es_ca_certs worked and it is now properly verifying! Thanks for the help.

I tested your hotfix locally and it seems to work as well. I deployed it and was able to connect without a ca_cert.