_notes.txt

RFC 3164:
            <https://tools.ietf.org/html/rfc3164>

RFC 5424:
            <https://tools.ietf.org/html/rfc5424>

Analyze syslog messages
https://blog.datalust.co/seq-input-syslog/

RFC5424 syslog Message Format introduction
https://hackmd.io/@njjack/syslogformat

Numerical         Severity       
code
              0                 Emergency: system is unusable
              1                 Alert: action must be taken immediately
              2                 Critical: critical conditions
              3                 Error: error conditions
              4                 Warning: warning conditions
              5                 Notice: normal but significant condition
              6                 Informational: informational messages
              7                 Debug: debug-level messages

syslog messages examples

    with no STRUCTURED-DATA

    <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

    PRI: <34>, Facility: 4, Severity: 2
    VERSION: 1
    timestamp:2003-10-11T22:14:15.003Z
    HOSTNAME: mymachine.example.com.
    APP-NAME: su
    PROCID: unknown, displayed as NULVALUE -
    MSGID: ID47. 
    MSG is 'su root' failed for lonvick..., encoded in UTF-8.
    There is no STRUCTURED-DATA, displayed as NULVALUE -

    with STRUCTURED-DATA

    <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] BOMAn application event log entry...

    with no MSG

    <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"][examplePriority@32473 class="high"]


Forward Rsyslog messages to Apache Kafka
https://github.com/uswitch/syslogger

Integrate syslog-ng with Kafka
https://jekhokie.github.io/syslog/syslog-ng/kafka/logging/linux/2016/12/21/integrate-syslog-ng-with-kafka.html

Send Syslog to Kafka
https://support.vectra.ai/s/article/KB-VS-1155

    Upload your CA Certificate, Client Certificate and Client key (all required)

    CA certificate
        Must be in unencrypted PEM, x.509 format

    Client Certificate
        Must be in unencrypted PEM, x.509 format
        Must be signed by CA certificate

    Client Key
        Must be in unencrypted PEM, x.509 format
        Must match Client Certificate




Syslog Source Connector for Confluent Platform
https://docs.confluent.io/kafka-connectors/syslog/current/overview.html#

You can use the Kafka Connect Syslog Source connector to consume data from network devices. 
Supported formats are rfc 3164, rfc 5424, and Common Event Format (CEF).

    Features
        The Syslog Source connector includes the following features:

        At least once delivery
            This connector guarantees that records are delivered at least once to the Kafka topic. 
            If the connector restarts, there may be some duplicate records in the Kafka topic.

        Supports one task
            The Syslog Source connector supports running one task.

Configuration
https://docs.confluent.io/kafka-connectors/syslog/current/syslog_source_connector_config.html

    topic
        Name of the topic to put all records.
        Type: string
        Default: syslog
        Importance: high

    SSL
        syslog.ssl.cert.chain.path
            Path to X.509 cert chain file in PEM format.

            Type: string
            Default: “”
            Importance: high

        syslog.ssl.key.password
            The password of the key file(syslog.ssl.key.path), or blank if it’s not password-protected

            Type: password
            Default: [hidden]
            Importance: high

        syslog.ssl.key.path
            Path to a PKCS#8 private key file in PEM format.

            Type: string
            Default: “”
            Importance: high

        syslog.ssl.self.signed.certificate.enable
            Flag to determine if a self signed certificate should be generated and used.

            Type: boolean
            Default: false
            Importance: high

        syslog.ssl.client.auth.enable
            Enable Client Authentication in SSL communication between the connector and syslog client. 
            * Type: boolean 
            * Default: false 
            * Importance: low

        syslog.ssl.root.ca.cert.chain.path
            Path to X.509 root CA certificate chain file in PEM format to verify client certificate when client authentication is enabled in SSL Communication.

            Type: string
            Default: “”
            Importance: low


[Cloudera] Syslog TCP Source connector
https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/kafka-connect/topics/kafka-connect-connector-syslog-tcp-source.html?



go syslogd
https://github.com/mcuadros/go-syslog

- MIT license
- v2.3.0 Aug 15, 2019
- last change Mar 30, 2020
- 22 issues [Mar 22, 2016:Dec 15, 2021]
- 9 pull requests [Sep 5, 2018:Jun 30, 2023]
- without go.mod
- Imported by: 121
- tls tests failed with panic => temporary tls tests disabled

- COMMON 
		"priority":  p.priority.P,              int
		"facility":  p.priority.F.Value,        int
		"severity":  p.priority.S.Value,        int
		"timestamp": p.header.timestamp,        time.Time
		"hostname":  p.header.hostname,         string

- RFC3164:
		"tag":       p.message.tag,             string
		"content":   p.message.content,         string

- RFC5424:
		"version":         p.header.version,    int
		"app_name":        p.header.appName,    string
		"proc_id":         p.header.procId,     string
		"msg_id":          p.header.msgId,      string
		"structured_data": p.structuredData,    string
		"message":         p.message,           string

- go-syslog/internal/syslogparser/ very old (4 years) fork of 

    - https://github.com/jeromer/syslogparser
      
      - last change Aug 13, 2021 v1.1.0 
      - BSD-3-Clause license
      - go 1.14
      - Imported by: 61


Ekanite
https://github.com/ekanite/ekanite


THE SYSLOG HELL
https://techblog.bozho.net/the-syslog-hell/#:~:text=RFC5424%20defines%20a%20key-value,define%20their%20own%20message%20formats.


Syslog Standards: A simple Comparison between RFC3164 & RFC5424
https://www.diaryfolio.com/2020/07/syslog-standards-simple-comparison.html

syslog parser
https://github.com/influxdata/go-syslog/


Syslog message format
https://github.com/luigiberrettini/NLog.Targets.Syslog-Personal/blob/master/docs/syslog-message-format.md


Filter messages based on priority
https://documentation.solarwinds.com/en/success_center/kss/content/kss_adminguide_filter_messages_based_on_priority.htm


unix - sockets for local interprocess communication
https://man7.org/linux/man-pages/man7/unix.7.html


syslog go client
https://sirsean.medium.com/srslog-sending-syslog-messages-from-go-a270d9c74ecd
https://github.com/RackSec/srslog


Creating self-signed TLS certificates in Go
https://atifali.ca/go-self-signed-tls/

How to setup HTTPS web server in Golang with self-signed SSL TLS certificate
https://www.bastionxp.com/blog/how-to-setup-https-web-server-in-golang-with-self-signed-ssl-tls-certificate/

TLS server with in-memory self-signed certificate
https://gist.github.com/shivakar/cd52b5594d4912fbeb46

Passing certificate and key as string to ListenAndServeTLS
https://stackoverflow.com/questions/47857573/passing-certificate-and-key-as-string-to-listenandservetls/47857805


Connecting with TLS and verify client identity
https://docs.nats.io/using-nats/developer/connecting/tls
https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls


Memphis Metadata TLS connection configuration
https://docs.memphis.dev/memphis/deployment/production-best-practices#memphis-metadata-tls-connection-configuration


Deploy Memphis with TLS (encrypted communication via SSL)
https://github.com/memphisdev/memphis-k8s#deploy-memphis-with-tls-encrypted-communication-via-ssl


https://github.com/memphisdev/memphis/blob/master/server/configs/seed_tls.conf
  tls {
    # Route cert
    cert_file: "../test/configs/certs/server-cert.pem"
    # Private key
    key_file:  "../test/configs/certs/server-key.pem"
    # Specified time for handshake to complete
    timeout: 2

    # Optional certificate authority verifying connected routes
    # Required when we have self-signed CA, etc.
    ca_file:   "../test/configs/certs/ca.pem"
  }

Memphis Certs
https://github.com/memphisdev/memphis/tree/master/server/configs/certs


Memphis go client library
https://github.com/memphisdev/memphis.go/tree/master

	// for TLS connection:
	...
    memphis.Tls("<cert-client.pem>", "<key-client.pem>",  "<rootCA.pem>")
    ...


Dynamically update TLS certificates
https://opensource.com/article/22/9/dynamically-update-tls-certificates-golang-server-no-downtime	


Environment variables under vscode(rest-gateway):

    "env": {
        "CONNECTION_TOKEN": "memphis",
        "ROOT_USER": "root",
        "MEMPHIS_HOST": "localhost",
        "CLIENT_CERT_PATH": "",
        "CLIENT_KEY_PATH": "",
        "ROOT_CA_PATH": "",
        "USER_PASS_BASED_AUTH": "true",
        "ROOT_PASSWORD": "memphis",
        "CLOUD_ENV": "false"
        "DEV_ENV": "true",
        "DEBUG": "true",
    }

Additional variables:
    "STATION": "syslog",
    "PRODUCER": "syslogsidecar",


Rest-gateway configuration:

    type Configuration struct {
        MEMPHIS_HOST                   string

        ROOT_USER                      string
        ROOT_PASSWORD                  string

        CONNECTION_TOKEN               string

        CLIENT_CERT_PATH               string
        CLIENT_KEY_PATH                string
        ROOT_CA_PATH                   string

        USER_PASS_BASED_AUTH           bool

        DEBUG                          bool
        CLOUD_ENV                      bool
        DEV_ENV                        string ??? bool ???


        VERSION                        string
        JWT_SECRET                     string
        JWT_EXPIRES_IN_MINUTES         int
        REFRESH_JWT_SECRET             string
        REFRESH_JWT_EXPIRES_IN_MINUTES int
        HTTP_PORT                      string

    }

Embedded NATS:
https://natsbyexample.com/examples/embedded/mtls/go

https://github.com/anycable/anycable-go/blob/master/enats/enats.go


Postgress Configuration:


    type Configuration struct {
        ..............................
        METADATA_DB_USER         string     "memphis"
        METADATA_DB_PASS         string     "memphis"
        METADATA_DB_DBNAME       string     "memphis"
        METADATA_DB_HOST         string     "localhost"
        METADATA_DB_PORT         string     "5005"
        METADATA_DB_MAX_CONNS    int
        METADATA_DB_TLS_ENABLED  bool
        METADATA_DB_TLS_MUTUAL   bool
        METADATA_DB_TLS_KEY      string
        METADATA_DB_TLS_CRT      string
        METADATA_DB_TLS_CA       string
        ..............................
}

Build broker:
    go build -o ./_obj/memphis main.go

Send messages to syslogd:
https://www.poftut.com/linux-logger-command-usage-tutorial-with-examples/


Parsing and Generating YAML in Go
https://betterprogramming.pub/parsing-and-creating-yaml-in-go-crash-course-2ec10b7db850

JSON Arrays:
https://www.sohamkamani.com/golang/json/#json-arrays


Convert JSON to Go struct:
https://mholt.github.io/json-to-go/


"Login":

    memphis 
        stand-alone.conf:

                    # Memphis ports
                    port: 6666

        launch.json "stand-alone" configuration
                    "USER_PASS_BASED_AUTH": "true",
                    "CONNECTION_TOKEN": "memphis",

    rest-gateway
        launch.json
                    "DEV_ENV": "true",
                    "DEBUG": "true",
                    "CONNECTION_TOKEN": "memphis",
                    "ROOT_USER": "root",
                    "MEMPHIS_HOST": "localhost",
                    "CLIENT_CERT_PATH": "",
                    "CLIENT_KEY_PATH": "",
                    "ROOT_CA_PATH": "",
                    "USER_PASS_BASED_AUTH": "true",
                    "ROOT_PASSWORD": "memphis",
                    "CLOUD_ENV": "false"




Send log to syslog- adapter:

    logger  --rfc5424 --server 127.0.0.1 --port 5141 --priority user.alert  $(date +%d-%m-%Y_%H-%M-%S)


Print files tree:
   tree . -I _*  -I *_test.go --gitignore > /tmp/tree.log


SyslogDecode — a new open source library from Microsoft for processing Syslog messages (.NET):
- https://rivantsov.medium.com/syslogdecode-a-new-open-source-library-from-microsoft-for-processing-syslog-messages-82a932521a9c
- https://github.com/microsoft/SyslogDecode

The Best Free & Paid Syslog Servers for Linux and Windows:
- https://www.comparitech.com/net-admin/best-free-syslog-servers-for-linux-and-windows/


Setting SO_REUSEPORT and similar socket options in Go 1.11
- https://rohanverma.net/blog/2019/01/08/setting-so_reuseport-and-similar-socket-options-in-go-1-11/


Cancelling a net.Listener via Context in Golang
- https://stackoverflow.com/questions/66755407/cancelling-a-net-listener-via-context-in-golang


Analysis of UDP packet loss problem in Linux system
- https://alibaba-cloud.medium.com/analysis-of-udp-packet-loss-problem-in-linux-system-a5b6bd59d97b#

Why syslog over UDP loses messages and how to avoid that
- https://axoflow.com/syslog-over-udp-message-loss-1/#

Setting SO_REUSEPORT and similar socket options in Go
- https://rohanverma.net/blog/2019/01/08/setting-so_reuseport-and-similar-socket-options-in-go-1-11/

How to Set Go net/http Socket Options
- https://iximiuz.com/en/posts/go-net-http-setsockopt-example/

Where do Sockets live in Go?
- https://about.sourcegraph.com/blog/go/gophercon-2019-socket-to-me-where-do-sockets-live-in-go

syslog.conf(5) - Linux man page
- https://linux.die.net/man/5/syslog.conf

Syslog Configuration Examples
- https://softpanorama.org/Logs/Syslog/syslog_configuration_examples.shtml

syslog.h
- https://sites.uclouvain.be/SystInfo/usr/include/sys/syslog.h.html

Severity Levels:

#define        LOG_EMERG       0        /* system is unusable */
#define        LOG_ALERT       1        /* action must be taken immediately */
#define        LOG_CRIT        2        /* critical conditions */
#define        LOG_ERR         3        /* error conditions */
#define        LOG_WARNING     4        /* warning conditions */
#define        LOG_NOTICE      5        /* normal but significant condition */
#define        LOG_INFO        6        /* informational */
#define        LOG_DEBUG       7        /* debug-level messages */

"debug", "info", "notice", "warning", "err", "crit", "alert", "emerg"



Facility codes:

#define        LOG_KERN        (0<<3)        /* kernel messages */
#define        LOG_USER        (1<<3)        /* random user-level messages */
#define        LOG_MAIL        (2<<3)        /* mail system */
#define        LOG_DAEMON      (3<<3)        /* system daemons */
#define        LOG_AUTH        (4<<3)        /* security/authorization messages */
#define        LOG_SYSLOG      (5<<3)        /* messages generated internally by syslogd */
#define        LOG_LPR         (6<<3)        /* line printer subsystem */
#define        LOG_NEWS        (7<<3)        /* network news subsystem */
#define        LOG_UUCP        (8<<3)        /* UUCP subsystem */
#define        LOG_CRON        (9<<3)        /* clock daemon */
#define        LOG_AUTHPRIV    (10<<3)       /* security/authorization messages (private) */
#define        LOG_FTP         (11<<3)       /* ftp daemon */

        /* other codes through 15 reserved for system use */

#define        LOG_LOCAL0        (16<<3)     /* reserved for local use */
#define        LOG_LOCAL1        (17<<3)     /* reserved for local use */
#define        LOG_LOCAL2        (18<<3)     /* reserved for local use */
#define        LOG_LOCAL3        (19<<3)     /* reserved for local use */
#define        LOG_LOCAL4        (20<<3)     /* reserved for local use */
#define        LOG_LOCAL5        (21<<3)     /* reserved for local use */
#define        LOG_LOCAL6        (22<<3)     /* reserved for local use */
#define        LOG_LOCAL7        (23<<3)     /* reserved for local use */

"kern"      0       kernel messages
"user"      1       random user-level messages
"mail"      2       mail system
"daemon"    3       system daemons
"auth"      4       security/authorization messages
"syslog"    5       messages generated internally by syslogd
"lpr"       6       line printer subsystem
"news"      7       network news subsystem
"uucp"      8       UUCP subsystem
"cron"      9       clock daemon
"authpriv"  10      security/authorization messages (private)
"ftp"       11      ftp daemon

"local0"    16      local use 0
"local1"    17
"local2"    18
"local3"    19
"local4"    20
"local5"    21
"local6"    22
"local7"    23      local use 7


Priority = Facility * 8 + Severity Level