- -PacketStreamer sensors collect raw network packets on remote hosts. It selects packets to capture using a BPF filter, and forwards them to a central reciever process where they are written in pcap format. Sensors are very lightweight and impose little performance impact on the remote hosts. PacketStreamer sensors can be run on bare-metal servers, on Docker hosts, and on Kubernetes nodes. - -The PacketStreamer receiver accepts network traffic from multiple sensors, collecting it into a single, central `pcap` file. You can then process the pcap file or live feed the traffic to the tooling of your choice, such as `Zeek`, `Wireshark` `Suricata`, or as a live stream for Machine Learning models. - -### When to use PacketStreamer - -PacketStreamer meets more general use cases than existing alternatives. For example, [PacketBeat](https://github.com/elastic/beats/tree/master/packetbeat) captures and parses the packets on multiple remote hosts, assembles transactions, and ships the processed data to a central ElasticSearch collector. [ksniff](https://github.com/eldadru/ksniff) captures raw packet data from a single Kubernetes pod. - -Use PacketStreamer if you need a lightweight, efficient method to collect raw network data from multiple machines for central logging and analysis. - -### Who uses PacketStreamer? - - * Deepfence [ThreatStryker](https://deepfence.io/threatstryker/) uses PacketStreamer to capture traffic from production platforms for forensics and anomaly detection. - - -# Quickstart: Build and Run PacketStreamer - -## Building PacketStreamer - -Build the `packetstreamer` binary using the `go` toolchain as follows: - -```bash -make -``` - -## Run a PacketStreamer receiver - -```bash -packetstreamer receiver --config [configuration_file] -``` - -You can use an example configuration file: - -```bash -packetstreamer receiver --config ./contrib/config/receiver-local.yaml -``` - -You can process the `pcap` output in a variety of ways: - -```bash -# pass the output file /tmp/dump_file to tcpdump: -tail -c +1 -f /tmp/dump_file | tcpdump -r - -``` - -```bash -# Edit the configuration to write to /dev/stdout, and pipe output to tcpdump: -./packet-streamer receiver --config contrib/receiver-stdout.yaml | tcpdump -r - -``` - -## Run a PacketStreamer sensor - -```bash -sudo packetstreamer sensor --config [configuration_file] -``` - -You can use an example configuration file: - -```bash -sudo packetstreamer sensor --config ./contrib/config/sensor-local.yaml -``` - -When running the sensor remotely, edit the configuration file to target the remote receiver. - -## Testing PacketStreamer - -You can generate some test traffic using your favorite load generator - `ab`, `wrk`, `httperf`, `vegeta`. For example, to use vegeta: - -```bash -# install vegeta -go install github.com/tsenart/vegeta@latest - -echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report -``` - -# Advanced Build Options - -Use the `RELEASE` parameter to strip the binary for a production environment: - -```bash -make RELEASE=1 -``` - -Use the `STATIC` parameter to statically-link the binary: - -```bash -make STATIC=1 -``` - -## Build using Docker - -Use the `docker-bin` target to build `packetstreamer` with Docker. The binary will be statically linked with `musl` and `libpcap`, making it portable across Linux distributions: - -```bash -make docker-bin - -# Alternatively, build a stripped release binary -make docker-bin RELEASE=1 -``` - -## Build a Container Image - -Use the `docker-image` target to build a container image: - -```bash -make docker-image - -# Alternatively, build a stripped release binary -make docker-image RELEASE=1 -``` - -## Deploy on Kubernetes - -PacketStreamer can be deployed on Kubernetes using Helm: - -```bash -kubectl apply -f ./contrib/kubernetes/namespace.yaml -helm install packetstreamer ./contrib/helm/ --namespace packetstreamer -``` - -By default, the Helm chart deploys a DaemonSet with sensor on all nodes and one receiver instance. For the custom configuration values, please refer to the [values.yaml file](/contrib/helm/values.yaml). - -# Advanced Test Scenarios - -## Testing on Docker - -PacketStreamer container images can be tested locally on Docker. - -### Receiver side - -```bash -docker run --rm -it \ - -v $(pwd)/contrib/config:/etc/packetstreamer \ - -v $HOME/container_tmp:/tmp \ - -p 8081:8081 \ - deepfenceio/deepfence_packetstreamer \ - receiver --config /etc/packetstreamer/receiver.yaml -``` - -### Sensor side - -```bash -docker run --rm -it \ - --cap-add=NET_ADMIN --net=host \ - -v $(pwd)/contrib/config:/etc/packetstreamer \ - deepfenceio/deepfence_packetstreamer \ - sensor --config /etc/packetstreamer/sensor-local.yaml -``` - -The sensor requires `--net=host` and `NET_ADMIN` capability in order to capture all of the packets on the host. - -```bash -echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report -``` - -The `pcap` dump file is available in `$HOME/container_tmp/dump_file`. - -## Testing on Vagrant - -On a single host, you may use [Vagrant](https://www.vagrantup.com) to run sensor and receiver hosts easily: - -Install Vagrant according to [the official instructions](https://www.vagrantup.com/downloads). By default, Vagrant uses Virtualbox; you should install [vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), using `vagrant plugin install vagrant-libvirt`. - -Start the two Vagrant VMs, `receiver` and `sensor`: - -```bash -vagrant up - -vagrant status -# Current machine states: -# -# receiver running (libvirt) -# sensor running (libvirt) -``` - -SSH to those VMs (in separate terminals) by using the following commands: - -```bash -vagrant ssh receiver -``` - -```bash -vagrant ssh sensor -``` - -On each, enter the source code directory: +* Portability, works across virtual machines, Kubernetes and AWS Fargate. Linux + and Windows. -### Receiver side +PacketStreamer **sensors** are started on the target servers. Sensors capture +traffic, apply filters, and then stream the traffic to a central reciever. +Traffic streams may be compressed and/or encrypted using TLS. -```bash -cd PacketStreamer -./packetstreamer receiver --config ./contrib/config/receiver-vagrant.yaml -``` +The PacketStreamer **receiver** accepts PacketStreamer streams from multiple +remote sensors, and writes the packets to a local `pcap` capture file -### Sensor side +
-```bash
-cd PacketStreamer
-sudo ./packetstreamer --config ./contrib/config/sensor-vagrant.yaml
-```
+PacketStreamer sensors collect raw network packets on remote hosts. It selects packets
+to capture using a BPF filter, and forwards them to a central reciever process
+where they are written in pcap format. Sensors are very lightweight and impose
+little performance impact on the remote hosts. PacketStreamer sensors can be
+run on bare-metal servers, on Docker hosts, and on Kubernetes nodes.
-Generate some live traffic
+The PacketStreamer receiver accepts network traffic from multiple sensors,
+collecting it into a single, central `pcap` file. You can then process the
+pcap file or live feed the traffic to the tooling of your choice, such as
+`Zeek`, `Wireshark` `Suricata`, or as a live stream for Machine Learning models.
-```bash
-echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report
-```
+## When to use PacketStreamer
-# PacketStreamer Configuration
+PacketStreamer meets more general use cases than existing alternatives. For
+example, [PacketBeat](https://github.com/elastic/beats/tree/master/packetbeat)
+captures and parses the packets on multiple remote hosts, assembles
+transactions, and ships the processed data to a central ElasticSearch
+collector. [ksniff](https://github.com/eldadru/ksniff) captures raw packet
+data from a single Kubernetes pod.
-`packetstreamer` is configured using a yaml-formatted configuration file.
+Use PacketStreamer if you need a lightweight, efficient method to collect raw
+network data from multiple machines for central logging and analysis.
-```yaml
-input: # required in 'receiver' mode
- address: _ip-address_
- port: _listen-port_
-output:
- server: # required in 'sensor' mode
- address: _ip-address_
- port: _listen-port_
- file: # required in 'receiver' mode
- path: _filename_
-tls: # optional
- enable: _true_|_false_
- certfile: _filename_
- keyfile: _filename_
-auth: # optional; receiver and sensor must use same shared key
- enable: _true_|_false_
- key: _string_
-compressBlockSize: _integer_ # optional; default: 65
-inputPacketLen: _integer_ # optional; default: 65535
-logFilename: _filename_ # optional
-pcapMode: _Allow_|_Deny_|_All_ # optional
-capturePorts: _list-of-ports_ # optional
-captureInterfacesPorts: _map: interface-name:port_ # optional
-ignorePorts: _list-of-ports_ # optional
-```
+## Who uses PacketStreamer?
-You can find example configuration files in the [`/contrib/config/`](contrib/config/) folder.
+ * Deepfence [ThreatStryker](https://deepfence.io/threatstryker/) uses
+ PacketStreamer to capture traffic from production platforms for forensics
+ and anomaly detection.
-# Get in touch
+## Get in touch
Thank you for using PacketStreamer.
* [
](https://join.slack.com/t/deepfence-community/shared_invite/zt-podmzle9-5X~qYx8wMaLt9bGWwkSdgQ) Got a question, need some help? Find the Deepfence team on Slack
-* https://github.com/deepfence/PacketStreamer/issues: Got a feature request or found a bug? Raise an issue
-* [productsecurity at deepfence dot io](SECURITY.md): Found a security issue? Share it in confidence
+* https://github.com/deepfence/PacketStreamer/issues: Got a feature request or found a bug? Raise an issue
+* [productsecurity *at* deepfence *dot* io](SECURITY.md): Found a security issue? Share it in confidence
* Find out more at [deepfence.io](https://deepfence.io/)
-# Security and Support
+## Security and Support
For any security-related issues in the PacketStreamer project, contact [productsecurity *at* deepfence *dot* io](SECURITY.md).
Please file GitHub issues as needed, and join the Deepfence Community [Slack channel](https://join.slack.com/t/deepfence-community/shared_invite/zt-podmzle9-5X~qYx8wMaLt9bGWwkSdgQ).
-# License
+## License
The Deepfence PacketStreamer project (this repository) is offered under the [Apache2 license](https://www.apache.org/licenses/LICENSE-2.0).
diff --git a/docs/book.toml b/docs/book.toml
new file mode 100644
index 0000000..c6bc03d
--- /dev/null
+++ b/docs/book.toml
@@ -0,0 +1,6 @@
+[book]
+authors = ["Michal Rostecki"]
+language = "en"
+multilingual = false
+src = "src"
+title = "PacketStreamer"
diff --git a/docs/src/README.md b/docs/src/README.md
new file mode 120000
index 0000000..fe84005
--- /dev/null
+++ b/docs/src/README.md
@@ -0,0 +1 @@
+../../README.md
\ No newline at end of file
diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md
new file mode 100644
index 0000000..6a80a4f
--- /dev/null
+++ b/docs/src/SUMMARY.md
@@ -0,0 +1,12 @@
+# Summary
+
+- [Introduction](./README.md)
+- [Quickstart](./quickstart/README.md)
+ - [Building](./quickstart/building.md)
+ - [Using locally](./quickstart/local.md)
+ - [Using with Docker](./quickstart/docker.md)
+ - [Using on Kubernetes](./quickstart/kubernetes.md)
+ - [Using on Vagrant](./quickstart/vagrant.md)
+- [Using with other tools](./tools/README.md)
+ - [Suricata](./tools/suricata.md)
+- [Configuration](./configuration.md)
diff --git a/docs/src/configuration.md b/docs/src/configuration.md
new file mode 100644
index 0000000..2cc190c
--- /dev/null
+++ b/docs/src/configuration.md
@@ -0,0 +1,32 @@
+# Configuration
+
+`packetstreamer` is configured using a yaml-formatted configuration file.
+
+```yaml
+input: # required in 'receiver' mode
+ address: _ip-address_
+ port: _listen-port_
+output:
+ server: # required in 'sensor' mode
+ address: _ip-address_
+ port: _listen-port_
+ file: # required in 'receiver' mode
+ path: _filename_
+tls: # optional
+ enable: _true_|_false_
+ certfile: _filename_
+ keyfile: _filename_
+auth: # optional; receiver and sensor must use same shared key
+ enable: _true_|_false_
+ key: _string_
+compressBlockSize: _integer_ # optional; default: 65
+inputPacketLen: _integer_ # optional; default: 65535
+logFilename: _filename_ # optional
+pcapMode: _Allow_|_Deny_|_All_ # optional
+capturePorts: _list-of-ports_ # optional
+captureInterfacesPorts: _map: interface-name:port_ # optional
+ignorePorts: _list-of-ports_ # optional
+```
+
+You can find example configuration files in the [`/contrib/config/`](https://github.com/deepfence/PacketStreamer/tree/main/contrib/config)
+folder.
diff --git a/docs/src/quickstart/README.md b/docs/src/quickstart/README.md
new file mode 100644
index 0000000..d05a582
--- /dev/null
+++ b/docs/src/quickstart/README.md
@@ -0,0 +1,7 @@
+# Quickstart
+
+- [Building](./building.md)
+- [Using locally](./local.md)
+- [Using with Docker](./docker.md)
+- [Using on Kubernetes](./kubernetes.md)
+- [Using on Vagrant](./vagrant.md)
diff --git a/docs/src/quickstart/building.md b/docs/src/quickstart/building.md
new file mode 100644
index 0000000..f566ebc
--- /dev/null
+++ b/docs/src/quickstart/building.md
@@ -0,0 +1,34 @@
+# Building
+
+Build the `packetstreamer` binary using the `go` toolchain as follows:
+
+```bash
+make
+```
+
+## Advanced Build Options
+
+Use the `RELEASE` parameter to strip the binary for a production environment:
+
+```bash
+make RELEASE=1
+```
+
+Use the `STATIC` parameter to statically-link the binary:
+
+```bash
+make STATIC=1
+```
+
+## Build using Docker
+
+Use the `docker-bin` target to build `packetstreamer` with Docker. The binary
+will be statically linked with `musl` and `libpcap`, making it portable across
+Linux distributions:
+
+```bash
+make docker-bin
+
+# Alternatively, build a stripped release binary
+make docker-bin RELEASE=1
+```
diff --git a/docs/src/quickstart/docker.md b/docs/src/quickstart/docker.md
new file mode 100644
index 0000000..7bfd958
--- /dev/null
+++ b/docs/src/quickstart/docker.md
@@ -0,0 +1,46 @@
+# Using with Docker
+
+## Build a Container Image
+
+Use the `docker-image` target to build a container image:
+
+```bash
+make docker-image
+
+# Alternatively, build a stripped release binary
+make docker-image RELEASE=1
+```
+
+## Testing on Docker
+
+PacketStreamer container images can be tested locally on Docker.
+
+### Receiver side
+
+```bash
+docker run --rm -it \
+ -v $(pwd)/contrib/config:/etc/packetstreamer \
+ -v $HOME/container_tmp:/tmp \
+ -p 8081:8081 \
+ deepfenceio/deepfence_packetstreamer \
+ receiver --config /etc/packetstreamer/receiver.yaml
+```
+
+### Sensor side
+
+```bash
+docker run --rm -it \
+ --cap-add=NET_ADMIN --net=host \
+ -v $(pwd)/contrib/config:/etc/packetstreamer \
+ deepfenceio/deepfence_packetstreamer \
+ sensor --config /etc/packetstreamer/sensor-local.yaml
+```
+
+The sensor requires `--net=host` and `NET_ADMIN` capability in order to capture
+all of the packets on the host.
+
+```bash
+echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report
+```
+
+The `pcap` dump file is available in `$HOME/container_tmp/dump_file`.
diff --git a/docs/src/quickstart/kubernetes.md b/docs/src/quickstart/kubernetes.md
new file mode 100644
index 0000000..94bf7fa
--- /dev/null
+++ b/docs/src/quickstart/kubernetes.md
@@ -0,0 +1,12 @@
+# Using on Kubernetes
+
+PacketStreamer can be deployed on Kubernetes using Helm:
+
+```bash
+kubectl apply -f ./contrib/kubernetes/namespace.yaml
+helm install packetstreamer ./contrib/helm/ --namespace packetstreamer
+```
+
+By default, the Helm chart deploys a DaemonSet with sensor on all nodes and one
+receiver instance. For the custom configuration values, please refer to the
+[values.yaml file](/contrib/helm/values.yaml).
diff --git a/docs/src/quickstart/local.md b/docs/src/quickstart/local.md
new file mode 100644
index 0000000..a9220a4
--- /dev/null
+++ b/docs/src/quickstart/local.md
@@ -0,0 +1,50 @@
+# Using locally
+
+## Run a PacketStreamer receiver
+
+```bash
+packetstreamer receiver --config [configuration_file]
+```
+
+You can use an example configuration file:
+
+```bash
+packetstreamer receiver --config ./contrib/config/receiver-local.yaml
+```
+
+You can process the `pcap` output in a variety of ways:
+
+```bash
+# pass the output file /tmp/dump_file to tcpdump:
+tail -c +1 -f /tmp/dump_file | tcpdump -r -
+```
+
+```bash
+# Edit the configuration to write to /dev/stdout, and pipe output to tcpdump:
+./packet-streamer receiver --config contrib/receiver-stdout.yaml | tcpdump -r -
+```
+
+## Run a PacketStreamer sensor
+
+```bash
+sudo packetstreamer sensor --config [configuration_file]
+```
+
+You can use an example configuration file:
+
+```bash
+sudo packetstreamer sensor --config ./contrib/config/sensor-local.yaml
+```
+
+When running the sensor remotely, edit the configuration file to target the remote receiver.
+
+## Testing PacketStreamer
+
+You can generate some test traffic using your favorite load generator - `ab`, `wrk`, `httperf`, `vegeta`. For example, to use vegeta:
+
+```bash
+# install vegeta
+go install github.com/tsenart/vegeta@latest
+
+echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report
+```
diff --git a/docs/src/quickstart/vagrant.md b/docs/src/quickstart/vagrant.md
new file mode 100644
index 0000000..a195fcf
--- /dev/null
+++ b/docs/src/quickstart/vagrant.md
@@ -0,0 +1,53 @@
+# Using on Vagrant
+
+On a single host, you may use [Vagrant](https://www.vagrantup.com) to run
+sensor and receiver hosts easily:
+
+Install Vagrant according to [the official instructions](https://www.vagrantup.com/downloads).
+By default, Vagrant uses Virtualbox; you should install
+[vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), using
+`vagrant plugin install vagrant-libvirt`.
+
+Start the two Vagrant VMs, `receiver` and `sensor`:
+
+```bash
+vagrant up
+
+vagrant status
+# Current machine states:
+#
+# receiver running (libvirt)
+# sensor running (libvirt)
+```
+
+SSH to those VMs (in separate terminals) by using the following commands:
+
+```bash
+vagrant ssh receiver
+```
+
+```bash
+vagrant ssh sensor
+```
+
+On each, enter the source code directory:
+
+### Receiver side
+
+```bash
+cd PacketStreamer
+./packetstreamer receiver --config ./contrib/config/receiver-vagrant.yaml
+```
+
+### Sensor side
+
+```bash
+cd PacketStreamer
+sudo ./packetstreamer --config ./contrib/config/sensor-vagrant.yaml
+```
+
+Generate some live traffic
+
+```bash
+echo 'GET http://some_ip:80' | vegeta attack -rate 100 -duration 5m | tee results.bin | vegeta report
+```
diff --git a/docs/src/tools/README.md b/docs/src/tools/README.md
new file mode 100644
index 0000000..2b5e81a
--- /dev/null
+++ b/docs/src/tools/README.md
@@ -0,0 +1,3 @@
+# Using with other tools
+
+- [Suricata](./suricata.md)
diff --git a/docs/src/tools/suricata.md b/docs/src/tools/suricata.md
new file mode 100644
index 0000000..9a29c11
--- /dev/null
+++ b/docs/src/tools/suricata.md
@@ -0,0 +1,24 @@
+# Suricata
+
+PacketStreamer receiver file output can be used for threat detection with
+Suricata with the following command:
+
+```bash
+tail -f /tmp/dump_file | suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin
+```
+
+It assumes that:
+
+* Suricata's configuration file is `/etc/suricata/suricata.yaml`.
+* PacketStreamer receiver is configured with output file to `/tmp/dump_file`.
+
+Example receiver configuration:
+
+```yaml
+input:
+ address: 0.0.0.0
+ port: 8081
+output:
+ file:
+ path: /tmp/dump_file
+```