diff --git a/A General Overview of TCPCopy Architecture.md b/A General Overview of TCPCopy Architecture.md new file mode 100644 index 00000000..8c4d6b9b --- /dev/null +++ b/A General Overview of TCPCopy Architecture.md @@ -0,0 +1,133 @@ +## A General Overview of TCPCopy Architecture + +TCPCopy is a tool for replicating requests, originally designed to capture packets from the data link layer and send them from the IP layer. Its architecture has evolved from a single program into a distributed load testing framework. The latest architecture recommends using `pcap` to capture packets on auxiliary servers, which enhances efficiency and scalability but requires more machine resources and specialized knowledge. Adopting a high-performance bypass mechanism in the third architecture minimizes impact on live services. + +In the field of server-based request replay, there are generally two main approaches: offline replay and real-time online replication. Researchers often focus on offline replay, with little exploration in real-time replication. Based on feedback from SIGCOMM reviewers, there seems to be minimal research in real-time replication. + +For real-time request replication, there are generally two types: + +1) Application-layer request replication +2) Packet-level request replication + +Traditional approaches often replicate requests at the application layer, for instance, server-based request replication. Although easier to implement, this approach has several drawbacks: + +1) Replicating requests from the application layer requires traversing the entire protocol stack, which can consume resources, such as valuable connection resources. +2) Testing becomes coupled with the actual application, increasing the potential impact on live systems. Server-based replication, for instance, can cause request processing times to depend on the slowest request (e.g., `max(actual request time, replicated request time)`). +3) Supporting high-stress replication is difficult and may severely impact live systems, according to feedback from some users. +4) Network latency is challenging to control. + +Packet-level request replication, however, can avoid traversing the entire protocol stack. The shortest path can capture and send packets directly from the data link layer, or alternatively, at the IP layer. As long as TCP is not involved, the impact on live systems is significantly reduced. + +From a packet-based approach, server-based request replication is indeed the right direction, with immense potential. Unfortunately, the creator of `tcpreplay` only briefly explored this path with `flowreplay` before abandoning it. From what I’ve seen, this area remains under-researched (most studies instead focus on entire networks; even SIGCOMM reviewers haven't suggested similar research approaches). + +## Diving into TCPCopy’s Architectural Evolution + +The TCPCopy architecture has gone through three generations. The core principle remains the same: leveraging online packet data to simulate a TCP client stack and deceive the upper application service on the test server. Since TCP interactions are inherently bidirectional, it is typically necessary to know the test server's response packets to construct a suitable request packet for the test server based on online requests. Thus, regardless of the implementation, capturing response packets is essential unless the TCP protocol is drastically altered. + +The three architectures differ primarily in where response packets are intercepted. + +The earliest TCPCopy architecture was as follows: + +![](images/first.png) + +Figure 1. Initial TCPCopy Architecture Diagram. + +As shown above, TCPCopy captured request packets from the data link layer (via `pcap`) and sent packets from the IP layer. The test server's TCP stack received no interference from mechanisms like `ip queue` or `nfqueue`, and response packets would directly return to the live machine (through routing settings). TCPCopy could capture these response packets at the data link layer, with packets typically being discarded at the IP layer (unless the client IP was the IP of the live machine itself, in which case the packets would reach the TCP layer but be reset by TCP). + +Special thanks to TCPCopy’s originator, Wang Bo, who pioneered this initial exploration. Designed and implemented in 2009, this original 300-line version supported the early development of NetEase’s ad delivery system, achieving zero deployment errors and resolving hundreds of issues pre-launch. While the initial, basic version had limited applications, I significantly modified the code in late 2010, expanding it to over 1,000 lines, which deepened my foundational understanding of the TCP protocol. + +Returning to the architecture, this early version generally functioned only within the same network segment. For web applications, it was mostly limited to single-machine traffic, lacking the depth required to fully uncover potential issues or explore the broader capabilities of NetEase’s ad delivery system. + +### Summary of the First Architecture + +**Advantages:** + +1) Simple and direct +2) Suitable for smoke testing +3) Relatively realistic testing outcomes + +**Disadvantages:** + +1) Higher impact on the live environment due to response packets returning to the live machine (though still less than application-layer replication). +2) Network segment limitations. +3) For web applications, it is challenging to utilize multiple live flows, which limits its value for stress testing. +4) Internal applications are heavily restricted because the client IP of requests cannot match the replicated live machine’s IP address. + +## The Second Architecture + +This architecture was initially designed by tcpcopy’s originator Wang Bo (designed in 2010 and handed over to me, in June 2011). The general architecture is outlined below: + +![](images/second.png) + +Figure 2. The Second TCPCopy Architecture Diagram. + +As shown in the diagram, tcpcopy now captures packets from the IP layer and also sends packets from the IP layer. Unlike the first architecture, this design intercepts response packets at the testing server, with the intercept program returning the necessary response packet information to tcpcopy. This approach enables distributed load testing, which greatly advanced tcpcopy’s evolution compared to the first architecture. + +To analyze the interception of response packets, in theory, we could capture response packets at the IP layer or data link layer on the testing server. Let’s examine these options: + +1) Capturing at the data link layer: Normally, the response packet would return to the actual client initiating the request, which would affect the client’s TCP module (frequent resets) and, under high load, could cause unnecessary interference to the switch, router, and even the entire network. + +2) Capturing at the IP layer: The netlink technology offers a solution to the above issues. Netlink is a communication method for interaction between user-space processes and the kernel. Specifically, we can use kernel modules such as ip_queue (for kernel versions below 3.5) or nfqueue (for kernel 3.5 or above) to capture response packets. + +We chose the second method, which captures response packets at the IP layer. Once a response packet is passed to intercept, we can retrieve the essential response packet information (generally TCP/IP header information) and transmit it to tcpcopy. We can also use a verdict to instruct the kernel on handling these response packets. If there is no whitelist setting, these response packets will be dropped at the IP layer, making them undetectable by tcpdump (which operates at the data link layer). + +This design allows for the replication of traffic from multiple live servers onto a single testing server. Within intercept, routing information is retained to determine which tcpcopy instance to return the response packet information. However, intercept does consume resources on the testing server, and ip_queue or nfqueue may not perform efficiently, particularly for high-stress tests or short-connection load testing, leading to significant challenges. + +### Summary of this architecture + +**Advantages:** + +1) Supports replicating traffic from multiple live servers +2) Minimizes impact on live servers, typically only returning TCP/IP header information + +**Disadvantages:** + +1) More complex than the first architecture +2) Performance limits are often tied to ip_queue or nfqueue +3) Intercept lacks scalability, restricted by ip_queue and nfqueue’s inability to support multi-process response packet capture +4) Intercept affects the final test results on the test server, especially under high-stress conditions +5) Incomplete testing on the test server (no coverage of data link layer egress) +6) Less convenient for maintenance + +## The Third Architecture + +The following diagram illustrates the latest architecture, designed specifically for extreme testing. This setup offloads the intercept function from the test server and places it on a separate, dedicated assistant server (preferably an idle server on the same network segment). In this setup, response packets are captured at the data link layer instead of the IP layer, significantly reducing interference with the test machine (aside from routing configuration) and greatly enhancing the ability to capture response packets. Consequently, this architecture provides a more realistic testing environment. + +![](images/tcpcopy.png) + +Figure 3. The Third TCPCopy Architecture Diagram. + +### Detailed Overview + +Routing information is configured on the test server, where the application to be tested routes the necessary response packets to the assistant server. On the assistant server, we capture the response packets at the data link layer, extract useful information, and return it to the corresponding tcpcopy instance. + +To achieve high efficiency, this architecture recommends using pcap for packet capture, allowing filtering to be handled in the kernel space. Without pcap, filtering would only be possible in user space. Filtering can be configured on either the intercept or tcpcopy side (using the -F parameter, similar to tcpdump filters), enabling packet capture to be handled in a divide-and-conquer approach across multiple instances. This design improves scalability and is ideal for handling extremely high concurrency. + +This architecture requires more machine resources and is more challenging to use, as it involves knowledge of TCP, routing, and pcap filters (similar to tcpdump filtering conditions). Therefore, this architecture requires users to be familiar with the above knowledge. + +It’s important to note that in certain scenarios, pcap packet capture may experience higher packet loss rates than raw socket capture. Therefore, it’s advisable to use pf_ring for support or switch to raw socket capture. + +### Summary + +**Advantages:** + +1. Provides a more realistic testing environment +2. Highly scalable +3. Suitable for high concurrency scenarios +4. Avoids the limitations of ip_queue and nfqueue +5. Virtually no performance impact on the test server +6. Easier maintenance on the test server running services +7. Will not crash alongside the service-running server in the event of a failure + +**Disadvantages:** + +1. More challenging to operate +2. Requires additional machine resources +3. Demands more knowledge +4. The assistant server (running intercept) must be on the same network segment as the test server (in principle) + +All three architectures have their merits. Currently, only the second and third architectures are open-source, and tcpcopy defaults to the third architecture. + +Finally, to minimize or eliminate the impact on the live environment when replicating requests, consider using the following approach: + +- Use a high-performance bypass mechanism (if using mirroring, modify the destination address of client data packets) to replicate request data packets to a separate system. In this separate system, apply the third architecture to capture requests via the pcap interface and then forward them to the application on the test server. diff --git a/README.md b/README.md index 11f5b456..4e24a46b 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ TCPCopy is a TCP stream replay tool for realistic testing of Internet server app [An Overview of TCPCopy for Beginners](Beginners.md) +[A General Overview of TCPCopy Architecture](A General Overview of TCPCopy Architecture.md) + ## Description Although real live traffic is crucial for testing Internet server applications, accurately simulating it is challenging due to the complexity of online environments. To enable more realistic testing, TCPCopy was developed as a live flow reproduction tool that generates test workloads closely resembling production workloads. TCPCopy is widely used by companies in China. @@ -13,6 +15,7 @@ Although real live traffic is crucial for testing Internet server applications, TCPCopy minimally impacts the production system, consuming only additional CPU, memory, and bandwidth. The reproduced workload mirrors the production environment in terms of request diversity, network latency, and resource usage. ## Use Cases + * Distributed Stress Testing - Use TCPCopy to replicate real-world traffic for stress testing your server software, uncovering bugs that only appear under high-stress conditions. * Live Testing @@ -22,7 +25,7 @@ TCPCopy minimally impacts the production system, consuming only additional CPU, * Performance comparison - Compare system performance across different versions or configurations. -## Architecture +## Architecture ![tcpcopy](images/tcpcopy.png) @@ -47,12 +50,13 @@ For `intercept`, you have two options: `git clone git://github.com/session-replay-tools/intercept.git`. For `tcpcopy`, you also have two options + * [Download the latest tcpcopy release](https://github.com/session-replay-tools/tcpcopy/releases). * Clone the repository: `git clone git://github.com/session-replay-tools/tcpcopy.git`. - ## Installing intercept on the Assistant Server + 1. Navigate to the `intercept` directory:
`cd intercept` 2. Run the configuration script:
@@ -64,6 +68,7 @@ For `tcpcopy`, you also have two options `make install` ### Configure Options for `intercept` + - `--single` Run `intercept` in non-distributed mode. @@ -74,18 +79,19 @@ For `tcpcopy`, you also have two options Compile `intercept` with debug support, with logs saved to a file. ## Installing `tcpcopy` on the Online Server + 1. Navigate to the `tcpcopy` directory:
`cd tcpcopy` -3. Run the configuration script:
+2. Run the configuration script:
`./configure`
Include any necessary configuration options as needed. -4. Compile the source code:
+3. Compile the source code:
`make` -5. Install the `tcpcopy` tool:
+4. Install the `tcpcopy` tool:
`make install` - ### Configure Options for `tcpcopy` + - `--offline` Replay TCP streams from a pcap file. @@ -110,38 +116,38 @@ For `tcpcopy`, you also have two options - `--with-debug` Compile `tcpcopy` with debug support, with logs saved to a file. - ## Running TCPCopy + Assume that both `tcpcopy` and `intercept` are configured using `./configure`. 1. **On the Target Server Running Server Applications:** - + Configure the route rules to direct response packets to the assistant server. For example, if `61.135.233.161` is the IP address of the assistant server, use the following route command to direct all responses from clients in the `62.135.200.x` range to the assistant server: `route add -net 62.135.200.0 netmask 255.255.255.0 gw 61.135.233.161` 2. **On the Assistant Server Running `intercept` (Root Privilege or CAP_NET_RAW Capability Required):** - + `./intercept -F -i ` - + Note that the filter format is the same as the pcap filter. For example: - + `./intercept -i eth0 -F 'tcp and src port 8080' -d` - + In this example, `intercept` will capture response packets from a TCP-based application listening on port 8080, using the eth0 network device. 3. **On the Online Source Server (Root Privilege or CAP_NET_RAW Capability Required):** - + `./tcpcopy -x localServerPort-targetServerIP:targetServerPort -s [-c ]` - + For example (assuming 61.135.233.160 is the IP address of the target server): - + `./tcpcopy -x 80-61.135.233.160:8080 -s 61.135.233.161 -c 62.135.200.x` - + In this example, `tcpcopy` captures packets on port 80 from the current server, changes the client IP address to one from the 62.135.200.x range, and sends these packets to port 8080 on the target server (61.135.233.160). It also connects to 61.135.233.161 to request `intercept` to forward response packets. While the `-c parameter` is optional, it is used here to simplify route rules. - ## Note + 1. Platform: Tested only on Linux (kernel 2.6 or above). 2. Packet Loss: TCPCopy may lose packets, which could result in lost requests. 3. Permissions: Requires root privilege or the `CAP_NET_RAW` capability (e.g., setcap CAP_NET_RAW=ep tcpcopy). @@ -152,19 +158,21 @@ Assume that both `tcpcopy` and `intercept` are configured using `./configure`. 8. Help: For more information, run `./tcpcopy -h` or `./intercept -h`. ## Influential Factors + Several factors can impact TCPCopy, as detailed in the following sections. ### 1. Capture Interface + By default, `tcpcopy` uses a raw socket input interface to capture packets at the network layer on the online server. Under high load, the system kernel may drop some packets. If configured with `--pcap-capture`, `tcpcopy` captures packets at the data link layer and can filter packets in the kernel. Using `PF_RING` with pcap capturing can reduce packet loss. For optimal capture, consider mirroring ingress packets via a switch and distributing the traffic across multiple machines with a load balancer. - ### 2. Sending Interface + `tcpcopy` defaults to using a raw socket output interface to send packets at the network layer to the target server. To avoid `ip_conntrack` issues or improve performance, use `--pcap-send` to send packets at the data link layer instead. +### 3. On the Way to the Target Server -### 3. On the Way to the Target Server Packets sent by `tcpcopy` may face challenges before reaching the target server. If the source IP address is the end-user's IP (by default), security devices may drop the packet as invalid or forged. To test this, use `tcpdump` on the target server. If packets are successfully sent within the same network segment but not across segments, packets may be dropped midway. To address this, deploy `tcpcopy`, target applications, and `intercept` within the same network segment. Alternatively, use a proxy in the same segment to forward packets to the target server in another segment. @@ -172,13 +180,15 @@ To address this, deploy `tcpcopy`, target applications, and `intercept` within t Deploying the target server’s application on a virtual machine within the same segment may still encounter these issues. ### 4. OS of the Target Server -The target server might use `rpfilter` to verify the legitimacy of source IP addresses, dropping packets deemed forged. If packets are captured by `tcpdump` but not processed, check `rpfilter` settings and adjust or remove them as needed. Other issues like `iptables` settings may also affect `tcpcopy`. +The target server might use `rpfilter` to verify the legitimacy of source IP addresses, dropping packets deemed forged. If packets are captured by `tcpdump` but not processed, check `rpfilter` settings and adjust or remove them as needed. Other issues like `iptables` settings may also affect `tcpcopy`. ### 5. Applications on the Target Server + Applications on the target server may not process all requests promptly. Bugs or limitations in the application can lead to delayed responses or unprocessed requests in the socket buffer. ### 6. OS of the Assistant Server + Ensure that `ip_forward` is set to false on the assistant server to prevent it from routing packets and ensure it functions as a black hole. ## Logical Analysis of the Issue Where the Test Server Fails to Receive Data @@ -198,9 +208,9 @@ Assume that during the `tcpcopy` test, the application on the test server does n **1.3 Test Server Responds with the Second Handshake Packet:** Capture packets on the assistant server to check if the second handshake packet has reached it. - - **If the packet hasn't reached the assistant server,** it suggests that the routing setup is not effective, and therefore `intercept` cannot capture the second handshake packet, preventing further replay. A potential solution is to run `intercept` directly on the test server (note: keep the routing setup unchanged, and ensure that the `-c` parameter in `tcpcopy` is not set to the IP address used by `tcpcopy` to connect to `intercept`, or else `tcpcopy` won’t connect to `intercept`). +- **If the packet hasn't reached the assistant server,** it suggests that the routing setup is not effective, and therefore `intercept` cannot capture the second handshake packet, preventing further replay. A potential solution is to run `intercept` directly on the test server (note: keep the routing setup unchanged, and ensure that the `-c` parameter in `tcpcopy` is not set to the IP address used by `tcpcopy` to connect to `intercept`, or else `tcpcopy` won’t connect to `intercept`). - - **If the second handshake packet is captured,** check if `ip_forward` is enabled. If it is, disable this setting, as it may cause the response packets to be sent directly back to the client, interfering with the test. +- **If the second handshake packet is captured,** check if `ip_forward` is enabled. If it is, disable this setting, as it may cause the response packets to be sent directly back to the client, interfering with the test. ### 2. **If the SYN packet does not reach the test server, there are two possible scenarios:** @@ -209,16 +219,17 @@ Assume that during the `tcpcopy` test, the application on the test server does n **2.2 `tcpcopy` Packets Not Captured on the Online Server:** - - **If no `all clt:xx` information is found in `tcpcopy`'s log,** it indicates that `tcpcopy` is unable to capture packets at the IP layer. In this case, use the `--pcap-capture` option to capture packets at the data link layer. Set the `-F` parameter (e.g., 'tcp and dst port 80 and dst host 10.100.1.2') and the `-i` parameter (network interface) to bypass IP layer capturing. - - - **If `all clt:xx`, where `xx > 0`, is seen in `tcpcopy`'s log,** it means `tcpcopy` successfully captured the packet, but it was filtered out by the IP layer on the online server. Check `iptables` restrictions on the output chain, among other settings. If `iptables` is the problem and cannot be modified on the online server, use the `--pcap-send` option to send packets from the data link layer. +- **If no `all clt:xx` information is found in `tcpcopy`'s log,** it indicates that `tcpcopy` is unable to capture packets at the IP layer. In this case, use the `--pcap-capture` option to capture packets at the data link layer. Set the `-F` parameter (e.g., 'tcp and dst port 80 and dst host 10.100.1.2') and the `-i` parameter (network interface) to bypass IP layer capturing. +- **If `all clt:xx`, where `xx > 0`, is seen in `tcpcopy`'s log,** it means `tcpcopy` successfully captured the packet, but it was filtered out by the IP layer on the online server. Check `iptables` restrictions on the output chain, among other settings. If `iptables` is the problem and cannot be modified on the online server, use the `--pcap-send` option to send packets from the data link layer. ## Release History + + 2014.09 v1.0 TCPCopy released + 2024.09 v1.0 Open source fully uses English ## Bugs and Feature Requests + Have a bug or a feature request? [Please open a new issue](https://github.com/session-replay-tools/tcpcopy/issues). Before opening any issue, please search for existing issues. ## Copyright and License diff --git a/images/first.png b/images/first.png new file mode 100644 index 00000000..0e51d7b2 Binary files /dev/null and b/images/first.png differ diff --git a/images/second.png b/images/second.png new file mode 100644 index 00000000..24611ac5 Binary files /dev/null and b/images/second.png differ