Currently Connected Clients - Experience Regular Disconnects #3898
Comments
denizciftci-sec
changed the title
|
Is your server behind a reverse proxy or load balancer? |
|
It is difficult to tell by your screenshots what the time axis is but it is quite normal for clients to reconnect quickly when they are sending messages. We do this to avoid stale connections from proxies etc. Each time a message is sent, the client reconnects with another HTTP request (if you are using websocket this is less noticeable). Usually we set the HTTP connection to reuse the TCP connection so there is very little overheads in reconnections. |
|
thanks for the response. Here is the thing, As you can see here, after sometime the endpoint agent connection stops for some reason. When I was initially trigger the client connection via Velociraptor.exe --config client.config.yaml client -v, then I can confirm that the conncetion is back again. Could you please assist here, what kind of further steps should I do for debugging? Status seems Green while client is disconnected. Here shows the green indicator but no connecion between server and client |
|
The color is controlled by the last seen time it will be green if the client is connected less than 15 min ago. To understand why you have connection issues you should go through the troubleshooting steps - in particular run with -v and see when the client is failing to connect. Are you using websockets or http style connections? |
|
We are using http style conncetion in client.config.yaml as below: Client: I think I am able to re-produce the issue. When I override the system.exe and execute the manual command in cmd as The problem appears when I closed the cmd.exe and endpoint communication was gone. But I can confirm that velociraptor service is running under services.exe sc qc velociraptor output> Is there a misconfiguration for end-point? or shall I try to degrade the velociraptor version? |
|
There should be no difference between running as a service or a client - you can also run the command directly (i.e. not in a service) by manually running the same as the binary path name above (this runs the exact same code as the service but in the console): You can also enable local logging temporarily which will cause the service to log everything - just change the client config file to add a logging clause Logging:
output_directory: c:\somedir\
separate_logs_per_component: true
debug:
rotation_time: 604800
max_age: 31536000
info:
rotation_time: 604800
max_age: 31536000
error:
rotation_time: 604800
max_age: 31536000You are looking for the logs with the component VelociraptorClient at the info level - look for why connections are dropping. Usually putting a proxy in front of the clients makes connections more complicated because proxies expect to buffer connections (you have to disable buffering). Try to enable websocket as that is more proxy friendly (if the proxy supports websocket protocol). |
|
Hi Mike, Is there a way to disable the buffering for server config? also how can I enable the websocekting for the proxy? Here is the debug output -I am not sure log has any indicator why the connections are dropped. {"level": "info", "msg": "Starting...", "time": "2024-11-15T16:46:46.7622759Z"} |
|
The buffering is done in the proxy not in the server - by adding a proxy in front of the server it is intercepting the HTTP connection and stopping it from reaching the server in the first place. This is because the way HTTP is used in Velociraptor is a long poll - the client connects to the server and just waits for messages without closing the connection. If the proxy attempts to buffer the connection it will timeout and disconnect it before any data is relayed. This is described here https://docs.velociraptor.app/blog/2020/2020-09-28-velociraptor-network-communications-30568624043a/#ssl-offloading with some potential solutions for nginx To enable websockets, you need to change the client URLs to use websocket protocol. In the client config: Replace |
Hi Team,
I am facing a strange issue relate to connected clients.
Is there a way to see the cause of this glitch? We dont experience in any network problem.
The disconnect interval is per 2-3 mins and it longs 5-10 seconds.
How can I debug it ?
Server and Client "version":"0.73.3"
The text was updated successfully, but these errors were encountered: