TCP and UDP Latency Monitoring with latency-monitor (OWD vs RTT Explained)

Over the years, we’ve covered a lot of ground when it comes to ping and latency monitoring on the NetBeez blog. From exploring different ping variants like prettyping, gping, fping, and nping, to diving deep into TCP and UDP performance metrics, we’ve demonstrated how network engineers can leverage various tools to diagnose connectivity issues, measure round-trip times, and troubleshoot performance problems.

The open-source ecosystem for network monitoring is remarkably diverse, with each project bringing its own unique philosophy and feature set. Some tools excel at visualizing latency trends, others focus on rapid host discovery, and still others provide granular control over packet parameters. This diversity reflects the varied needs of network professionals—from quick troubleshooting to long-term performance analysis.

Today, we’re exploring latency-monitor by Mircea Ulinic. While we’ve extensively covered ICMP-based ping utilities, latency-monitor takes a different approach by focusing on TCP and UDP latency measurements with some compelling characteristics that set it apart.

What Makes latency-monitor Unique?

Persistent TCP Connections: Unlike tools that establish new connections for each probe, latency-monitor maintains persistent TCP connections to monitored endpoints. This reduces the impact of connection establishment overhead, firewall interference, and TCP handshake variability, giving you more accurate latency data for the actual data path.

One-Way Delay (OWD) and RTT: While most ping tools only measure round-trip time, latency-monitor provides both OWD and RTT metrics. This is invaluable for diagnosing asymmetric routing issues, immediately identifying which direction of a link is experiencing problems.

Nanosecond Precision: The tool collects measurements in nanoseconds and can execute probes every millisecond. This makes it ideal for capturing micro-bursts and transient congestion that would be invisible to tools operating at second-level intervals.

Pluggable Metrics Backend: Rather than forcing you into a single ecosystem, latency-monitor supports multiple backends including Datadog, Prometheus Pushgateway, ClickHouse, ZeroMQ, and CLI/log outputs. This flexibility means easy integration into existing monitoring infrastructure.

Lightweight and Cross-Platform: Built in Python with minimal dependencies, it runs on any OS where Python is available with a small resource footprint.

Dual-Protocol Support: Native TCP and UDP monitoring lets you understand how your network handles both reliable and unreliable traffic patterns.

Bidirectional Deployment Required: Unlike ICMP ping tools, latency-monitor must run on both ends of each monitored link, each side acts as both probe sender and responder, which is essential for accurate OWD measurements.

Real-World Use Cases

Detecting Asymmetric Routing

Traditional RTT-only tools show elevated latency but won’t tell you which direction is problematic. With latency-monitor’s OWD measurements, you immediately see the issue:

• Site A → Site B: 5ms
• Site B → Site A: 45ms

This clarity dramatically speeds up troubleshooting.

Capturing Micro-bursts

Modern networks experience brief congestion events lasting only milliseconds but causing significant application problems. With millisecond-level probing, latency-monitor captures these transient events invisible to traditional monitoring, critical for time-sensitive applications like trading systems or real-time collaboration tools.

Multi-Cloud Connectivity

For organizations with infrastructure across multiple cloud providers, latency-monitor provides continuous visibility into inter-cloud latency between AWS, GCP, Azure, and on-premises data centers, with tagging support to organize metrics by provider, region, or service tier.

WAN Link Performance

When provisioning new circuits or troubleshooting existing ones, deploy latency-monitor at both ends to verify SLA compliance, detect degradation trends before users complain, and provide concrete data when engaging with service providers.

Integration Options

latency-monitor’s pluggable backend supports:

• Datadog: Seamless integration with dashboards and alerts
• ClickHouse: High-volume analytics and long-term trends
• Prometheus Pushgateway: Standard monitoring intervals
• ZeroMQ: Custom integrations and analytics pipelines
• CLI/Log: Debugging and ad-hoc troubleshooting

Getting Started

For installation, configuration examples, and detailed documentation, visit the latency-monitor GitHub repository. You can find the documentation here.

Testing latency-monitor: Cross-Region AWS Deployment

To evaluate latency-monitor in a real-world scenario, I deployed it across two AWS regions: us-east-1 (Virginia) and us-west-1 (Northern California). This setup allowed me to test inter-region latency monitoring over a distance of approximately 2,500 miles.

I installed latency-monitor on EC2 instances in both regions and configured them to monitor each other using both TCP and UDP protocols. The deployment process was straightforward, after installing the Python package and creating a simple TOML configuration file on each instance, the monitoring began immediately.

I started the latency-monitor server  on us-west-1 as a the received of traffic with:

latency-monitor --udp-port=custom_udp_port

and the client on us-east-1 with the following (I used the –no-rtt flag to focus on OWD only).

 latency-monitor  -c latency-to-cal.toml  --no-rtt --no-tcp

Here is the output from us-west-1

{"metric": "udp.wan.owd", "points": [[1770233660780445109, 28090648]], "tags": ["source:US-EAST-1 (VA)", "target:ip-172-31-11-222"]}

{"metric": "udp.wan.owd", "points": [[1770233661780171084, 28060873]], "tags": ["source:US-EAST-1 (VA)", "target:ip-172-31-11-222"]}

{"metric": "udp.wan.owd", "points": [[1770233662780213452, 28014094]], "tags": ["source:US-EAST-1 (VA)", "target:ip-172-31-11-222"]}

{"metric": "udp.wan.owd", "points": [[1770233663780380233, 28054922]], "tags": ["source:US-EAST-1 (VA)", "target:ip-172-31-11-222"]}

{"metric": "udp.wan.owd", "points": [[1770233664780553048, 28134312]], "tags": ["source:US-EAST-1 (VA)", "target:ip-172-31-11-222"]}

Each point shows the timestamp (e.g. 1770233660780445109) and the OWD (e.g. 28090648) in nanoseconds.

I repeated the same on the other direction and I got:

{"metric": "udp.wan.owd", "points": [[1770233805867781302, 29451916]], "tags": ["source:US-WEST-1 (CA)", "target:ip-172-31-29-48"]}

{"metric": "udp.wan.owd", "points": [[1770233806867706894, 29433687]], "tags": ["source:US-WEST-1 (CA)", "target:ip-172-31-29-48"]}

{"metric": "udp.wan.owd", "points": [[1770233807867703136, 29424833]], "tags": ["source:US-WEST-1 (CA)", "target:ip-172-31-29-48"]}

{"metric": "udp.wan.owd", "points": [[1770233808867823378, 29388053]], "tags": ["source:US-WEST-1 (CA)", "target:ip-172-31-29-48"]}

{"metric": "udp.wan.owd", "points": [[1770233809867856190, 29380766]], "tags": ["source:US-WEST-1 (CA)", "target:ip-172-31-29-48"]}

By looking at OWD from both ends we see pretty symmetric latency:

• us-east-1 -> us_west-1: ~28.0ms
• us-west-1 -> us-east-1: ~29.4ms

Note: I checked the clock synchronization between the two machines and they were off by 0.9ms.

When I executed it by with RTT enabled, I see it was around 57.7ms, which is very close to adding the two OWD: 28.0 + 29.4 = 57.8 ms.

{"metric": "udp.wan.rtt", "points": [[1770234500464925026, 57962469]], "tags": ["source:US-WEST-1 (CA)", "target:US-WEST-1 (VA)"]}

{"metric": "udp.wan.rtt", "points": [[1770234501464821072, 57757533]], "tags": ["source:US-WEST-1 (CA)", "target:US-WEST-1 (VA)"]}

{"metric": "udp.wan.rtt", "points": [[1770234502464855339, 57676480]], "tags": ["source:US-WEST-1 (CA)", "target:US-WEST-1 (VA)"]}

{"metric": "udp.wan.rtt", "points": [[1770234503464988136, 57683442]], "tags": ["source:US-WEST-1 (CA)", "target:US-WEST-1 (VA)"]}

{"metric": "udp.wan.rtt", "points": [[1770234504465057162, 57629789]], "tags": ["source:US-WEST-1 (CA)", "target:US-WEST-1 (VA)"]}

The ability to see directional latency differences was particularly valuable. While the round-trip time averaged around 57ms (typical for this geographic distance), the OWD measurements showed symmetric delay between regions—information that’s invisible when using standard ping tools.

Conclusion

latency-monitor fills an important niche by combining high-frequency monitoring, persistent connection stability, flexible metrics backends, and simple deployment. Whether you’re troubleshooting asymmetric routing, verifying WAN SLAs, monitoring multi-cloud connectivity, or capturing transient network issues, it provides the precision and flexibility needed for modern network environments. Its dual TCP/UDP support, nanosecond precision, and pluggable architecture make it a valuable addition to any network engineer’s toolkit.If you want to dig into it more, the author of latency-monitor has a very detailed write up about it here.