Network metrics of a host are captured by the SystemEdge extension of the environment’s HpaInfrastructure agent.
They are found under SystemEdge|<host>|Network|....

Example: SystemEdge|srv-calypso01|Network|Interfaces|InterfaceDetails|enp1s0f1:In Octets 64-bit

Warning

Warning: metrics are collected every 15 s. A sample collected late covers a little more than 15 s, which can rarely produce a value slightly above the link capacity.

Warning: the first sample after a restart of the agent can be abnormally high and must be ignored.

Warning

Interfaces

Each network interface known to Linux gets its own branch. Names follow the Linux predictable naming:

Interface Meaning
enp1s0f0 Physical Ethernet port: en Ethernet, p1 PCI bus 1, s0 slot 0, f0 first port of the card
enp1s0f1 Second port (f1) of the same card
enp1s0f1.2008 VLAN 2008 (802.1Q) sub-interface built on enp1s0f1. Not a card: its traffic is also counted on enp1s0f1
lo Loopback (127.0.0.1): traffic of the host with itself. Never leaves the machine
docker0, br-*, tun0, wlp* Docker bridge, VPN tunnel, Wi-Fi…
  • The load on the wire is measured on the physical interfaces. A VLAN is never added to its parent interface.
  • lo is not a network load. A high value only means many local exchanges between processes.

Metrics

Interface details

Path: SystemEdge|<host>|Network|Interfaces|InterfaceDetails|<interface>:<metric>

Metric Meaning Unit Semantics
In Octets 64-bit Bytes received during the interval, framing included bytes / 15 s DELTA
Out Octets 64-bit Bytes sent during the interval, framing included bytes / 15 s DELTA
In Ucast Packets 64-bit, Out Ucast Packets 64-bit Unicast packets received / sent packets / 15 s DELTA
In/Out Multicast Packets, In/Out Broadcast Packets Multicast / broadcast packets. Useful only to detect a broadcast storm packets / 15 s DELTA
High Speed (Bits/Second) Negotiated link speed. Despite its name, in Mb/s (1000 = 1 Gb/s). Reference value for the utilization Mb/s STATE_COUNTER
Speed (Bits/Second) Same in bit/s, but 32-bit: capped at 4.29 Gb/s. High Speed is preferred bit/s STATE_COUNTER
Operation Status Actual link state: 1 up, 2 down, 5 dormant, 6 not present enum STATE_COUNTER
Admin Status Configured state (1 up, 2 down). Up while Operation Status is down = cable, switch or driver issue enum STATE_COUNTER
MTU (Octets) Largest packet size. Must be the same at both ends of a link bytes GAUGE

The 64-bit counters are preferred. The 32-bit ones (InOctets, OutOctets) wrap every 4 GB.

Interface traffic and health

Path: SystemEdge|<host>|Network|Interfaces|<IP address>:<metric>

These metrics are stored per IP address, not per interface name. The mapping between addresses and interfaces is not exposed by the agent. It is given on the host by the Linux command ip -br addr, which lists each interface with its IP addresses.

Metric Meaning Unit Semantics
InDiscards Packets received without error but dropped by the host, typically because its buffers are full. Main sign of reception saturation packets / 15 s DELTA
OutDiscards Packets dropped before sending, typically because the output queue is full packets / 15 s DELTA
InErrors, OutErrors Packets lost because of transmission errors. Points at hardware: cable, port, card, switch packets / 15 s DELTA
OutQueueLength (Packets) Length of the output queue. Growing = the interface cannot keep up packets STATE_COUNTER
Traffic Incoming (Packets/Sec), Traffic Outgoing (Packets/Sec) Packet rate. A host can saturate in packets/s (CPU, card) before saturating in bytes packets / s GAUGE

TCP

Path: SystemEdge|<host>|Network|TCP:<metric>. Whole host, all interfaces together.

Metric Meaning Unit Semantics
Segments Sent TCP segments sent, retransmissions excluded segments / 15 s DELTA
Segments Retransmitted TCP segments sent again because they were not acknowledged in time. Packets are lost somewhere on the path, possibly outside the host segments / 15 s DELTA
Segments Received, Segments Receive Error Segments received, and those rejected (bad checksum…) segments / 15 s DELTA
EstablishedConnections Open TCP connections (ESTABLISHED + CLOSE-WAIT) connections STATE_COUNTER
ResetConnections Established connections closed abruptly per 15 s GAUGE
AttemptFailures Failed connection attempts: refused, unreachable or timed out per 15 s GAUGE
Active Opens, Passive Opens Connections opened by the host / accepted by the host per 15 s DELTA
Resets Sent Segments with the RST flag sent: the host refuses or kills connections per 15 s DELTA

UDP, IP routing and IP mapping metrics are also collected (Network|UDP, Network|IP|...). They are rarely useful for Calypso and are not described here.

Use cases

Utilization of one direction of a physical interface, from In Octets 64-bit (or Out) and High Speed:

Utilization (%) = Octets x 8 / (15 x High Speed x 1 000 000) x 100  =  Octets / (18 750 x High Speed)

Each direction is computed separately: a link is full-duplex.

Link (High Speed) 50 % 75 % 100 %
100 Mb/s (100) 93.75 MB / 15 s 140.6 MB / 15 s 187.5 MB / 15 s
1 Gb/s (1000) 937.5 MB / 15 s 1.41 GB / 15 s 1.875 GB / 15 s

Sustained utilization above 75 % is a saturation risk. 15 s averages hide shorter bursts: an interval at 60 % very likely contains seconds at 100 %.

Example: a host with a public interface at 1 Gb/s and a private interface at 100 Mb/s carrying the database traffic. The public interface stays below 7 %, while the private one averages 20 % with daily peaks between 53 % and 100 %. The private link is the bottleneck, 10 times slower than the public one while it carries almost all the useful traffic.

Does the host suffer?

Utilization alone does not prove a slowdown. Drops and retransmissions are checked over the same period:

  1. InDiscards / OutDiscards on the IP addresses of the physical interfaces: any increase during a traffic peak means saturation.
  2. InErrors / OutErrors: not a load issue. An increase calls for a check of the cable, the switch port and the link speed.
  3. Retransmission rate = sum of Segments Retransmitted / sum of Segments Sent. Below 0.5 % is normal. Above 1 %, packets are lost on the path.
  • A saturated link does not always produce drops on the host: TCP slows the senders down instead. Discards can stay at 0 for weeks on a link regularly full. Utilization remains the main indicator, and the absence of drops is not proof of good health.
  • Correlating a retransmission peak with a traffic peak points at the interface involved.
  • Operation Status must stay at 1. Any other value is an outage of the interface.
  • High Speed must match the expected speed. A 1 Gb/s card shown at 100 usually means a failed auto-negotiation (cable, switch port) or an offer limited by the hosting provider. It is checked on the host with the Linux command ethtool <interface>: Speed gives the negotiated speed, Supported link modes the speeds the card can reach.

TCP activity

  • EstablishedConnections: baseline of the activity. A sudden change reveals a new client, a connection leak or a restart.
  • ResetConnections and AttemptFailures: a sudden increase reveals a server that refuses or drops connections (restart, exhausted pool, firewall).
  • Traffic Incoming/Outgoing (Packets/Sec): a high packet rate with a low byte rate means many small packets, which load the CPU more than the link.

Reading the data correctly

Situation Symptom Action
Agent restart One isolated, huge point (hundreds of GB) Filtered out with a value selector, e.g. LT the 100 % value of the link
Late collection A 15 s point slightly above 100 % of the link Capped at 100 %, or reasoning on 1 min to 1 h averages
Collection gap No points over a period The agent status is checked before concluding there was no traffic
VLAN and parent VLAN traffic close to its parent interface The two are never added together