HPA Metrics summary HPA self monitoring metrics

HPA collects several metrics used to monitor the platform itself.

No dedicated screen in the portal except the view on status of alerts based on self monitoring metrics and potential saved custom graphs.

UI access to metrics

Custom graph

WebView

Some metrics used to monitor HPA health

Most of those metrics are linked to default alerts provided by HPA in the Hpa Self Monitoring management module.

The Semantics column gives how each metric adds up over time, see Metric semantics.

Metric Comment Semantics
Agents|(.*):ConnectionStatus Captures the connection status between the agent and the EM
- value 1 means “connected”
- value 3 means the agent has stopped sending metrics for a couple of minutes
- no other values

By building when an agent is not connected, this metric is not captured, then lack of data		 UNKNOWN
Enterprise Manager|Connections:Number of Agents Captures the number of agents connected to the collector
By default the maximum number of agents supported is 500		 STATE_COUNTER
SystemEdge Relative to the metrics captured by the infrastructure agent N/A - a metric family, not a single metric

Internal agent metrics events management

A metric event is an action of a tracer, with two main categories

  • start of a method
  • end of a method, meaning by default metric(s) publishing

Those events are managed by specific agent threads in each JVM where an HPA agent is included
Those events are managed by those threads via a specific queue

By default, the queue size by agent is 100k except for the DataServer where the size is 400k

HPA captures 3 self metrics per agent about the agent metrics events management:

Metric Comment
Hpa|SelfMetric|AsyncQueue|HpaFAsyncTracerThread:droppedPerInterval number of events dropped (not added to queue because max has been reached) - previously ignored
Hpa|SelfMetric|AsyncQueue|HpaFAsyncTracerThread:processedPerInterval number of events sents (processed) per interval
Hpa|SelfMetric|AsyncQueue|HpaFAsyncTracerThread:queueSize number of events/metrics to wait to be processed by AsyncThread

Clamped metrics management

Both agent and servers have limits regarding the number of metrics they can manage
All those metrics are self monitoring metrics

Metric Webview path Comment Semantics
Agents|(.*):Is Clamped SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Agents |<host>|<process>|<agent>:Is Clamped Captures the case where new metrics are not saved because of agent maximum number of metrics defined by the EM reached
value = 1 if maximum reached
metrics already recorded will continue to be captured		 UNKNOWN
Agents|(.*):Metric Count SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Agents |<host>|<process>|<agent>:Metric Count Captures the number of metrics per agent received by the EM from the restart of the agent UNKNOWN
Agents|(.*):Historical Metric Count SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Agents |<host>|<process>|<agent>:Historical Metric Count Captures the total number of metrics per agent recently received by the EM
generally higher than Metric Count		 UNKNOWN

This limit is defined in the HPA server with the introscope.enterprisemanager.agent.metrics.limit properties in apm-events-thresholds-config.xml and is compared with agent historical metric
In the EM log, specific messages can appear in case of clamped The Agent <x> is reporting too many metrics (current=…, max=…) or The Agent <x> is exceeding the per-agent metric clamp (current=…, max=…). New metrics will not be accepted

Metric Webview path Comment Semantics
Enterprise Manager|(.*):Number of Metrics SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Enterprise Manager |Connections:Number of Metrics Captures the number of metrics received by the EM since its last restart STATE_COUNTER
Enterprise Manager|(.*):Number of Historical Metrics SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Enterprise Manager |Connections:Number of Historical Metrics Captures the total number of metrics received by the EM STATE_COUNTER
Enterprise Manager|(.*):EM Live Metric Clamped SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Enterprise Manager |Connections:EM Live Metric Clamped Captures the case where new metrics are not saved because of EM maximum number of metrics reached
value = 1 if maximum reached
metrics already recorded will continue to be captured		 STATE_COUNTER
Enterprise Manager|(.*):EM Historical Metric Clamped SuperDomain|Custom Metric Host (Virtual) |Custom Metric Process (Virtual) |Custom Metric Agent (Virtual) |Enterprise Manager |Connections:EM Historical Metric Clamped Captures the case where new metrics are not saved because of EM maximum historical number of metrics reached
value = 1 if maximum reached
metrics already recorded will continue to be captured		 STATE_COUNTER

Two other limits exist linked to those metrics also available in apm-events-thresholds-config.xml: introscope.enterprisemanager.metrics.live.limit and introscope.enterprisemanager.metrics.historical.limit

Another limit exists agent wise and is defined in the .profile of each agent with the introscope.agent.metricClamp
In case this maximum is reached a specific error message is logged in the agent log Absolute metric clamp of … reached and the agent stops sending metrics

Attention point on agent metric count

Two limits apply to the number of metrics of an agent, and they do not count the same metrics:

  • introscope.agent.metricClamp, in the agent .profile: counts the metrics created by the agent since the monitored application started
    • when reached, Absolute metric clamp of … reached is logged in the agent log: no new metric is created, metrics already created keep reporting
    • Is Clamped stays to 0, as this limit is not tracked on EM side; Metric Count is the closest metric to monitor it
  • introscope.enterprisemanager.agent.metrics.limit, in the EM apm-events-thresholds-config.xml: counts the live and historical metrics of the agent known by the EM (Historical Metric Count), kept across agent restarts
    • when reached, a message is logged in the EM log: new metrics are rejected by the EM, metrics already recorded keep reporting
    • Is Clamped moves to 1; nothing is logged in the agent log as long as the agent stays below its own introscope.agent.metricClamp

Comparing both property values is therefore not enough: even with introscope.agent.metricClamp < introscope.enterprisemanager.agent.metrics.limit, historical metrics accumulated by the EM can reach the EM limit, and Is Clamped moves to 1.

Removing old metrics with the procedure described here lowers Historical Metric Count.