Alert

This alert is raised when an agent reach its maximum number of metrics.

To check the limit (by default 50 000 metrics per agent) In the alerts, you have the agent name :

SuperDomain |
 Custom Metric Host (Virtual) |
  Custom Metric Process (Virtual) |
   Custom Metric Agent (Virtual) |
    Agents |
     <Host> |
      <Environment> |
       <Agent> :
        Historical Metric Count

Action

There is two actions possible to handle this

Remove some metrics

It’s often due to metric names than can contain random characters or, for dataserver, a big number of different SQL requests. The best option is to remove oldest metrics using HPA tool :

• you can select the limit period
• you can select agent by regular expression
• you can select metrics by regular expression

Steps :

1. Stop HPA
2. Create a target directory for the data

mkdir ./var/em/data2

2. Select a limit period (metrics older than these days will be removed)
3. Filter metrics by agent and name
4. Execute a dry run :

./controller.sh smartstor -dryRun 
       -src ./var/em/data/ -dest ./var/em/data2 
       -agentsRegexp ".*sd-138618\|v17\|DataServer-v17-Risk" 
       -metricsRegexp ".*SQL.*" 
       -olderThan 250

Check the output (metrics pruned or not, number of metrics parsed and removed) 5. Execute for real (remove -dryRun) :

./controller.sh smartstor 
       -src ./var/em/data/ -dest ./var/em/data2 
       -agentsRegexp ".*sd-138618\|v17\|DataServer-v17-Risk" 
       -metricsRegexp ".*SQL.*" 
       -olderThan 250

5. Exchange the data directories

mkdir ./var/em/data ./var/em/data_backup
mkdir ./var/em/data2 ./var/em/data 

6. Start HPA

(Not recommended) Raise the limit in the EM.

1. You can raise the limit of metrics per agent :

   {{hpa_base_dir}}/module/introscope/config/apm-events-thresholds-config.xml
   

2. Modify :

        <clamp id="introscope.enterprisemanager.agent.metrics.limit">
            <description>
                Most properties here put limits on # of metrics.
                The last limits metric data. The metric clamping properties support hot config.
                Per Agent limit. Takes into account live and historical metrics.
            </description>
            <threshold value="50000"/>
        </clamp>

3. Restart HPA