Those sets of files can be analysed and/or shared in case of HPA issue:

Component	Path
Server	{{hpa_base_dir}}/hpa/modules/introscope/logs/em.log*
Server	{{hpa_base_dir}}/hpa/modules/hpa-extractor-*/logs/*
Server	{{hpa_base_dir}}/hpa/modules/nginx/logs/output.log
Agent	{{hpa_agent_dir}}/wily/logs/*

Agents behaviour

• In case of violent crash of the JVM, as the agent is running inside, it will also end up
  • For JVM running very shortly, like ST, in a such context, agent can even miss its first sent of data to the portal, and then the ST execution could not appear into
• In case of JVM stop, the agent will send data immediately
  • Depending on the nature of metrics captured by the agent, close to the JVM end, some values could be not fully accurate
  • Specially true for some ST metrics, as they JVM life duration is limited compared to back-end components

Debug session

• Minimum free space required for the agents at 2.59T
  • A property is missing in the agent profile
  • hpa.dumpDirectory.freeSpaceInMb=20000
  • Details available here
• Disk free space for agent at 0 - for all agents of the same host
  • The user used to deploy the agent is probably not the correct one or is not having the sufficient Linux privileges
• Debug session monitor not running on a host
  • The user used to deploy the agent is probably not the correct one or is not having the sufficient Linux privileges

Deactivating HPA agents

3 ways are available to quickly deactivate the HPA agents:

• Select all the flags and services in the Agents Settings and deactivate them
  • Does not require any Calypso JVM restart
• Change the ENABLE value in hpa_agent_config.sh
  • Requires a Calypso JVM restart

export ENABLE=0    

• Rename the HPA agent folder
  • Requires a Calypso JVM restart

Extractor

• In case of OOO (Out Of Memory) issue on the hpa extractor, memory must be increased:
  • Edit the file {{hpa_base_dir}}/hpa/versions/<version>/hpa-extractor-<extractor version>/module_controller.bash
  • Modify the line HPA_MODULE_JOPTS="$HPA_MODULE_JOPTS -Xmx2G" and consider the value to use
  • Stop the extractor via the controller {{hpa_base_dir}}/hpa/./controller.sh stop hpa-extractor-<extractor version>
  • Start again (the controller starts only the stopped component(s)) {{hpa_base_dir}}/hpa/./controller.sh start

JVM memory allocation

-Xmx runtime parameter sets the Maximum amount of JVM Heap Memory for this JVM. An additional parameter, that is usually used in tandem with -Xmx, is -Xms, which controls the initial allocation of JVM Heap Memory.

For example, if the -Xms512m -Xmx3g combination of parameters were used to start a Java program, HPA would show 512 MB as the value GC Heap:Bytes Total metric. When the GC Heap:Bytes In Use reaches a certain point near the 512 MB mark, the JVM will increase the GC Heap:Bytes Total metric by a certain percentage (the exact number depends on the JVM provider and the GC algorithms used).

If the Heap memory usage ever grows (due to a potential memory leak or an unoptimized JVM heap for this particular application usage), the JVM will throw a java.lang.OutOfMemoryError indicating that a maximum of 3GB of HEAP allocated to this JVM was not enough to perform its activities.

To avoid false alert when monitoring the memory usage of a JVM, it is recommended to set the threshold to 95% or higher for GC Monitor:Percentage of Java Heap Used.

Nginx

• nginx: [emerg] bind() to 0.0.0.0:443 failed (13: Permission denied)
  • Visible in the nginx output.log file
  • If nginx is listening on a port lower than 1024, special rights must be allocated
  • Those rights can disappear when nginx is reinstalled or when security policies patches are deployed
  • The best action to correct this issue is to launch this command
    • setcap cap_net_bind_service=ep {{hpa_base_dir}}/hpa/modules/nginx-<nginx version>-bin/sbin/nginx