HPA installation and deployment HPA agent Calypso Calypso Scheduled Tasks

Scheduled tasks launch

Scheduled tasks can be launched in different manner, however, whatever the way used, scheduled tasks should be launched with an HPA Agent

Do not instrument very short ST with complete profile
HPA Agent can have an overhead of 10 to 20 seconds at startup depending of the classpath length
ST that ran during less than one minute will have a major overhead

Use the basic profile (memory monitoring only) for those ones

Two profiles exist for the scheduled task monitoring:

  • The full monitoring is scheduletasks - usual overhead with full range of metrics and features
  • The basic monitoring is scheduletasks_basic - no overhead on launch time or execution time - only CPU, GC, and memory, infrastructure metrics
    • This profile also integrates the data collection session features

After choosing your launch method below (via scheduler or via script), refer to the configuration: Configuration file for ST profiles

back to top

ST Launch method #1 : By shell script

This procedure is suitable for ST being launched from custom shell scripts (also from scheduling software like Dollar Universe, Control-M, etc.), as opposed to using the Calypso Scheduler (see method #2 below).

Identify the script(s) used to launch the STs, then add the below HPA code snippet:

# HPA start modification
if [ -r {{hpa_agent_dir}}/hpa_agent_scheduledtask.sh ]; then
  source {{hpa_agent_dir}}/hpa_agent_scheduledtask.sh "<external ref>"
fi
# HPA end modification

The lines must be included:

  • either at the beginning of the default/custom script
  • either in a parent script
  • either before the call to java

Once this is done, you may proceed with the Configuration file for ST profiles.

Example of ST run script with HPA agent

The example below is a partial script, it illustrates how to use the code snippet in your own scripts.

The important thing to note is that HPA will append its options to $JAVA_OPTS.

#!/usr/bin/env bash
ST_EXTERNAL_REF="REPORT_EOD"
JAVA_OPTS="-Xms512m -Xmx1024m"

# HPA start modification
if [ -r {{hpa_agent_dir}}/hpa_agent_scheduledtask.sh ]; then
  source {{hpa_agent_dir}}/hpa_agent_scheduledtask.sh "${ST_EXTERNAL_REF}"
fi
# HPA end modification

java "$JAVA_OPTS" com.calypso.apps.startup.StartChainQuartzTaskRunner -log -env CAL1800 -user calypso_user -password calypso -taskExtRef "${ST_EXTERNAL_REF}"

back to top

ST Launch method #2 : By Calypso Scheduler

This procedure is to be used if your scheduled tasks are launched via the native Calypso Scheduler component.

Two options (see below):

  • Option 1: You enable the agent on the scheduler which will automatically instrument all ST.
  • Option 2: (not recommended): You manually add the HPA option in the scheduled tasks JVM options using Calypso GUI.

Once this is done, you may proceed with the Configuration file for ST profiles.

Option 1: Agent on scheduler with ST auto-monitoring feature

The scheduler like any other component can run with its own HPA agent.

So first, adapt the scheduler run script. To do so, you can use the agent wizard on HPA portal which will generate the proper code snippet.

HPA Portal > Help menu > Agent wizard for others components > Scheduler

In a second part, the ST auto-monitoring feature must be enabled (disabled by default) on the scheduler agent.

This feature add the required parameters on the fly when the Calypso job start only if Scheduler is, itself, already monitored
Unlike the others parts, this extension modifies Calypso execution to add the HPA parameters

back to top

Activate the auto-monitoring

Go to Agent Settings

Agent Settings

Select Schedule Server

Schedule server selection

Find the Scheluded tasks auto monitoring flag, easiest way is to use the filter and entering auto

Schedule server selection

Activate the flag.

Schedule server selection

back to top

Logs

This feature traces the new parameters added to the scheduled tasks in the Schedule Server agent file, usually

{{hpa_agent_dir}}/wily/logs/IntroscopeAgent-HpaGenericSchedulerServer.log

Option 2: Add HPA parameters in the JAVA OPTIONS (not recommended)

Using the Calypso GUI or directly in the database, add the Java parameters in the JAVA_OPTIONS argument of the scheduled task (same place as the memory options usually)

Why it is not recommended ?

When a database is copied to another environment, ST parameters that have been added in this way will have a wrong environment name

Worse ! It could prevent the start of the ST if agent is not present

This solution requires a strong automated deployment system to update all the parameters on each deployment

HPA Portal > Help menu > Parameters to add on the command line > Others components

IMPORTANT The below Java system properties (-Dxxx=yyy) must follow these requirements:

  • Agent Name: Must be named like ScheduledTask-<external_ref>
  • Agent Profile: Either scheduletasks or scheduletask_basic.

back to top

Configuration file for ST profiles (common to both launch methods)

Whether the ST are launched via the Calypso Scheduler or via custom script, a common configuration file exists to assign a given monitoring profile to a given external reference (or pattern).

Configuration file is :

{{hpa_agent_dir}}/override_config/automatic_st_monitoring.list

This file is not overwritten when a new agent is installed/deployed

If the file does not exist, use the template:

cp {{hpa_agent_dir}}/default_config/automatic_st_monitoring.list {{hpa_agent_dir}}/automatic_st_monitoring.list

or if the template does not exist, create the file with this content :

// You can comment
// line pattern is <regular expression>;;<profile>
// profile can be : scheduletasks (full monitoring) or scheduletasks_basic (only cpu / memory / gc, no overhead)
EXAMPLE_HPA_MY_EXTERNAL_REF.*;;scheduletasks
// the token HPA_MONITOR_ALL is used as a default value if present
// if not present, default action is to do not add monitoring on ST
HPA_MONITOR_ALL;;scheduletasks_basic

The lines are evaluated in order, add as many lines as needed - evaluation stops when a pattern matches

  • 1st part of the line is a regular expression that must match the external reference of the scheduled task
  • 2nd part is the profile to apply - scheduletasks or scheduletasks_basic

If the constant HPA_MONITOR_ALL is present in the file and the no other lines match previously then this default profile is applied

If the constant is not found, and the external reference did not match any line, there is no monitoring considered

It is recommended to keep HPA_MONITOR_ALL on the last line

back to top

Scheduled tasks metrics