This document is part of the Caché Installation Guide for UNIX and Linux. This document explains how you can calculate the best parameters for your system. It is divided into two sections:
For optimal Caché performance, you need to calculate proper values for certain Caché system parameters. These values allow you to determine whether you need to adjust certain system level parameters. The values you choose should minimize swapping and paging that require disk accesses, and thus improve system performance.
Review this section carefully and calculate the proper values for both your operating system and Caché before proceeding. Use the tables provided here to record the current and calculated values for your system level parameters. You can then refer to these tables when you install Caché. After your system is running, you may need to adjust these values to gain optimal performance.
If you are not already familiar with the memory organization at your operating system level, consult the appropriate system documentation.
Determine Memory and Disk Requirements
This section outlines the basic system requirements for most systems. Because these requirements vary by platform, consult your platform documentation for additional information.
Calculate Memory Requirements
Use the breakdown of memory usage shown in the following table to calculate the memory your system needs for Caché. It is assumed that you are using the Caché Configuration Manager (from a Windows client) to change parameters.
Memory Requirements
Components Memory Requirements
Operating system 1800 KB (operating system dependent)
Caché 842 KB
Global database cache 2.2 KB per buffer if using a 2-KB block size, or 8 KB per buffer if using an 8-KB block size.
Routine cache 32 KB
User overhead 1024 KB per process. The user partition size, which comprises some of user overhead, is tunable via the Caché Configuration Manager. The partition grows dynamically up to 16 MB.
Network (if present) 300 KB per port for each network system process (DMNNET, DCP, and RECEIVE). Caché ports have two DMNNET system processes per port. In addition, there is a network shared memory requirement, which depends on the number of ports and the number of remote hosts configured. For a basic system, this requirement is about 304 KB.
By default, Caché allocates 7 MB for its swap space for shared memory, global buffers, and routine buffers. If you are installing Caché for the first time, this amount should be sufficient. If you plan to run large applications or support large numbers of users, tune the system according to the following formula:
                        (number of routine buffers)*32 KB
                      +   (number of global buffers)*8 KB
                      +                              4 MB
                      ___________________________________
                      =              Shared memory needed
Calculate Swap Space
The amount of swap space available on your system should never be less than the amount of real memory plus 256 KB.
With this minimum in mind, InterSystems recommends the following value as the minimum amount of swap space needed for Caché:
                       ((# of processes + 4)† * (1024 KB)‡
                     +           total global buffer space
                     +          total routine buffer space
                     _____________________________________
                     =                  Minimum swap space
† You add 4 to the # of processes for the Caché Control Process, the Write daemon, the Garbage Collector, and the Journal daemon. You should also add 1 for each slave Write daemon. The # of processes must include all user and jobbed processes which might run concurrently. If you are running networking, add 1 for the RECEIVE system process plus the number of DMNNET daemons you have running (2 per port). If your system is a server, add the number of DCP processes that will be created when all potential clients access the server.
‡ The 1024 KB number is approximate. It is based on the current size of the Caché executable and grows with the partition size you allocate in the Configuration Manager to each Caché process. On most systems, you will want to provide only as much swap space as necessary. However, some systems require you to provide swap space for the worst case. Under these conditions, you need to increase this number to as high as 1.5 MB, depending on the partition size you specify.
Be sure to confirm that your UNIX system permits the amount of swap space you require. For specific information about swap space on your system, consult your UNIX operating system manual.
Solaris Swap Space
To calculate swap space for the Solaris 8 and 9 platforms:
swap –l
Example:
>swap –l
swapfile                   dev     swaplo blocks   free
/dev/dsk/c0t2d0s0         136,0      16    526304   526304
/dev/dsk/c0t2d0s1         136,1      16    2101184 2101184
Tru64 UNIX Swap Space
To display swap space for Tru64 UNIX:
/usr/sbin/swapon -s 
Information for each swap partition is displayed similar to the following example:
Swap partition /dev/disk/dsk1b (default swap):
         Allocated space:        16384 pages (128MB)
         In-use space:           10452 pages ( 63%)
         Free space:              5932 pages ( 36%)
   
Swap partition /dev/disk/dsk4c:
         Allocated space:        128178 pages (1001MB)
         In-use space:            10242 pages (  7%)
         Free space:             117936 pages ( 92%)
    
Total swap allocation:
         Allocated space:        144562 pages (1.10GB)
         Reserved space:          34253 pages ( 23%)
         In-use space:            20694 pages ( 14%)
         Available space:        110309 pages ( 76%)
AIX Swap Space
To display swap space for AIX:
lsps –a
Page Space  Physical Volume   Volume Group    Size %Used  
Active  Auto  Type
hd6         hdisk2            rootvg                   512MB      72     
yes   yes    lv
HP-UX Swap Space
To display swap space for HP-UX:
swapinfo (3M)
# /usr/sbin/swapinfo
             Kb      Kb      Kb   PCT  START/      Kb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev      524288  138260  386028   26%       0       -    1  /dev/vg00/lvol2
reserve       -   78472  -78472
memory   195132  191668    3464   98%
Calculate Disk Requirements
In addition to the swap space you just calculated, you need disk space for the following items:
Although you do not need to remove any installation files after completing the installation procedure, you can do so if you are short on disk space. The installation program will tell you how much space can be saved, and asks if you want to delete the installation files.
Determine the Number of Global Buffers
Caché supports the following maximum values for the number of global buffers:
Note:
Even if your configuration is using only 2-KB databases, some of the Caché databases, namely CACHESYS and CACHETEMP, are 8-KB. The system always sets aside a minimum of 200 8-KB buffers for these, regardless of what you specify. In general, increase this value, especially if you are making heavy use of Caché SQL.
Set your values to less than the maximum number of buffers listed above.
Determine the Number of Routine Buffers
Caché supports the following maximum value for the number of routine buffers:
65,535
Set your values to less than the maximum number of buffers listed above.
Determine the Maximum Number of Users
The maximum users allowed by Caché is the lowest of the following values:
Determine the Maximum Database Size
The ulimit parameter in UNIX determines the maximum file size available to a process. For the Caché Manager group, the value of ulimit should either be unlimited or as large as the largest database the user may have.
Configure UNIX Kernel Parameters
The following sections describe issues related to tuning and performance on various UNIX platforms:
Set Values for Tunable UNIX Parameters
Caché uses a configurable number of semaphores, in sets whose size you define. The parameters SEMMNI, SEMMNS, and SEMMSL reflect the number of semaphores per set and the total number of semaphores Caché uses. The UNIX/Linux parameters that govern shared memory allocation are SHMMAX, SHMMNI, SHMSEG, and SHMALL. Caché uses shared memory and allocates one segment of shared memory; the size of this segment depends on the area set aside for global buffers and routine buffers. It uses the following formula to determine the segment’s minimum size:
                       space required for routine buffers
                     +  space required for global buffers
                     +                               4 MB
                    _____________________________________
                     =         Shared memory segment size
If you are distributing your data across multiple computers, Caché allocates a second segment; by default, there is no memory allocated for the second segment. (If you plan to use distributed data, contact your VAR or InterSystems support for configuration guidelines.) You can alter NBUF and NHBUF according to other system requirements. Because Caché does all its own disk buffering, you should keep NBUF and NHBUF small. The following table lists the most common names of the UNIX parameters that you may need to change, the value InterSystems recommends for each parameter, and a brief description of each. Verify that your parameter values are set to at least the minimum value. Certain parameters may not be implemented on all platforms or may be referred to differently. Refer to platform-specific tuning notes for more information.
Tunable UNIX Parameters
Kernel Parameter Minimum Recommended Value Definition
CDLIMIT Number of bytes in largest virtual volume Maximum size of a file.
MSGMAX 2 Maximum message size, in bytes.
MSGMNI Number of Caché instances x 2; each Caché instance uses two message queues Minimum number of uniquely identifiable message queues that may exist simultaneously.
NOFILES 35 Number of open files per process.
SEMMNI Product of SEMMNI and SEMMSL must be greater than the # of user processes + 4 Number of semaphore identifiers in the kernel; this is the number of unique semaphore sets that can be active at any one time.
SEMMNS 128 or ... Total number of semaphores in the system. User processes include jobbed processes and all other semaphores required by other software.
Number of processes expected to run. If the process table might expand, use a larger number to provide for expansion.
SEMMSL See SEMMNI Maximum number of semaphores per identifier list.
SHMALL 60 KB or ... Maximum total shared memory system-wide. Units should be in KB. 1000 represents the MCOMMON shared region.
1000 + total global buffer space+ total routine buffer space *
SHMMNI 3 Maximum number of shared memory identifiers system-wide.
SHMSEG 3 Number of attached shared memory segments per process.
SHMMAX 60 KB or ... Maximum shared memory segment size in KB.
1000 + total global buffer space+ total routine buffer space
* This is the minimum value for SHMALL required for Caché UNIX. You must also take into account any other applications that use shared memory. If you are unsure of other shared memory use, calculate SHMALL as SHMSEG multiplied by SHMMAX, in pages; this larger value will suffice in all cases.
Important:
Enough swap space must be created to support the memory allocated, unless the operating system documentation explicitly state otherwise. On certain operating systems (Solaris, for example) Caché creates “locked shared memory segments”, which are not pageable. However these memory segments still need swap space unless the operating system documentation states otherwise.
Platform Configuration Issues
The following sections contain configuration issues for some individual platforms. For more information, consult the system documentation for your platform.
Tru64 UNIX Platform Notes
For the Tru64 UNIX supported releases, set the following kernel parameters to the size of all the shared memory to allocate (including space allocated for the buffer pool, routine buffers, and other data structures):
The following parameters are not implemented:
SMMAX and SMSEG are referred to as SHMMAX and SHMSEG, respectively.
The sysconfig -q subsystem [attribute] command displays the current values for the attributes of the specified subsystem, or the particular attribute specified. See the Tru64 UNIX documentation entry for sysconfig(8) for more information.
For example:
>sysconfig -q ipc
msg_max = 8192
msg_mnb = 16384
msg_mni = 64
msg_tql = 40
shm_max = 268435456
shm_min = 1
shm_mni = 128
shm_seg = 32
sem_mni = 16
sem_msl = 1000
sem_opm = 10
sem_ume = 10
sem_vmx = 32767
sem_aem = 16384
max_kernel_ports = 56864
ssm_threshold = 8388608
ssm_enable_core_dump = 1
shm_allocate_striped = 1
shm_enable_core_dump = 1
To permanently modify the value of an attribute, use the sysconfigdb -a -f stanza_file subsystem command to specify the stanza-formatted file that contains the subsystem, the attribute, and the new permanent attribute value. The subsystem argument specifies the subsystem whose attribute you want to modify.
The following is an example of a stanza-formatted file that changes the permanent values of the ipc subsystem attributes shm_max and sem_msl:
ipc:
  shm_max = 268435456
  sem_msl = 1000
See stanza(4) and sysconfigdb(8) for information about stanza-formatted files.
To use the new permanent value, restart the system or, if the attribute can be tuned at run time, use the sysconfig -r command to change the current value (see Section 3.6.4 of the Tru64 UNIX System Configuration and Tuning guide).
SUN Solaris Platform Notes
The following are notes to keep in mind when configuring and tuning Caché on the SUN Solaris version 8 and 9 platform:
  1. The following parameters are not implemented:CDLIMIT, NOCALL, and SHMALL.
  2. The NOFILES parameter is unlimited.
  3. Edit the /etc/system file and add the following variables to increase shared memory segments:
    set shmsys:shminfo_shmmax=value
    set shmsys:shminfo_shmmin=value
    set shmsys:shminfo_shmmni=value
    set shmsys:shminfo_shmseg=value
    set semsys:seminfo_semmap=value
    set semsys:seminfo_semmni=value
    set semsys:seminfo_semmns=value
    set semsys:seminfo_semmsl=value
    set semsys:seminfo_semmnu=value
    set semsys:seminfo_semume=value
    
    Shared Memory Variables
    Variable Description
    shmsys:shminfo_shmmax Maximum shared memory segment size
    shmsys:shminfo_shmmin Minimum shared memory segment size
    shmsys:shminfo_shmmni Number of shared memory identifiers
    shmsys:shminfo_shmseg Number of segments, per process
    semsys:seminfo_semmap Number of entries in the semaphore map
    semsys:seminfo_semmni Number of semaphore identifiers
    semsys:seminfo_semmns Number of semaphores in the system
    semsys:seminfo_semmsl Maximum number of semaphores, per ID
    semsys:seminfo_semmnu Number of processes using the undo facility
    semsys:seminfo_semume Maximum number of undo structures per process
  4. Verify the shared memory value changes:
    # grep shmsys /etc/system
  5. Restart the system:
    # init 6
The following sample shared memory values accommodate a system with a large amount of memory (128 MB) that is running a large database application:
set shmsys:shminfo_shmmax=268435456
set shmsys:shminfo_shmmin=200
set shmsys:shminfo_shmmni=200
set shmsys:shminfo_shmseg=200
set semsys:seminfo_semmap=250
set semsys:seminfo_semmni=500
set semsys:seminfo_semmns=500
set semsys:seminfo_semmsl=500
set semsys:seminfo_semmnu=500
set semsys:seminfo_semume=100
AIX Platform Notes
This topic include the following categories of note:
AIX Default Parameters
The default settings of several AIX parameters can adversely affect performance. The settings and recommendations are detailed below for the following:
I/O Pacing Parameters
The default values for the disk-I/O pacing high-water and low-water marks (maxpout and minpout parameters) may cause severe performance problems on Caché production systems. These default values may significantly hinder Caché Write daemon performance by inappropriately putting the Write daemon to sleep causing prolonged Write daemon cycles.
If you are using HACMP clusters, I/O pacing is automatically enabled. If your system is not part of an HACMP cluster, set both the high- (maxpout) and low- (minpout) water marks to 0 (zero) to disable I/O pacing.
View and change the current settings for the I/O pacing high-water and low-water marks by issuing the smitty chgsys command.
InterSystems currently recommends the following IBM calculation for determining the appropriate high-water mark:
high-water mark = (4 * n) + 1
Where n = the maximum number of spindles any one file (database, journal, or WIJ) spans across. Set the low-water mark to 50%-75% of the high-water mark.
For example, a CACHE.DAT database file is stored on a storage array, and the LUN (or file system) where it resides consists of 16 spindles/drives. Calculate:
High-water mark = (4 * 16) + 1 = 65

Low-water mark = between (.50 * 65) and (.75 * 65) = between 33 and 49
For more details, see the “Using Disk-I/O Pacing” section of the AIX Performance Management Guide at the following IBM Web page:
http://publib16.boulder.ibm.com/pseries/en_US/aixbman/prftungd/diskperf6.htm
Important:
InterSystems recommends the current IBM calculation for determining the appropriate high-water mark; verify the preceding calculation with IBM, as it may change.
If you have any questions about the impact to your system, contact the InterSystems WRC Performance Team or your AIX supplier before making any changes. These recommendations are independent of Caché versions and apply to both JFS and Enhanced JFS (JFS2) file systems.
File System Mount Option
For optimal performance, place all CACHE.DAT, CACHE.WIJ, and Caché journal files on a file system with release-behind-when-reading and release-behind-when-writing capabilities by using the rbrw mount option.
For more information, see the “mount Command” page of the AIX Commands Reference, Volume 3, i - m at:
http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/cmds/aixcmds3/mount.htm
Memory Management Parameters
The number of file systems and the amount of activity on them can limit the number of memory structures available to JFS or JFS2, and delay I/O operations waiting for those memory structures.
To monitor these metrics, issue a vmstat -vs command, wait two minutes, and issue another vmstat -vs command. The output looks similar to the following:
# vmstat -vs
              1310720 memory pages
              1217707 lruable pages
               144217 free pages
                    1 memory pools
               106158 pinned pages
                 80.0 maxpin percentage
                 20.0 minperm percentage
                 80.0 maxperm percentage
                 62.8 numperm percentage
               764830 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 32.1 numclient percentage
                 80.0 maxclient percentage
               392036 client pages
                    0 remote pageouts scheduled
                    0 pending disk I/Os blocked with no pbuf
                 5060 paging space I/Os blocked with no psbuf
              5512714 filesystem I/Os blocked with no fsbuf
               194775 client filesystem I/Os blocked with no fsbuf
                    0 external pager filesystem I/Os blocked with no fsbuf
If you see an increase in the following parameters, increase the values for better Caché performance:
When increasing these parameters from the default values:
  1. Increase the current value by 50%.
  2. Check the vmstat output.
  3. Run vmstat twice, two minutes apart.
  4. If the field is still increasing, increase again by the same amount; continue this step until the field stops increasing between vmstat reports.
Important:
Change both the current and the reboot values, and check the vmstat output regularly because I/O patterns may change over time (hours, days, or weeks).
See the following IBM Web pages for more detailed information:
AIX Tunable Parameters
None of the following listed parameters requires tuning because each is dynamically adjusted as needed by the kernel. See the appropriate AIX operating system documentation for more information.
The following table lists the tunable parameters for the IBM pSeries AIX 5.2 operating system.
AIX Interprocess Communication Tunable Parameters
Parameter Purpose Dynamic Values
msgmax Specifies maximum message size. Maximum value of 4 MB
msgmnb Specifies maximum number of bytes on queue. Maximum value of 4 MB
msgmni Specifies maximum number of message queue IDs. Maximum value of 4096
msgmnm Specifies maximum number of messages per queue. Maximum value of 524288
semaem Specifies maximum value for adjustment on exit. Maximum value of 16384
semmni Specifies maximum number of semaphore IDs. Maximum value of 4096
semmsl Specifies maximum number of semaphores per ID. Maximum value of 65535
semopm Specifies maximum number of operations per semop() call. Maximum value of 1024
semume Specifies maximum number of undo entries per process. Maximum value of 1024
semvmx Specifies maximum value of a semaphore. Maximum value of 32767
shmmax Specifies maximum shared memory segment size. Maximum value of 256 MB for 32-bit processes and 0x80000000u for 64-bit
shmmin Specifies minimum shared-memory-segment size. Minimum value of 1
shmmni Specifies maximum number of shared memory IDs. Maximum value of 4096
HP-UX Platform Notes
For HP-UX release 11i the CDLIMIT and NOFILES parameters are not implemented. In some versions, SEMMSL is hard-coded to 500. NCALL is referred to as NCALLOUT.
Use the HP System V IPC Shared-Memory Subsystem to update parameters. See the HP System V Inter-Process Communication Mechanisms online documentation page for additional information. To change a value, perform the following steps:
  1. Enter the /usr/sbin/sam command to start the System Administration Manager (SAM) program.
  2. Double-click the Kernel Configuration icon.
  3. Double-click the Configurable Parameters icon.
  4. Double-click the parameter you want to change and enter the new value in the Formula/Value field.
  5. Click OK.
  6. Repeat these steps for all of the kernel configuration parameters that you want to change.
  7. When you are finished setting all of the kernel configuration parameters, select Process New Kernel from the Action menu.
The HP-UX operating system automatically reboots after you change the values for the kernel configuration parameters.
Red Hat Linux Platform Notes
The default shared memory limit (shmmax) on Linux platforms is 32 MB. This value is too small for Caché, but it can be changed in the proc file system without a restart.
If the machine is being used only for Caché, InterSystems recommends setting the shared memory to approximately half the total memory.
For example, to allow 128 MB, type the following command:
$ echo 134217728 >/proc/sys/kernel/shmmax
You can put this command into a script run at startup.
Alternatively, you can use sysctl(8), if available, to control this parameter. Look for a file called /etc/sysctl.conf and add a line similar to the following:
kernel.shmmax = 134217728
This file is usually processed at startup, but sysctl can also be called explicitly later.
Important:
The msgmni parameter may also be set too low if you are running more than one instance of Caché on a machine. As stated in the Tunable UNIX Parameters table, set this value to two times the number of instances of Caché that run simultaneously on your system.
Other parameters are sufficiently sized for a Caché application. To view the values of other parameters, look in the files /usr/src/linux/include/asm-xxx/shmparam.h and /usr/src/linux/include/linux/sem.h.
For more information, reference the Managing Kernel Resources chapter of the Red Hat Database: Administrator and User's Guide.
SuSE Linux Platform Notes
The default shared memory limits (shhmax and shmall) on SuSE Linux platforms are too small for Caché, and can be changed in the proc file system without a restart.
If the machine is being used only for Caché, InterSystems recommends setting the shared memory to approximately half the total memory.
For example, to allow 512 MB, type the following commands:
#sets shmall and shmmax shared memory
echo 536870912 >/proc/sys/kernel/shmall     #Sets shmall to 512 MB
echo 536870912 >/proc/sys/kernel/shmmax     #Sets shmmax to 512 MB
You can also put these commands into a script run at startup.
Also change the settings for the system memory user limits by modifying a file called /etc/profile. Add lines similar to the following:
 
#sets user limits (ulimit) for system memory resources
ulimit -v 512000     #set virtual (swap) memory to 512 MB 
ulimit -m 512000     #set physical memory to 512 MB
 
In this same file, you can permanently change the values for the PATH and CLASSPATH parameters by adding lines similar to the following:
#sets env values PATH and CLASSPATH
export PATH=$PATH:/usr/cache/bin:/path/to/j2sdk/bin:/.
export CLASSPATH=
      $CLASSPATH:/cache/dev/java/lib/CacheDB.jar:/path/to/otherjar/file:/.