For optimal Caché performance, you need to calculate proper values for
certain Caché system parameters. These values allow you to determine whether
you need to adjust certain system level parameters. The values you choose should minimize
swapping and paging that require disk accesses, and thus improve system performance.
Review this section carefully and calculate the proper values for both your
operating system and Caché before proceeding. Use the tables provided here
to record the current and calculated values for your system level parameters. You
can then refer to these tables when you install Caché. After your system is
running, you may need to adjust these values to gain optimal performance.
If you are not already familiar with the memory organization at your operating
system level, consult the appropriate system documentation.
Determine Memory and Disk Requirements
This section outlines the basic system requirements for most systems. Because
these requirements vary by platform, consult your platform documentation for additional
information.
Calculate Memory Requirements
Use the breakdown of memory usage shown in the following table to calculate
the memory your system needs for Caché. It is assumed that you are using the
Caché Configuration Manager (from a Windows client) to change parameters.
Memory Requirements
Components |
Memory Requirements |
Operating system |
1800 KB (operating
system dependent) |
Caché |
842 KB |
Global database cache |
2.2
KB per buffer if using a 2-KB block size, or 8 KB per buffer if using an 8-KB block
size. |
Routine cache |
32 KB |
User overhead |
1024 KB per
process. The user partition size, which comprises some of user overhead, is tunable
via the Caché Configuration Manager. The partition grows dynamically up to
16 MB. |
Network (if present) |
300 KB
per port for each network system process (DMNNET, DCP, and RECEIVE). Caché
ports have two DMNNET system processes per port. In addition, there is a network shared
memory requirement, which depends on the number of ports and the number of remote
hosts configured. For a basic system, this requirement is about 304 KB. |
By default, Caché allocates 7 MB for its swap space for shared memory,
global buffers, and routine buffers. If you are installing Caché for the first
time, this amount should be sufficient. If you plan to run large applications or support
large numbers of users, tune the system according to the following formula:
(number of routine buffers)*32 KB
+ (number of global buffers)*8 KB
+ 4 MB
___________________________________
= Shared memory needed
The amount of swap space available on your system should never be less than
the amount of real memory plus 256 KB.
With this minimum in mind, InterSystems recommends the following value as the
minimum amount of swap space needed for Caché:
((# of processes + 4)† * (1024 KB)‡
+ total global buffer space
+ total routine buffer space
_____________________________________
= Minimum swap space
† You add 4 to the # of processes for the Caché Control Process,
the Write daemon, the Garbage Collector, and the Journal daemon. You should also add
1 for each slave Write daemon. The # of processes must include all user and jobbed
processes which might run concurrently. If you are running networking, add 1 for the
RECEIVE system process plus the number of DMNNET daemons you have running (2 per port).
If your system is a server, add the number of DCP processes that will be created when
all potential clients access the server.
‡ The 1024 KB number is approximate. It is based on the current size
of the Caché executable and grows with the partition size you allocate in the
Configuration Manager to each Caché process. On most systems, you will want
to provide only as much swap space as necessary. However, some systems require you
to provide swap space for the worst case. Under these conditions, you need to increase
this number to as high as 1.5 MB, depending on the partition size you specify.
Be sure to confirm that your UNIX system permits the amount of swap space you
require. For specific information about swap space on your system, consult your UNIX
operating system manual.
To calculate swap space for the Solaris 8 and 9 platforms:
>swap –l
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s0 136,0 16 526304 526304
/dev/dsk/c0t2d0s1 136,1 16 2101184 2101184
To display swap space for Tru64 UNIX:
Information for each swap partition is displayed similar to the following example:
Swap partition /dev/disk/dsk1b (default swap):
Allocated space: 16384 pages (128MB)
In-use space: 10452 pages ( 63%)
Free space: 5932 pages ( 36%)
Swap partition /dev/disk/dsk4c:
Allocated space: 128178 pages (1001MB)
In-use space: 10242 pages ( 7%)
Free space: 117936 pages ( 92%)
Total swap allocation:
Allocated space: 144562 pages (1.10GB)
Reserved space: 34253 pages ( 23%)
In-use space: 20694 pages ( 14%)
Available space: 110309 pages ( 76%)
To display swap space for AIX:
lsps –a
Page Space Physical Volume Volume Group Size %Used
Active Auto Type
hd6 hdisk2 rootvg 512MB 72
yes yes lv
To display swap space for HP-UX:
swapinfo (3M)
# /usr/sbin/swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 138260 386028 26% 0 - 1 /dev/vg00/lvol2
reserve - 78472 -78472
memory 195132 191668 3464 98%
Calculate Disk Requirements
In addition to the swap space you just calculated, you need disk space for the
following items:
-
-
3 MB for the Caché Server Pages (CSP).
-
3.5 MB for Caché ODBC support.
-
2.5 MB for the Caché manager sources.
-
6.6 MB for the Caché engine link libraries.
-
-
Space for your Caché application database.
-
Approximately 12.5% of the buffer pool size for the initial size of
the write image journal file. If your disk does not have enough space for the write
image journal file, when you start Caché it displays a message indicating that
the system did not start.
-
Desired space for the journal file.
Although you do not need to remove any installation files after completing the
installation procedure, you can do so if you are short on disk space. The installation
program will tell you how much space can be saved, and asks if you want to delete
the installation files.
Determine the Number of Global Buffers
Caché supports the following maximum values for the number of global
buffers:
-
For 32-bit platforms, any combination of 2-KB and 8-KB buffers that
are:
-
-
Less than 2 GB for other 32-bit platforms
The 2-GB value is the total address space the operation system allocates for
the process data, which includes not only shared memory, but other Caché and
operating system data as well. Therefore, it represents an upper limit that is not
achievable in practice.
-
For 64-bit platforms: Tru64 UNIX, Solaris/UltraSPARC, HP-UX/HP-PA64,
and AIX/PowerPC64
The number of global buffers is limited only by the
operating system and the available memory.
Note:
Even if your configuration is using only 2-KB databases, some of the Caché
databases, namely
CACHESYS and
CACHETEMP,
are 8-KB. The system always sets aside a minimum of 200 8-KB buffers for these, regardless
of what you specify. In general, increase this value, especially if you are making
heavy use of Caché SQL.
Set your values to less than the maximum number of buffers listed above.
Determine the Number of Routine Buffers
Caché supports the following maximum value for the number of routine
buffers:
Set your values to less than the maximum number of buffers listed above.
Determine the Maximum Number of Users
The maximum users allowed by Caché is the
lowest of
the following values:
Determine the Maximum Database Size
The
ulimit parameter in UNIX determines the maximum file
size available to a process. For the Caché Manager group, the value of
ulimit should
either be unlimited or as large as the largest database the user may have.
Configure UNIX Kernel Parameters
The following sections describe issues related to tuning and performance on
various UNIX platforms:
Set Values for Tunable UNIX Parameters
Caché uses a configurable number of semaphores, in sets whose size you
define. The parameters
SEMMNI,
SEMMNS, and
SEMMSL reflect
the number of semaphores per set and the total number of semaphores Caché uses.
The UNIX/Linux parameters that govern shared memory allocation are
SHMMAX,
SHMMNI,
SHMSEG,
and
SHMALL. Caché uses shared memory and allocates one segment
of shared memory; the size of this segment depends on the area set aside for global
buffers and routine buffers. It uses the following formula to determine the segment’s
minimum size:
space required for routine buffers
+ space required for global buffers
+ 4 MB
_____________________________________
= Shared memory segment size
If you are distributing your data across multiple computers, Caché allocates
a second segment; by default, there is no memory allocated for the second segment.
(If you plan to use distributed data, contact your VAR or InterSystems support for
configuration guidelines.) You can alter
NBUF and
NHBUF according
to other system requirements. Because Caché does all its own disk buffering,
you should keep
NBUF and
NHBUF small. The following
table lists the most common names of the UNIX parameters that you may need to change,
the value InterSystems recommends for each parameter, and a brief description of each.
Verify that your parameter values are set to at least the minimum value. Certain parameters
may not be implemented on all platforms or may be referred to differently. Refer to
platform-specific tuning notes for more information.
Tunable UNIX Parameters
Kernel Parameter |
Minimum Recommended
Value |
Definition |
CDLIMIT |
Number of bytes in
largest virtual volume |
Maximum size of a file. |
MSGMAX |
2 |
Maximum
message size, in bytes. |
MSGMNI |
Number of Caché
instances x 2; each Caché instance uses two message queues |
Minimum
number of uniquely identifiable message queues that may exist simultaneously. |
NOFILES |
35 |
Number
of open files per process. |
SEMMNI |
Product of SEMMNI and
SEMMSL must be greater than the # of user processes + 4 |
Number
of semaphore identifiers in the kernel; this is the number of unique semaphore sets
that can be active at any one time. |
SEMMNS |
128
or ... |
Total number of semaphores
in the system. User processes include jobbed processes and all other semaphores required
by other software. |
Number of processes expected to run. If the process table
might expand, use a larger number to provide for expansion. |
SEMMSL |
See SEMMNI |
Maximum number of semaphores per identifier list. |
SHMALL |
60
KB or ... |
Maximum total
shared memory system-wide. Units should be in KB. 1000 represents the MCOMMON shared
region. |
1000 + total global buffer space+ total routine buffer
space * |
SHMMNI |
3 |
Maximum
number of shared memory identifiers system-wide. |
SHMSEG |
3 |
Number
of attached shared memory segments per process. |
SHMMAX |
60
KB or ... |
Maximum shared
memory segment size in KB. |
1000 + total global buffer space+ total routine buffer
space |
* This is the minimum value for
SHMALL required for Caché
UNIX. You must also take into account any other applications that use shared memory.
If you are unsure of other shared memory use, calculate
SHMALL as
SHMSEG multiplied
by
SHMMAX, in pages; this larger value will suffice in all cases.
Important:
Enough swap space must be created to support the memory allocated, unless the
operating system documentation explicitly state otherwise. On certain operating systems
(Solaris, for example) Caché creates
locked shared memory segments,
which are not pageable. However these memory segments still need swap space unless
the operating system documentation states otherwise.
Platform Configuration Issues
The following sections contain configuration issues for some individual platforms.
For more information, consult the system documentation for your platform.
Tru64 UNIX Platform Notes
For the Tru64 UNIX supported releases, set the following kernel parameters to
the size of all the shared memory to allocate (including space allocated for the buffer
pool, routine buffers, and other data structures):
The following parameters are not implemented:
>sysconfig -q ipc
msg_max = 8192
msg_mnb = 16384
msg_mni = 64
msg_tql = 40
shm_max = 268435456
shm_min = 1
shm_mni = 128
shm_seg = 32
sem_mni = 16
sem_msl = 1000
sem_opm = 10
sem_ume = 10
sem_vmx = 32767
sem_aem = 16384
max_kernel_ports = 56864
ssm_threshold = 8388608
ssm_enable_core_dump = 1
shm_allocate_striped = 1
shm_enable_core_dump = 1
The following is an example of a stanza-formatted file that changes the permanent
values of the ipc subsystem attributes
shm_max and
sem_msl:
ipc:
shm_max = 268435456
sem_msl = 1000
To use the new permanent value, restart the system or, if the attribute can
be tuned at run time, use the
sysconfig -r command to change the
current value (see
Section
3.6.4 of the Tru64 UNIX System Configuration and Tuning guide).
SUN Solaris Platform Notes
The following are notes to keep in mind when configuring and tuning Caché
on the SUN Solaris version 8 and 9 platform:
-
-
The
NOFILES parameter is unlimited.
-
Edit the
/etc/system file and add the following
variables to increase shared memory segments:
set shmsys:shminfo_shmmax=value
set shmsys:shminfo_shmmin=value
set shmsys:shminfo_shmmni=value
set shmsys:shminfo_shmseg=value
set semsys:seminfo_semmap=value
set semsys:seminfo_semmni=value
set semsys:seminfo_semmns=value
set semsys:seminfo_semmsl=value
set semsys:seminfo_semmnu=value
set semsys:seminfo_semume=value
Shared Memory Variables
Variable |
Description |
shmsys:shminfo_shmmax |
Maximum shared memory segment size |
shmsys:shminfo_shmmin |
Minimum shared memory segment size |
shmsys:shminfo_shmmni |
Number of shared memory identifiers |
shmsys:shminfo_shmseg |
Number of segments, per process |
semsys:seminfo_semmap |
Number of entries in the semaphore
map |
semsys:seminfo_semmni |
Number of semaphore identifiers |
semsys:seminfo_semmns |
Number of semaphores in the system |
semsys:seminfo_semmsl |
Maximum number of semaphores, per
ID |
semsys:seminfo_semmnu |
Number of processes using the undo
facility |
semsys:seminfo_semume |
Maximum number of undo structures
per process |
-
Verify the shared memory value changes:
# grep shmsys /etc/system
-
The following sample shared memory values accommodate a system with a large
amount of memory (128 MB) that is running a large database application:
set shmsys:shminfo_shmmax=268435456
set shmsys:shminfo_shmmin=200
set shmsys:shminfo_shmmni=200
set shmsys:shminfo_shmseg=200
set semsys:seminfo_semmap=250
set semsys:seminfo_semmni=500
set semsys:seminfo_semmns=500
set semsys:seminfo_semmsl=500
set semsys:seminfo_semmnu=500
set semsys:seminfo_semume=100
This topic include the following categories of note:
The default settings of several AIX parameters can adversely affect performance.
The settings and recommendations are detailed below for the following:
I/O Pacing Parameters
The default values for the disk-I/O pacing high-water and low-water marks (
maxpout and
minpout parameters)
may cause severe performance problems on Caché production systems. These default
values may significantly hinder Caché Write daemon performance by inappropriately
putting the Write daemon to sleep causing prolonged Write daemon cycles.
If you are using HACMP clusters, I/O pacing is automatically enabled. If your
system is not part of an HACMP cluster, set both the high- (
maxpout)
and low- (
minpout) water marks to 0 (zero) to disable I/O pacing.
View and change the current settings for the I/O pacing high-water and low-water
marks by issuing the
smitty chgsys command.
InterSystems currently recommends the following IBM calculation for determining
the appropriate high-water mark:
high-water mark = (4 * n) + 1
Where
n = the maximum number of spindles any one file (database,
journal, or WIJ) spans across. Set the low-water mark to 50%-75% of the high-water
mark.
For example, a
CACHE.DAT database file is stored on a storage
array, and the LUN (or file system) where it resides consists of 16 spindles/drives.
Calculate:
High-water mark = (4 * 16) + 1 = 65
Low-water mark = between (.50 * 65) and (.75 * 65) = between 33 and 49
Important:
InterSystems recommends the current IBM calculation for determining the appropriate
high-water mark; verify the preceding calculation with IBM, as it may change.
If you have any questions about the impact to your system, contact the
InterSystems WRC Performance
Team or your AIX supplier before making any changes. These recommendations
are independent of Caché versions and apply to both JFS and Enhanced JFS (JFS2)
file systems.
File System Mount Option
For optimal performance, place all
CACHE.DAT,
CACHE.WIJ,
and Caché journal files on a file system with release-behind-when-reading and
release-behind-when-writing capabilities by using the
rbrw mount
option.
Memory Management Parameters
The number of file systems and the amount of activity on them can limit the
number of memory structures available to JFS or JFS2, and delay I/O operations waiting
for those memory structures.
To monitor these metrics, issue a
vmstat -vs command, wait
two minutes, and issue another
vmstat -vs command. The output looks
similar to the following:
# vmstat -vs
1310720 memory pages
1217707 lruable pages
144217 free pages
1 memory pools
106158 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
62.8 numperm percentage
764830 file pages
0.0 compressed percentage
0 compressed pages
32.1 numclient percentage
80.0 maxclient percentage
392036 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
5060 paging space I/Os blocked with no psbuf
5512714 filesystem I/Os blocked with no fsbuf
194775 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
If you see an increase in the following parameters, increase the values for
better Caché performance:
When increasing these parameters from the default values:
-
Increase the current value by 50%.
-
-
Run
vmstat twice, two minutes apart.
-
If the field is still increasing, increase again by the same amount;
continue this step until the field stops increasing between
vmstat reports.
Important:
Change both the current and the reboot values, and check the
vmstat output
regularly because I/O patterns may change over time (hours, days, or weeks).
See the following IBM Web pages for more detailed information:
None of the following listed parameters requires tuning because each is dynamically
adjusted as needed by the kernel. See the appropriate
AIX
operating system documentation for more information.
The following table lists the tunable parameters for the IBM pSeries AIX 5.2
operating system.
AIX Interprocess Communication Tunable Parameters
Parameter |
Purpose |
Dynamic Values |
msgmax |
Specifies maximum message size. |
Maximum
value of 4 MB |
msgmnb |
Specifies maximum number of bytes on queue. |
Maximum value of 4 MB |
msgmni |
Specifies maximum number of message queue IDs. |
Maximum value of 4096 |
msgmnm |
Specifies maximum number of messages per queue. |
Maximum value of 524288 |
semaem |
Specifies maximum value for adjustment on exit. |
Maximum value of 16384 |
semmni |
Specifies maximum number of semaphore IDs. |
Maximum value of 4096 |
semmsl |
Specifies maximum number of semaphores per ID. |
Maximum value of 65535 |
semopm |
Specifies maximum number of operations per semop()
call. |
Maximum value of 1024 |
semume |
Specifies maximum number of undo entries per process. |
Maximum value of 1024 |
semvmx |
Specifies maximum value of a semaphore. |
Maximum
value of 32767 |
shmmax |
Specifies maximum shared memory segment size. |
Maximum value of 256 MB for 32-bit processes and 0x80000000u for 64-bit |
shmmin |
Specifies minimum shared-memory-segment size. |
Minimum value of 1 |
shmmni |
Specifies maximum number of shared memory IDs. |
Maximum value of 4096 |
-
Enter the
/usr/sbin/sam command to start the System
Administration Manager (SAM) program.
-
-
-
Double-click the parameter you want to change and enter the new value
in the
Formula/Value field.
-
-
Repeat these steps for all of the kernel configuration parameters
that you want to change.
-
When you are finished setting all of the kernel configuration parameters,
select
from the
menu.
The HP-UX operating system automatically reboots after you change the values
for the kernel configuration parameters.
Red Hat Linux Platform Notes
The default shared memory limit (
shmmax) on Linux platforms
is 32 MB. This value is too small for Caché, but it can be changed in the
proc file
system without a restart.
If the machine is being used only for Caché, InterSystems recommends
setting the shared memory to approximately half the total memory.
For example, to allow 128 MB, type the following command:
$ echo 134217728 >/proc/sys/kernel/shmmax
You can put this command into a script run at startup.
Alternatively, you can use
sysctl(8), if available, to
control this parameter. Look for a file called
/etc/sysctl.conf and
add a line similar to the following:
kernel.shmmax = 134217728
This file is usually processed at startup, but
sysctl can
also be called explicitly later.
Important:
The
msgmni parameter may also be set too low if you are running
more than one instance of Caché on a machine. As stated in the
Tunable
UNIX Parameters table, set this value to two times the number of instances
of Caché that run simultaneously on your system.
SuSE Linux Platform Notes
The default shared memory limits (
shhmax and
shmall)
on SuSE Linux platforms are too small for Caché, and can be changed in the
proc file
system without a restart.
If the machine is being used only for Caché, InterSystems recommends
setting the shared memory to approximately half the total memory.
For example, to allow 512 MB, type the following commands:
#sets shmall and shmmax shared memory
echo 536870912 >/proc/sys/kernel/shmall #Sets shmall to 512 MB
echo 536870912 >/proc/sys/kernel/shmmax #Sets shmmax to 512 MB
You can also put these commands into a script run at startup.
Also change the settings for the system memory user limits by modifying a file
called
/etc/profile. Add lines similar to the following:
#sets user limits (ulimit) for system memory resources
ulimit -v 512000 #set virtual (swap) memory to 512 MB
ulimit -m 512000 #set physical memory to 512 MB
In this same file, you can permanently change the values for the
PATH and
CLASSPATH parameters
by adding lines similar to the following:
#sets env values PATH and CLASSPATH
export PATH=$PATH:/usr/cache/bin:/path/to/j2sdk/bin:/.
export CLASSPATH=
$CLASSPATH:/cache/dev/java/lib/CacheDB.jar:/path/to/otherjar/file:/.