Revision as of 17:12, 2 December 2015

To run Sybase ASE successfully on a Linux environment you need to configure a number of items. Below you will find a few of these, but please note that this is by far not an full overview. There could be more improvements to be made.

Memory – Huge Pages

The configuration of Huge Pages is needed to get an efficient translation from virtual memory pages to physical pages. When you configure for Huge Pages, the amount of memory assigned to Huge Pages is allocated and reserved for shared memory. Do not over configure to prevent swapping.

When ASE reports the below message in the errorlog, you need to configure for huge pages:

Could not allocate memory using Huge Pages. Allocated using regular pages. For better performance, reboot the server after configuring enough Huge Pages.

To get the values for huge pages first determine the maximum amount of memory you want to give to ASE. For instance, on a Linux server with 512Gb physical memory you can give 90% to ASE. 512 Gb x 90% = 460 Gb = 471040 Mb.

Then determine the size of a single huge page:

cat /proc/meminfo | grep Hugepagesize 

Sample output:

Hugepagesize:       2048 kB

In this case 1 huge page is 2 Mb, so the number of huge pages we need will be 471040 Mb / 2 Mb = 235520

Then set the correct value in /etc/sysctl.conf:

vm.nr_hugepages = <value> 

Next step is to calculate the number of huge pages in Kb and give the “sybase” user permission to grab the memory. Take the number of huge pages as calculated above. Then multiply by 2048. That wil give you the number of pages in Kb. Example: 235520 x 2048 = 482344960. Then have it stored in /etc/security/limits.conf like this

sybase soft memlock 482344960 
sybase hard memlock 482344960 

In this example, "sybase" is assumed to be the unix user running ASE.

Now reboot the Linux server to check if everything works as expected.

To see the configured number of huge pages at runtime:

cat /proc/sys/vm/nr_hugepages 

Notes:

• As per Sybase recommendation (CR 627114) “lock shared memory” must be switched off.
• When ASE uses huge pages, it will allocate memory to the nearest multiple of 256Mb so don’t configure the ASE exactly to the maximum number of huge pages, but leave a small amount unused (e.g. 300 Mb). In case a small increase in size is needed due to the change of a Sybase config you will not get into trouble.

Number of file descriptors

You may need to increase the number of file descriptors per process if you get this error:

The number of file descriptors per ASE process required to support the current configuration exceeds the limit 4096 by 3955.

Make the following changes to /etc/security/limits.conf

sybase soft nofile 16384 
sybase hard nofile 16384 

In this example, "sybase" is assumed to be the unix user running ASE.

Log off and log on again to pick up the new settings.

DISK I/O

fs.aio-max-nr

fs.aio-max-nr is the maximum number of asynchronous I/Os.

The recommended minimum value for fs.aio-max-nr is 1048576, but on a busy ASE with many engines you may need to have a higher number configured.

In ASE, set these values as a starting point.

• Set “disk i/o structures” to 1048576.
• Set “max async i/os per engine” to 8192.
• Keep "max async i/os per server" at it’s default value (2147483647).

When you have multiple engines configured within ASE then use this formula to calculate the value for fs.aio-max-nr

(disk i/o structures + max async i/os per engine) * max online engines

For example: (1048576 + 8192) * 32 engines = 33816576

Then set the value in /etc/sysctl.conf:

fs.aio-max-nr = 33816576 

To get the value at runtime

cat /proc/sys/fs/aio-max-nr 

Troubleshooting fs.aio-max-nr

When the below messages appear in the ASE errorlog, the configuration of fs.aio-max-nr is not correct:

kernel asynchronous I/O not initialized. The io_setup() system call returned -22.

or

kernel  kistartup: asynchronous I/O not available: No such file or directory

Either recalculate the vale or reduce the Sybase related configuration options (“disk i/o structures” and/or “max async i/os per engine”)

Troubleshooting dstartio

When this message appears in the errorlog

kernel  dstartio: I/O request repeatedly delayed; block number: <nr>, vdn: <nr>

then you need to increase "disk i/o structures" or "max async i/os per engine".

To determine if "disk i/o structures" or "max async i/os per engine" needs to be increased look at the output of sp_sysmon that was captured during the time that the message appeared. Snapshot of output:

 I/Os Delayed by 
   Disk I/O Structures               n/a           n/a           0      n/a 
   Server Config Limit               n/a           n/a           0      n/a 
   Engine Config Limit               n/a           n/a       24582      n/a 
   Operating System Limit            n/a           n/a           0      n/a 

When a non-zero value is reported, increase the corresponding Sybase parameter. Alternatively, you can run sp_monitorconfig “all” to see if the maximum number of “disk i/o structures” has reached (see column Max_Used).

Be aware that the Linux setting for fs.aio-max-nr may need to be increased as well after changing the Sybase configuration settings.

Queue depth

Per Linux volume used by ASE the queue depth must be configured. The default value is 128 but it must be set to 1024.

You can only set the required value at boottime and, depending on the Linux distribution, you need to do it in /etc/init.d/boot.local (Suse) or /etc/init.d/rc.local (Ubuntu) or /etc/rc.d/rc.local (RedHat)

For example:

echo 1024 >/sys/block/sdb/queue/nr_requests 

To retrieve the runtime value:

cat /sys/block/sd*/queue/nr_requests 

kernel.randomize_va_space

This may be Suse and ASE 15.0.3 specific, but it may apply to other Linux / ASE version combinations as well.

Due to a high I/O load you may see stacktraces referring to the cm_writedes module, see below snippet.

00:00000:00006:2012/11/11 01:18:41.45 kernel  pc: 0x00007ffff73a56a0 (null)+0x7ffff73a56a0()
00:00000:00006:2012/11/11 01:18:41.45 kernel  pc: 0x00000000018fd036 cm_writedes+0x606()

In the later versions of ASE 15.7 this should be fixed and an error message is printed. To prevent the issue, set the value for kernel.randomize_va_space to 0 in combination with a higher value of “cpu grace time” in ASE, like 1500. To see the runtime value

cat /proc/sys/kernel/randomize_va_space 

To see the boottime value, check /etc/sysctl.conf there must be line like kernel.randomize_va_space = 0

See also http://www.novell.com/support/kb/doc.php?id=7004855

After changing kernel.randomize_va_space do a reboot of the Linux server, as that seems to be necessary.

Prevent a firewall from dropping idle connections

For connections to ASE

When your Sybase server is behind a firewall there is always the risk that a connection from the client side (e.g. running isql) to ASE will be dropped when the firewall sees it as an idle session. This is regardless weather the session is really idle or that it is just a very long running query. To prevent it, put this in /etc/sysctl.conf

net.ipv4.tcp_keepalive_time = 300

and activate it

sysctl -p

For ssh sessions

Putty

Enable TCP keepalive. On the start screen of Putty click "Connection" and activate the check-box "Enable TCP keepalives".

ssh

Create a config file in your ssh directory (~/.ssh/config) with the following content

Host *
ServerAliveInterval 60
ServerAliveCountMax 1000
TCPKeepAlive=yes

Whenever you use ssh tell it to use the config file

ssh -F ~/.ssh/config <server>

Soft / Hard limits for memory allocation (Suse only)

By default, no more than 80 % of available memory (RAM and swap) can be claimed by a single process. This is the soft-virtual limit. To check the setting: cat /etc/sysconfig/ulimit and look for the value specified as SOFTVIRTUALLIMIT. Default is “80”.

Increase the setting as setting as needed, or set to “0” for unlimited.

Associated Sybase error message:

Attempt to attach shared memory (id 32768) at address (nil), with flags 0 failed