If you use the Direct Access Programming Library (DAPL) fabric when running your Message Passing Interface (MPI) application, and your application fails with an error message like this:
………………..
[0] MPI startup(): RLIMIT_MEMLOCK too small
[0] MPI startup(): RLIMIT_MEMLOCK too small
libibverbs: Warning: RLIMIT_MEMLOCK is 0 bytes.
………………..
The warning “RLIMIT_MEMLOCK too small” coming from the OpenFabrics Enterprise Distribution (OFED) driver, indicates that the permitted size of the locked memory in your system is too small. When running an MPI in symmetric mode, that warning may indicate the locked memory is insufficient in either the Intel ®Xeon® host or the Intel® Xeon Phi™ coprocessors. So you need to increase the maximum permitted memory in order to run your MPI program successfully.
For example, if I run the command below on the host, it will display user’s current limits on the host:
$ ulimit –a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 514546
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
As shown in the above example, the maximum size of locked memory in this system is only 64 K bytes, I need to increase it as suggested by the program. In this blog, I would like to share with you some Best Known Methods (BKMs) on how to set the locked memory size properly on the Intel® Xeon® host and the Intel® Xeon Phi™ coprocessor.
There are multiple methods available to set the locked memory size. In this blog, I discuss two of them.
1. Method 1: Changing Locked Memory Permanently
This method shows how to alter a configuration limit to allow a user to change the locked memory size.
1.1 Change locked memory size in the host:
Edit the file /etc/security/limits.conf
$ sudo vi /etc/security/limits.conf
Add the following line at the end of the file to set an unlimited locked memory for the user named “user1”. The second field value can be “hard” or “soft”: “hard” enforces the resource limit, and “soft” defines the default value within the maximum value specified by “hard”.
user1 hard memlock unlimited
user1 soft memlock unlimited
By adding this, user1 can raise the locked memory size without limit. To allow all users to set unlimited memlock, simply replace user1 with the wildcard *
* hard memlock unlimited
* soft memlock unlimited
Reboot the system. Login and verify that the locked memory is now set to unlimited:
$ ulimit –l
unlimited
1.2 Change locked memory size in the coprocessor:
If the locked memory problem actual happens in the coprocessor, then you need to increase the permitted size in the coprocessor. You can change the default setting using micctrl, a multi-purpose toolbox for the system administrator. Depending on the Intel® Manycore Platform Software Stack (Intel®MPSS) version that you installed, the way for setting the locked memory limit differs depending upon whether you are using a Yocto-based (Intel® MPSS 3.x) or pre-Yocto-based (Intel® MPSS 2.x) distribution.
1.2.1 Intel® MPSS 3.x and later:
For Intel® MPSS 3.x, you can use the standard process for setting the parameters in the configuration file limits.conf.
Create a new directory if it was not created previously:
$ sudo mkdir /var/mpss/mic0/etc/security
Copy the standard configuration limits.conf:
$ sudo /etc/security/limits.conf /var/mpss/mic0/etc/security/.
$ sudo vi /var/mpss/mic0/etc/security/limits.conf
Add the following line at the end of the limits.conf file to set an unlimited locked memory for the user named “user1”:
user1 hard memlock unlimited
user1 soft memlock unlimited
Restart the MPSS service (for RHEL 6.x and SUSE):
$ sudo service mpss restart
For RHEL 7.x, type the following command instead:
$ sudo systemctl restart mpss
Login to the coprocessor and verify the locked memory setting is now unlimited:
$ ssh mic0 ulimit -l
unlimited
1.2.2 Intel® MPSS 2.x:
For Intel® MPSS 2.x, you need to get the permission to run programs with security privileges:
$ sudo su
Create a new file: /opt/intel/mic/filesystem/base/etc/limits.sh and add the following line to that file:
ulimit -l unlimited
Change the permissions of this file:
# chmod 744 limits.sh
Edit the existing file /opt/intel/mic/filesystem/base/etc/rc.d/rc.sysinit, and then add this one line just above the line that reads: echo_info "Start runlevel 3 services":
[ -x /etc/limits.sh ] && . /etc/limits.sh
Edit the existing file /opt/intel/mic/filesystem/base.filelist, and append the following line to the end of the file:
file /etc/limits.sh etc/limits.sh 0744 0 0
Restart the MPSS service:
# service mpss restart
Login to the coprocessor as a user and verify the locked memory now set to unlimited:
$ ssh mic0 ulimit –l
unlimited
2. Method 2: Changing Locked Memory by Using Scripts
In this method, I create a shell script to change the locked memory size. Inside the script, I set the locked memory properly and then specify the program to be executed. Instead of running the MPI program directly, now I pass the scripts instead of the program. Inside the script, the shell sets the locked memory size accordingly and then runs the application.
The example below shows how I create two scripts, one for the host (hostscript.sh) and one for the coprocessor (micscript.sh). Inside each script, I use the command “ulimit” to set the locked memory to unlimited before launching the application, IMB-MPI1 in this case.
$ cat hostscript.sh
echo "Current max locked memory in host: "
ulimit -l
echo "Set max locked memory to unlimited in host"
ulimit -l unlimited
echo "New max locked memory: "
ulimit -l
# MPI application
/opt/intel/impi/5.0.1.035/bin64/IMB-MPI1 PingPong
$ cat micscript.sh
echo "Current max locked memory in coprocessor: "
ulimit -l
echo "Set max locked memory to unlimited in coprocessor"
ulimit -l unlimited
echo "New max locked memory: "
ulimit -l
# MPI application
/tmp/IMB-MPI1
Transfer the coprocessor script to the home directory on the coprocessor and the application:
$ scp micscript.sh mic0:/home/user1
$ scp /opt/intel/impi/5.0.1.035/mic/bin/IMB-MPI1 mic0:/tmp
Finally, run the MPI program by passing the scripts:
$ export I_MPI_MIC=enable
$ mpirun -host localhost -n 1 ./hostscript.sh : -host mic0 -n 1 ./home/user1/micscript.sh
In summary, the above methods can be used to change the locked memory size. Method 1 is preferred when one needs to reboot the system many times during the test since the change is permanent in the system. Method 2 is used when a user wants the change only occurs in the running session, thus all the default setting are reserved after the running session is done.