Tweaking the entropy pool for a better SOA performance

At my current assignment, I’ve gone to great lengths to improve the Oracle SOA Enterprise Manager Console performance. One of the actions I took, was purging about 700 gigabytes of data from the dehydration store. Although that solved lots of other problems, the EM still wasn’t very responsive. Loading composite details could take op to 20 minutes, it they ever loaded at all. Apparently, the database was ok, because there was no load at all there. The java processes used a lot of CPU and memory though.

I found out recently that the SOA suite uses a Linux feature called the kernel entropy pool, or just entropy pool. This is a small pool of random data (generally about 4kb), generated by events on the host system, like mouse movements and IDE timings. the /dev/random module continuously feeds this pool with pseudo-random data. But when not enough seed-data is available, because more entropy data is requested than is generated, the pool gets depleted. When this happens, processes depending on the entropy pool will stall, slowing down your system.

Oracle SOA suite and it’s Java processes apparently use a lot of entropy data. They are able to quickly drain the entropy pool, at which point they will stall until new entropy data comes available. Which may take a long time!

Fortunately, there is a relatively simple solution. It’s a linux package called haveged:

haveged is a userspace entropy daemon which is not dependent upon the standard mechanisms for harvesting randomness for the system entropy pool. This is important in systems with high entropy needs or limited user interaction (e.g. headless servers).

haveged uses HAVEGE (HArdware Volatile Entropy Gathering and Expansion) to maintain a 1M pool of random bytes used to fill /dev/random whenever the supply of random bits in dev/random falls below the low water mark of the device. More information about HAVAGE is available at http://www.irisa.fr/caps/projects/hipsor/

Usually you can install it just by typing yum install haveged or apt-get install haveged. If the package is not available in the repositories, you can always build it yourself. Coming up is a little script that does most of the work for you. It also adds the haveged daemon to the startup file and appends the library path environment settings to your profile. Please review and test this script thoroughly before applying it to your production environment!

wget http://www.issihosts.com/haveged/haveged-1.9.1.tar.gz
tar zxvf haveged-1.9.1.tar.gz
cd haveged-1.9.1
./configure
make
make install
cp /etc/rc.local /etc/rc.local.org
echo '# 2014-03-14 Autostart haveged' >> /etc/rc.local
echo '/usr/local/sbin/haveged -w 1024' >> /etc/rc.local
# add library path to  ~/.bash_profile
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib' >> ~/.bash_profile
# start service right away
/usr/local/sbin/haveged -w 1024

Configured like this, haveged will monitor your entropy pool and add more entropy to it, whenever the entropy falls below the watermark of 1024 bytes.

It’s interesting to run the following command a few times, either before and after installing haveged, so you can measure the change in entropy pool size.

cat /proc/sys/kernel/random/entropy_avail

Actually, the result on our production environment was stunning. EM responsiveness improved enormously and other problems, like not being able to abort old running instances, where fixed too.