Operations/Server Documentation/Resource Analysis

From Habari Project

Jump to: navigation, search


Contents

Abstract

Due to the un-funded nature of the Habari project and the wide range of services offered under the habariproject.org domain, efficient utilization of server resources is a priority. To that end significant monitoring and research has been done to best characterize the typical working environment available on the hp.o server.

Data Gathering

Before making any assumptions we need data. While memory usage fluctuates greatly there are baselines that have been established over a significant monitoring interval. Both average and extreme usage is illustrated below. For analysis purposes we are only looking at the Resident Set Size (RSS), which should exclude things like SWAP usage from the calculations.

Average

The below snippet [1] portrays a good estimate of the average system load.

# ps -Aly --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
...
S    33 32348  2881  0  80   0  4036 18770 -      ?        00:05:46 nginx
S    33 32347  2881  0  80   0  4620 18849 -      ?        00:05:44 nginx
S    33 32346  2881  0  80   0  4872 18928 -      ?        00:05:48 nginx
S    33 32345  2881  0  80   0  4968 18898 -      ?        00:05:40 nginx
S    33  1209 21562  0  80   0 25388 93509 -      ?        00:00:19 apache2
S   103  2363     1  1  80   0 43228 78957 -      ?        06:30:32 mysqld
S    33 12477  2694  1  80   0 69228 62492 -      ?        00:52:32 php5-fpm
S    33 12476  2694  1  80   0 69308 62449 -      ?        00:52:23 php5-fpm
S    33  2935  2694  0  80   0 69800 62850 -      ?        03:58:39 php5-fpm
S    33  2695  2694  0  80   0 71080 62711 -      ?        03:57:32 php5-fpm
S    33  2886  2694  0  80   0 77708 63618 -      ?        03:58:08 php5-fpm
S    33  2845 21562  0  80   0 103924 117171 -    ?        00:04:06 apache2
S    33 15958     1 17  80   0 114684 100944 -    ?        03:36:18 tracd

Extreme

In some instances the memory usage of individual Apache and Trac processes can balloon to significantly higher levels, as illustrated in the below snippet [2]:

# ps -Aly --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
...
S    33 11554  6275  0  80   0 10484 78622 semtim ?        00:00:00 apache2
S    33 11437  6275  0  80   0 12048 78690 semtim ?        00:00:00 apache2
S     0  6275     1  0  80   0 15920 78473 d_look ?        00:00:00 apache2
S    33 11158  6275  0  80   0 21580 79239 semtim ?        00:00:00 apache2
S    33 11397  6275  0  80   0 36056 79854 semtim ?        00:00:00 apache2
S    33  7326  6276  0  80   0 41940 61848 -      ?        00:00:16 trac.fcgi
S    33  7324  6276  1  80   0 46500 64761 -      ?        00:00:21 trac.fcgi
S    33  7325  6276  1  80   0 47504 81228 -      ?        00:00:24 trac.fcgi
S    33 11350  6275  0  80   0 58588 88353 -      ?        00:00:00 apache2
S    33  6317  6276  2  80   0 70028 70428 -      ?        00:01:04 trac.fcgi
S   106  7182  7143  0  80   0 113976 85284 -     ?        00:00:16 mysqld

Analysis

The top consumers of memory are MySQL, Apache, and Trac. Each has a baseline defined.

MySQL

Virtually all of the services provided on hp.o are database-based, so the highest memory consumption on the box is logically the MySQL process. The caching values of MySQL have been tuned extensively for the specific use cases involved and are detailed in MySQL Configuration.

In general the assumption is that MySQL will regularly use at least 128M of RAM for caching and up to an additional 40M for combined other temporary sorting of results, etc. Total, it can be expected to easily approach 180M during load.

Apache

Behind MySQL is the web server itself, Apache. The main variable in an mpm_prefork Apache environment is the number of processes allowed.

For our purposes Apache can easily consume over 50 - 55M of RAM per process.

Trac

Trac's memory usage varies more widely than anything else on the system, making it exceptionally difficult to predict.

Each Trac process should be expected to easily consume 35 - 40M. As seen in the Extreme example, Trac should be afforded extra care and anticipated to use up to more than twice this amount.

References

  1. Captured 2011-02-03, morning EDT.
  2. Captured 2010-09-13, evening EDT.
Personal tools