Когда кончается память...

CentOS, кончилась память на одной машине, своп тоже. Удалось подключиться по ссш и снять ошибки, кусок ниже (они повторяются)

Swap cache: add 273609, delete 273183, find 8928/9325, race 0+3
Free swap = 0kB
Total swap = 1081336kB
audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[] out_of_memory+0x72/0x1a3
[] __alloc_pages+0x24e/0x2cf
[] __do_page_cache_readahead+0xc8/0x18b
[] filemap_nopage+0x157/0x34c
[] __handle_mm_fault+0x329/0x1487
[] do_futex+0x710/0xbf2
[] do_readv_writev+0x22e/0x247
[] do_page_fault+0x6c0/0xc23
[] sys_futex+0x111/0x127
[] do_page_fault+0x0/0xc23
[] error_code+0x2b/0x30
=======================
DMA per-cpu:
cpu 0 hot: high 186, batch 31 used:17
cpu 0 cold: high 62, batch 15 used:14
cpu 1 hot: high 186, batch 31 used:2
cpu 1 cold: high 62, batch 15 used:49
DMA32 per-cpu: empty
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 2876kB (0kB HighMem)
Active:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:1audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[] out_of_memory+0x72/0x1a3
[] __alloc_pages+0x24e/0x2cf
[] __do_page_cache_readdequeue_task+0x13/0x26
[] _spin_lock_irqsave+0x8/0x28
[] lock_hrtimer_base+0x19/0x35
[] hrtimer_try_to_cancel+0x3c/0x42
[] do_page_fault+0x6c0/0xc23
[] hrtimer_nanosleep+0x50/0x106
[] do_page_fault+0sys_select+0x15a/0x180
[] do_page_fault+0x0/0xc23
[] error_codecpu 0 hot: high 186, batch 31 used:15
cpu 0 cold: high 62, batch 15 used:16
cpu 1 hot: high 186, batch 31 used:24
cpu 1 cold: high 62, batch 15 used:16
DMA32 per-cpu: empty
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 2916kB (0kB HighMem)
Active:40997 inactive:44364 dirty:2 writeback:0 unstable:0 free:729 slab:21897 mapped-file:1182 mapped-anon:83744 pagetables:17893
DMA free:2916kB min:2916kB low:3644kB high:4372kB active:163988kB inactive:177456kB present:532480kB pages_scanned:878701 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB mincrond invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[] out_of_memory+0x72/0x1a3
[] __alloc_pages+0x24e/0x2cf
[Active:48666 inactivDMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inaHighMem free:0kB min:128kB lo0*4096kB = 2888kB
DMA32: empty
Normal: empty
HighMem: empty
2569 pagecache pages
Swap cache: add 1314270, delete 1313843, find 200003/331661, racOut of memory: Killed process 31528, UID 0, (php).

Единственный вариант при этом - сделать хард резет. Софт тоже выполниться не может.

Борьба с этим - настраивать лимиты и настроить сервер так, чтобы процессы не могли занять памяти сильно больше, чем доступно.