环境说明

OS操作系统：SUSE Linux 11

数据库版 本：MYSQL 5.6

故障问题描述

客户反映MYSQL数据库因为内存不足被重启。

故障分析

1、根据MYSQL报警日志确认问题：

2016-10-24 12:28:49 29226 [Note] Plugin

'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The

InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB:

Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB:

Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB:

Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB:

Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB:

Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log

scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB:

Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB:

Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB:

Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB:

Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB:

from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log

sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB:

Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0

401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128

rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB:

5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering

after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting

crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash

recovery finished.

2016-10-24 12:29:06 29226 [Note] Server

hostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is

available.

2016-10-24 12:29:06 29226 [Note]- '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Server

socket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning]

'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event

Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note]

/usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log'socket: '/etc/mysock/it_mgmad.sock'port: 60002UpSQL Server

2016-10-24 12:30:21 29226 [ERROR]

/usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR]

Out of memory; check if mysqld or some other process uses all available memory;

if not, you may have to use 'ulimit' to allow mysqld to use more memory or you

can add more swap space

161024 12:31:35 mysqld_safe Number of

processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659

[Warning] Buffered warning: Could not increase number of max_open_files to more

than 10240 (request: 25000)

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

2.操作系统日志提示错误

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482]

Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996485] CPU0: hi:0, btch:1 usd:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996486] CPU1: hi:0, btch:1 usd:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996488] CPU0: hi:186, btch:31 usd:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996489] CPU 1: hi:186, btch:31 usd:5

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996491] CPU0: hi:186, btch:31 usd:30

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996492] CPU1: hi:186, btch:31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996496]active_file:353

inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996496]unevictable:0 dirty:6

writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996497]free:21800

slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498]mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB

active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB

unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB

mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB

slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB

writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996505] lowmem_reserve[]: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB

high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB

inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB

present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB

slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB

pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611

all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996513] lowmem_reserve[]: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB

high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB

inactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB

present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB

shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB

kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB

writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996520] lowmem_reserve[]: 0 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB

0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB

43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB

5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996556] Swap cache stats: add 5506519, delete 5470793, find

22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996557] Free swap= 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960]

[ pid ]uidtgid total_vmrss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006974] [571]0571422311-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006981] [ 1689]016899981000 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006983] [ 1704]10017045444117000 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006986] [ 1722]01722482462000 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006988] [ 1725]0172510650000 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006991] [10038]0

1003831835168100 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006994] [10096]0

10096151931000 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006996] [10185]101

101858608146000 hald

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.006999] [10188]0

10188272780000 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007001] [10189]0

1018945052000 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007004] [10284]0

1028486900000 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007]

[10308]0 10308869017000 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007009] [10312]101

1031291290100 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007012] [10313]0

10313869016000 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007014] [12046]0

12046740980-17-1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007017] [12048]0

1204836060000 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007020] [12081]0

12081472316000 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007022] [12083]0

12083208817000 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007025] [12140]0

12140421611-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007027] [12141]0

12141421610-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007029] [12224]0

1222467591100 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031]

[12701]0 1270117296280-17-1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007034] [12715]0

12715179330100 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007036] [12730]0

127305955771000 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007038] [12798]0

12798955228000 master

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007041] [12825]51

128251005231100 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007043] [12864]0

128641671147100 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007046] [12878]0 1287827377121000 X

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007048] [12885]0

12885369715000 cron

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007051] [12924]0

1292411212100 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007055] [12925]0

1292511212100 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007058] [12926]0

1292611212000 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007061] [12927]0

1292711212100 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007064] [12928]0

1292811212100 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007067] [12929]0

1292911212000 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007071] [13064]107

1306455480100 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007074] [13065]107

1306533620100 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007077] [13066]107

130663955182000 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007080] [13079]107

13079986367000 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007083] [13083]107

13083318610100 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007086] [13084]107

13084708312912000 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007090] [13086]107

13086202070100 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007092] [13094]107

1309457536135000 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007095] [13096]107

13096108540000 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007098] [13097]107

13097542460000 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007101] [21349]1018

2134928572100 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007105] [12777]1016

1277728572000 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007109] [18450]0

184501338449100 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007112] [18452]0

184526680913462100 X

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007115] [18489]107

1848955480100 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007118] [18518]0

18518246130100 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007121] [18540]0

185403958489100 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007124] [18613]0

18613417819000 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126]

[18619]0 1861955480100 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007129] [18620]0

1862034290000 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007132] [18623]0

186239887123100 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007135] [18627]0

1862791696332000 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007138] [18631]0

18631238141100 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007141] [18635]0

18635109080100 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007144] [18639]0

186391682219000 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007147] [18645]0

1864551782400100 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007149] [18650]0

1865093659621100 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007152] [18651]0

1865113590513563100 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007155] [18653]0

18653366240000 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007160] [18659]0

1865982105455100 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007171] [18662]0

18662120830100 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007174] [18666]0

18666136091000 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007178] [18668]0

18668157741100 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007181] [18677]0

18677754561100 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007184] [18681]0

18681569471000 python

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007187] [18686]0

1868639298113000 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007189] [18688]0

1868857564125000 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007192] [18692]0

1869238020114100 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007195] [18693]0

18693481860100 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007198] [18718]0

18718109620000 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007201] [21417]0

2141759168142100 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007204] [21420]0

2142057129199000 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007208] [21463]0

21463358831000 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007211] [21464]0

21464134454000 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007214] [21471]0

2147132211000 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007217] [21501]0

21501118060114000 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007221] [21621]0

2162155974123000 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007224] [21622]0

2162216160000 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007227] [21623]0

2162335051000 bash

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007230] [21649]0

2164992441000 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007233] [31142]1016

311421100222219744100 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007236] [31181]1018

311811039509163992100 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007239] [699]1017699285698000 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007242] [ 1625]10171625107667741429810 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007246] [ 1861]1019186164137248223100 java

Oct 25 13:41:07 TSPTONLDB01 kernel:

[14317510.007284] [14971]51

1497110014128000 pickup

Oct 25 13:41:07 TSPTONLDB01

kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283

or sacrifice child

Oct 25 13:41:07 TSPTONLDB01

kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB,

anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01

kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da,

order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: GX 3.0.13-0.27-default #1

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351803][] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351814][] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351825][] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834][]

oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351841][] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351848][] __alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351854][] __alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351863][] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351871][] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351878][] __do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351884][]

handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351893][] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.351900][] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel:

[14317788.353042]

通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统KILL掉MYSQLD进程的情况。

在该服务器上有3个MYSQL实例，每个MYSQL实例的INNODB缓冲池设置为2GB，但操作系统的物理内存

只有4GB，另外在服务器上还有一个数据同步的JAVA进程。结合日志分析，MYSQL内存分配不合理，建议把服务器内存扩展到10GB。

建议MYSQL INNODB池计算方法：INNODB缓冲池内存=操作系统物理内存-(预留给操作系统的内存1-2GB)

-(JAVA进程的内存)-(MYSQL最大连接数*2MB)