Web lists-archives.com

Unable to start mysql after power Failure




Hi All

We are trying to High Availability solution for our database. We have our application running on a node-1 and the DB running on the shared storage which is shared across node-1 & node-2. Whenever there is a power Failure on node-1 , we start our application on node-2 but our problem is that now after a failover onto node-2 the DB deamon on node-2 will fail to start.

Mysql version information:

******************************************
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 5.6.10-enterprise-commercial-advanced MySQL Enterprise Server - Advanced Edition (Commercial)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
*******************************************

On the node-2 when the DB deamon tries to start we are seeing the following error messages in the log file:

Error Message *************************************************************
150703 16:01:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/data 2015-07-03 16:01:11 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-07-03 16:01:11 16278 [Note] Plugin 'FEDERATED' is disabled.
2015-07-03 16:01:12 16278 [Note] InnoDB: The InnoDB memory heap is disabled
2015-07-03 16:01:12 16278 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-07-03 16:01:12 16278 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-07-03 16:01:12 16278 [Note] InnoDB: CPU does not support crc32 instructions
2015-07-03 16:01:12 16278 [Note] InnoDB: Using Linux native AIO
2015-07-03 16:01:12 16278 [Note] InnoDB: Initializing buffer pool, size = 1.0G 2015-07-03 16:01:12 16278 [Note] InnoDB: Completed initialization of buffer pool 2015-07-03 16:01:12 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:12 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:12 16278 [Note] InnoDB: Retrying to lock the first data file 2015-07-03 16:01:13 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:13 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:14 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:01:14 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:01:15 16278 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:02:52 16278 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:02:52 16278 [Note] InnoDB: Unable to open the first data file
2015-07-03 16:02:52 7f9b9194e720 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2015-07-03 16:02:52 16278 [ERROR] InnoDB: Can't open './ibdata1'
2015-07-03 16:02:52 16278 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' init function returned error. 2015-07-03 16:02:52 16278 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-07-03 16:02:52 16278 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-07-03 16:02:52 16278 [ERROR] Aborting

2015-07-03 16:02:52 16278 [Note] Binlog end
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'partition'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'ARCHIVE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'BLACKHOLE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_INSERTED'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_METRICS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_CMP'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'INNODB_TRX'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'CSV'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MEMORY'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MRG_MYISAM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'MyISAM'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'sha256_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'mysql_old_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'mysql_native_password'
2015-07-03 16:02:52 16278 [Note] Shutting down plugin 'binlog'
2015-07-03 16:02:52 16278 [Note] /var/lib/mysql/bin/mysqld: Shutdown complete

150703 16:02:52 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 150703 16:04:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/data 2015-07-03 16:04:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-07-03 16:04:03 17753 [Note] Plugin 'FEDERATED' is disabled.
2015-07-03 16:04:03 17753 [Note] InnoDB: The InnoDB memory heap is disabled
2015-07-03 16:04:03 17753 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-07-03 16:04:03 17753 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-07-03 16:04:03 17753 [Note] InnoDB: CPU does not support crc32 instructions
2015-07-03 16:04:03 17753 [Note] InnoDB: Using Linux native AIO
2015-07-03 16:04:03 17753 [Note] InnoDB: Initializing buffer pool, size = 1.0G 2015-07-03 16:04:03 17753 [Note] InnoDB: Completed initialization of buffer pool 2015-07-03 16:04:03 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:03 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:03 17753 [Note] InnoDB: Retrying to lock the first data file 2015-07-03 16:04:04 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:04 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:05 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:05 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:06 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:06 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:07 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:04:07 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2015-07-03 16:04:08 17753 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-03 16:05:43 17753 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2015-07-03 16:05:43 17753 [Note] InnoDB: Unable to open the first data file
2015-07-03 16:05:43 7f4cbe5a3720 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2015-07-03 16:05:43 17753 [ERROR] InnoDB: Can't open './ibdata1'
2015-07-03 16:05:43 17753 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' init function returned error. 2015-07-03 16:05:43 17753 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-07-03 16:05:43 17753 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-07-03 16:05:43 17753 [ERROR] Aborting

2015-07-03 16:05:43 17753 [Note] Binlog end
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'partition'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'ARCHIVE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'BLACKHOLE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_INSERTED'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_METRICS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_CMP'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'INNODB_TRX'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'CSV'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MEMORY'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MRG_MYISAM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'MyISAM'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'sha256_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'mysql_old_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'mysql_native_password'
2015-07-03 16:05:43 17753 [Note] Shutting down plugin 'binlog'
2015-07-03 16:05:43 17753 [Note] /var/lib/mysql/bin/mysqld: Shutdown complete

150703 16:05:43 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended End of Error Message *****************************************************************

Appreciate your help in this regard.

Thanks
Jatin