23.5.2.1 NDB Cluster: Messages in the Cluster Log

The following table lists the most common NDB cluster log messages. For information about the cluster log, log events, and event types, see Section 23.5.3, “Event Reports Generated in NDB Cluster”. These log messages also correspond to log event types in the MGM API; see The Ndb_logevent_type Type, for related information of interest to Cluster API developers.

Table 23.52 Common NDB cluster log messages

Log MessageDescriptionEvent NameEvent TypePrioritySeverity
Node mgm_node_id: Node data_node_id ConnectedThe data node having node ID node_id has connected to the management server (node mgm_node_id).ConnectedConnection8INFO
Node mgm_node_id: Node data_node_id DisconnectedThe data node having node ID data_node_id has disconnected from the management server (node mgm_node_id).DisconnectedConnection8ALERT
Node data_node_id: Communication to Node api_node_id closedThe API node or SQL node having node ID api_node_id is no longer communicating with data node data_node_id.CommunicationClosedConnection8INFO
Node data_node_id: Communication to Node api_node_id openedThe API node or SQL node having node ID api_node_id is now communicating with data node data_node_id.CommunicationOpenedConnection8INFO
Node mgm_node_id: Node api_node_id: API versionThe API node having node ID api_node_id has connected to management node mgm_node_id using NDB API version version (generally the same as the MySQL version number).ConnectedApiVersionConnection8INFO
Node node_id: Global checkpoint gci startedA global checkpoint with the ID gci has been started; node node_id is the master responsible for this global checkpoint.GlobalCheckpointStartedCheckpoint9INFO
Node node_id: Global checkpoint gci completedThe global checkpoint having the ID gci has been completed; node node_id was the master responsible for this global checkpoint.GlobalCheckpointCompletedCheckpoint10INFO
Node node_id: Local checkpoint lcp started. Keep GCI = current_gci oldest restorable GCI = old_gciThe local checkpoint having sequence ID lcp has been started on node node_id. The most recent GCI that can be used has the index current_gci, and the oldest GCI from which the cluster can be restored has the index old_gci.LocalCheckpointStartedCheckpoint7INFO
Node node_id: Local checkpoint lcp completedThe local checkpoint having sequence ID lcp on node node_id has been completed.LocalCheckpointCompletedCheckpoint8INFO
Node node_id: Local Checkpoint stopped in CALCULATED_KEEP_GCIThe node was unable to determine the most recent usable GCI.LCPStoppedInCalcKeepGciCheckpoint0ALERT
Node node_id: Table ID = table_id, fragment ID = fragment_id has completed LCP on Node node_id maxGciStarted: started_gci maxGciCompleted: completed_gciA table fragment has been checkpointed to disk on node node_id. The GCI in progress has the index started_gci, and the most recent GCI to have been completed has the index completed_gci.LCPFragmentCompletedCheckpoint11INFO
Node node_id: ACC Blocked num_1 and TUP Blocked num_2 times last secondUndo logging is blocked because the log buffer is close to overflowing.UndoLogBlockedCheckpoint7INFO
Node node_id: Start initiated versionData node node_id, running NDB version version, is beginning its startup process.NDBStartStartedStartUp1INFO
Node node_id: Started versionData node node_id, running NDB version version, has started successfully.NDBStartCompletedStartUp1INFO
Node node_id: STTORRY received after restart finishedThe node has received a signal indicating that a cluster restart has completed.STTORRYRecievedStartUp15INFO
Node node_id: Start phase phase completed (type)The node has completed start phase phase of a type start. For a listing of start phases, see Section 23.5.4, “Summary of NDB Cluster Start Phases”. (type is one of initial, system, node, initial node, or <Unknown>.)StartPhaseCompletedStartUp4INFO
Node node_id: CM_REGCONF president = president_id, own Node = own_id, our dynamic id = dynamic_idNode president_id has been selected as “president”. own_id and dynamic_id should always be the same as the ID (node_id) of the reporting node.CM_REGCONFStartUp3INFO
Node node_id: CM_REGREF from Node president_id to our Node node_id. Cause = causeThe reporting node (ID node_id) was unable to accept node president_id as president. The cause of the problem is given as one of Busy, Election with wait = false, Not president, Election without selecting new candidate, or No such cause.CM_REGREFStartUp8INFO
Node node_id: We are Node own_id with dynamic ID dynamic_id, our left neighbor is Node id_1, our right is Node id_2The node has discovered its neighboring nodes in the cluster (node id_1 and node id_2). node_id, own_id, and dynamic_id should always be the same; if they are not, this indicates a serious misconfiguration of the cluster nodes.FIND_NEIGHBOURSStartUp8INFO
Node node_id: type shutdown initiatedThe node has received a shutdown signal. The type of shutdown is either Cluster or Node.NDBStopStartedStartUp1INFO
Node node_id: Node shutdown completed [, action] [Initiated by signal signal.]The node has been shut down. This report may include an action, which if present is one of restarting, no start, or initial. The report may also include a reference to an NDB Protocol signal; for possible signals, refer to Operations and Signals.NDBStopCompletedStartUp1INFO
Node node_id: Forced node shutdown completed [, action]. [Occurred during startphase start_phase.] [ Initiated by signal.] [Caused by error error_code: 'error_message(error_classification). error_status'. [(extra info extra_code)]]The node has been forcibly shut down. The action (one of restarting, no start, or initial) subsequently being taken, if any, is also reported. If the shutdown occurred while the node was starting, the report includes the start_phase during which the node failed. If this was a result of a signal sent to the node, this information is also provided (see Operations and Signals, for more information). If the error causing the failure is known, this is also included; for more information about NDB error messages and classifications, see NDB Cluster API Errors.NDBStopForcedStartUp1ALERT
Node node_id: Node shutdown abortedThe node shutdown process was aborted by the user.NDBStopAbortedStartUp1INFO
Node node_id: StartLog: [GCI Keep: keep_pos LastCompleted: last_pos NewestRestorable: restore_pos]This reports global checkpoints referenced during a node start. The redo log prior to keep_pos is dropped. last_pos is the last global checkpoint in which data node the participated; restore_pos is the global checkpoint which is actually used to restore all data nodes.StartREDOLogStartUp4INFO
startup_message [Listed separately; see below.]There are a number of possible startup messages that can be logged under different circumstances. These are listed separately; see Section 23.5.2.2, “NDB Cluster Log Startup Messages”.StartReportStartUp4INFO
Node node_id: Node restart completed copy of dictionary informationCopying of data dictionary information to the restarted node has been completed.NR_CopyDictNodeRestart8INFO
Node node_id: Node restart completed copy of distribution informationCopying of data distribution information to the restarted node has been completed.NR_CopyDistrNodeRestart8INFO
Node node_id: Node restart starting to copy the fragments to Node node_idCopy of fragments to starting data node node_id has begunNR_CopyFragsStartedNodeRestart8INFO
Node node_id: Table ID = table_id, fragment ID = fragment_id have been copied to Node node_idFragment fragment_id from table table_id has been copied to data node node_idNR_CopyFragDoneNodeRestart10INFO
Node node_id: Node restart completed copying the fragments to Node node_idCopying of all table fragments to restarting data node node_id has been completedNR_CopyFragsCompletedNodeRestart8INFO
Node node_id: Node node1_id completed failure of Node node2_idData node node1_id has detected the failure of data node node2_idNodeFailCompletedNodeRestart8ALERT
All nodes completed failure of Node node_idAll (remaining) data nodes have detected the failure of data node node_idNodeFailCompletedNodeRestart8ALERT
Node failure of node_idblock completedThe failure of data node node_id has been detected in the blockNDB kernel block, where block is 1 of DBTC, DBDICT, DBDIH, or DBLQH; for more information, see NDB Kernel BlocksNodeFailCompletedNodeRestart8ALERT
Node mgm_node_id: Node data_node_id has failed. The Node state at failure was state_codeA data node has failed. Its state at the time of failure is described by an arbitration state code state_code: possible state code values can be found in the file include/kernel/signaldata/ArbitSignalData.hpp.NODE_FAILREPNodeRestart8ALERT
President restarts arbitration thread [state=state_code] or Prepare arbitrator node node_id [ticket=ticket_id] or Receive arbitrator node node_id [ticket=ticket_id] or Started arbitrator node node_id [ticket=ticket_id] or Lost arbitrator node node_id - process failure [state=state_code] or Lost arbitrator node node_id - process exit [state=state_code] or Lost arbitrator node node_id - error_message [state=state_code]This is a report on the current state and progress of arbitration in the cluster. node_id is the node ID of the management node or SQL node selected as the arbitrator. state_code is an arbitration state code, as found in include/kernel/signaldata/ArbitSignalData.hpp. When an error has occurred, an error_message, also defined in ArbitSignalData.hpp, is provided. ticket_id is a unique identifier handed out by the arbitrator when it is selected to all the nodes that participated in its selection; this is used to ensure that each node requesting arbitration was one of the nodes that took part in the selection process.ArbitStateNodeRestart6INFO
Arbitration check lost - less than 1/2 nodes left or Arbitration check won - all node groups and more than 1/2 nodes left or Arbitration check won - node group majority or Arbitration check lost - missing node group or Network partitioning - arbitration required or Arbitration won - positive reply from node node_id or Arbitration lost - negative reply from node node_id or Network partitioning - no arbitrator available or Network partitioning - no arbitrator configured or Arbitration failure - error_message [state=state_code]This message reports on the result of arbitration. In the event of arbitration failure, an error_message and an arbitration state_code are provided; definitions for both of these are found in include/kernel/signaldata/ArbitSignalData.hpp.ArbitResultNodeRestart2ALERT
Node node_id: GCP Take over startedThis node is attempting to assume responsibility for the next global checkpoint (that is, it is becoming the master node)GCP_TakeoverStartedNodeRestart7INFO
Node node_id: GCP Take over completedThis node has become the master, and has assumed responsibility for the next global checkpointGCP_TakeoverCompletedNodeRestart7INFO
Node node_id: LCP Take over startedThis node is attempting to assume responsibility for the next set of local checkpoints (that is, it is becoming the master node)LCP_TakeoverStartedNodeRestart7INFO
Node node_id: LCP Take over completedThis node has become the master, and has assumed responsibility for the next set of local checkpointsLCP_TakeoverCompletedNodeRestart7INFO
Node node_id: Trans. Count = transactions, Commit Count = commits, Read Count = reads, Simple Read Count = simple_reads, Write Count = writes, AttrInfo Count = AttrInfo_objects, Concurrent Operations = concurrent_operations, Abort Count = aborts, Scans = scans, Range scans = range_scansThis report of transaction activity is given approximately once every 10 secondsTransReportCountersStatistic8INFO
Node node_id: Operations=operationsNumber of operations performed by this node, provided approximately once every 10 secondsOperationReportCountersStatistic8INFO
Node node_id: Table with ID = table_id createdA table having the table ID shown has been createdTableCreatedStatistic7INFO
Node node_id: Mean loop Counter in doJob last 8192 times = countJobStatisticStatistic9INFO
Mean send size to Node = node_id last 4096 sends = bytes bytesThis node is sending an average of bytes bytes per send to node node_idSendBytesStatisticStatistic9INFO
Mean receive size to Node = node_id last 4096 sends = bytes bytesThis node is receiving an average of bytes of data each time it receives data from node node_idReceiveBytesStatisticStatistic9INFO
Node node_id: Data usage is data_memory_percentage% (data_pages_used 32K pages of total data_pages_total) / Node node_id: Index usage is index_memory_percentage% (index_pages_used 8K pages of total index_pages_total) This report is generated when a DUMP 1000 command is issued in the cluster management clientMemoryUsageStatistic5INFO
Node node1_id: Transporter to node node2_id reported error error_code: error_messageA transporter error occurred while communicating with node node2_id; for a listing of transporter error codes and messages, see NDB Transporter Errors, in MySQL NDB Cluster Internals ManualTransporterErrorError2ERROR
Node node1_id: Transporter to node node2_id reported error error_code: error_messageA warning of a potential transporter problem while communicating with node node2_id; for a listing of transporter error codes and messages, see NDB Transporter Errors, for more informationTransporterWarningError8WARNING
Node node1_id: Node node2_id missed heartbeat heartbeat_idThis node missed a heartbeat from node node2_idMissedHeartbeatError8WARNING
Node node1_id: Node node2_id declared dead due to missed heartbeatThis node has missed at least 3 heartbeats from node node2_id, and so has declared that node “dead”DeadDueToHeartbeatError8ALERT
Node node1_id: Node Sent Heartbeat to node = node2_idThis node has sent a heartbeat to node node2_idSentHeartbeatInfo12INFO
Node node_id: Event buffer status (object_id): used=bytes_used (percent_used% of alloc) alloc=bytes_allocated max=bytes_available latest_consumed_epoch=latest_consumed_epoch latest_buffered_epoch=latest_buffered_epoch report_reason=report_reasonThis report is seen during heavy event buffer usage, for example, when many updates are being applied in a relatively short period of time; the report shows the number of bytes and the percentage of event buffer memory used, the bytes allocated and percentage still available, and the latest buffered and consumed epochs; for more information, see Section 23.5.2.3, “Event Buffer Reporting in the Cluster Log”EventBufferStatus2Info7INFO
Node node_id: Entering single user mode, Node node_id: Entered single user mode Node API_node_id has exclusive access, Node node_id: Entering single user modeThese reports are written to the cluster log when entering and exiting single user mode; API_node_id is the node ID of the API or SQL having exclusive access to the cluster (for more information, see Section 23.5.6, “NDB Cluster Single User Mode”); the message Unknown single user report API_node_id indicates an error has taken place and should never be seen in normal operationSingleUserInfo7INFO
Node node_id: Backup backup_id started from node mgm_node_idA backup has been started using the management node having mgm_node_id; this message is also displayed in the cluster management client when the START BACKUP command is issued; for more information, see Section 23.5.8.2, “Using The NDB Cluster Management Client to Create a Backup”BackupStartedBackup7INFO
Node node_id: Backup backup_id started from node mgm_node_id completed. StartGCP: start_gcp StopGCP: stop_gcp #Records: records #LogRecords: log_records Data: data_bytes bytes Log: log_bytes bytesThe backup having the ID backup_id has been completed; for more information, see Section 23.5.8.2, “Using The NDB Cluster Management Client to Create a Backup”BackupCompletedBackup7INFO
Node node_id: Backup request from mgm_node_id failed to start. Error: error_codeThe backup failed to start; for error codes, see MGM API ErrorsBackupFailedToStartBackup7ALERT
Node node_id: Backup backup_id started from mgm_node_id has been aborted. Error: error_codeThe backup was terminated after starting, possibly due to user interventionBackupAbortedBackup7ALERT