oracle respawn,ohas无法启动 - Oracle数据库管理 - ITPUB论坛－中国专业的IT技术社区...

女爷i 2022-10-21 13:53 172阅读 0赞

How to Troubleshoot Grid Infrastructure Startup Issues \[ID 1050908.1\]

Case 1: OHASD.BIN does not start

As ohasd.bin is responsible to start up all other cluserware processes directly or indirectly, it needs to start up properly for the rest of the stack to come up.

Automatic ohasd.bin start up depends on the following:

1. OS is at appropriate run level:

OS need to be at specified run level before CRS will try to start up.

To find out at which run level the clusterware needs to come up:

cat /etc/inittab|grep init.ohasd

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1

Above example shows CRS suppose to run at run level 3 and 5; please note depend on platform, CRS comes up at different run level.

To find out current run level:

who -r

2. "init.ohasd run" is up

On Linux/UNIX, as "init.ohasd run" is configured in /etc/inittab, process init (pid 1, /sbin/init on Linux, Solaris and hp-ux, /usr/sbin/init on AIX) will start and respawn "init.ohasd run" if it fails. Without "init.ohasd run" up and running, ohasd.bin will not start:

ps -ef|grep init.ohasd|grep -v grep

root      2279     1  0 18:14 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run

If any rc Snncommand script (located in rcn.d, example S98gcstartup) stuck, init process may not start "/etc/init.d/init.ohasd run"; please engage OS vendor to find out why relevant Snncommand script stuck.

3. Cluserware auto start is enabled - its enabled by default

By default CRS is enabled for auto start upon node reboot, to enable:

$GRID\_HOME/bin/crsctl enable crs

To verify whether its currently enabled or not:

cat $SCRBASE/$HOSTNAME/root/ohasdstr

enable

SCRBASE is /etc/oracle/scls\_scr on Linux and AIX, /var/opt/oracle/scls\_scr on hp-ux and Solaris

Note: NEVER EDIT THE FILE MANUALLY, use "crsctl enable/disable crs" command instead.

4. syslogd is up and OS is able to execute init script S96ohasd

OS may stuck with some other Snn script while node is coming up, thus never get chance to execute S96ohasd; if that's the case, following message will not be in OS messages:

Jan 20 20:46:51 rac1 logger: Oracle HA daemon is enabled for autostart.

If you don't see above message, the other possibility is syslogd(/usr/sbin/syslogd) is not fully up. Grid may fail to come up in that case as well. This may not apply to AIX.

To find out whether OS is able to execute S96ohasd while node is coming up, modify ohasd:

From:

case \`$CAT $AUTOSTARTFILE\` in

enable\*)

$LOGERR "Oracle HA daemon is enabled for autostart."

To:

case \`$CAT $AUTOSTARTFILE\` in

enable\*)

/bin/touch /tmp/ohasd.start."\`date\`"

$LOGERR "Oracle HA daemon is enabled for autostart."

After a node reboot, if you don't see /tmp/ohasd.start.timestamp get created, it means OS stuck with some other Snn script. If you do see /tmp/ohasd.start.timestamp but not "Oracle HA daemon is enabled for autostart" in messages, likely syslogd is not fully up. For both case, you will need engage System Administrator to find out the issue on OS level. For latter case, the workaround is to "sleep" for about 2 minutes, modify ohasd:

From:

case \`$CAT $AUTOSTARTFILE\` in

enable\*)

$LOGERR "Oracle HA daemon is enabled for autostart."

To:

case \`$CAT $AUTOSTARTFILE\` in

enable\*)

/bin/sleep 120

$LOGERR "Oracle HA daemon is enabled for autostart."

5. File System that GRID\_HOME resides is online when init script S96ohasd is executed; once S96ohasd is executed, following message should be in OS messages file:

Jan 20 20:46:51 rac1 logger: Oracle HA daemon is enabled for autostart.

Jan 20 20:46:57 rac1 logger: exec /ocw/grid/perl/bin/perl -I/ocw/grid/perl/lib /ocw/grid/bin/crswrapexece.pl /ocw/grid/crs/install/s\_crsconfig\_rac1\_env.txt /ocw/grid/bin/ohasd.bin "reboot"

If you see the first line, but not the last line, likely the filesystem containing the GRID\_HOME was not online while S96ohasd is executed.

6. Oracle Local Registry (OLR, $GRID\_HOME/cdata/$\{HOSTNAME\}.olr) is accessible

ls -l $GRID\_HOME/cdata/\*.olr

\-rw------- 1 root  oinstall 272756736 Feb  2 18:20 rac1.olr

If the OLR is inaccessible or corrupted, likely ohasd.log will have similar messages like following:

2010-01-24 22:59:10.470: \[ default\]\[1373676464\] Initializing OLR

2010-01-24 22:59:10.472: \[  OCROSD\]\[1373676464\]utopen:6m':failed in stat OCR file/disk /ocw/grid/cdata/rac1.olr, errno=2, os err string=No such file or directory

2010-01-24 22:59:10.472: \[  OCROSD\]\[1373676464\]utopen:7:failed to open any OCR file/disk, errno=2, os err string=No such file or directory

2010-01-24 22:59:10.473: \[  OCRRAW\]\[1373676464\]proprinit: Could not open raw device

2010-01-24 22:59:10.473: \[  OCRAPI\]\[1373676464\]a\_init:16!: Backend init unsuccessful : \[26\]

2010-01-24 22:59:10.473: \[  CRSOCR\]\[1373676464\] OCR context init failure.  Error: PROCL-26: Error while accessing the physical storage Operating System error \[No such file or directory\] \[2\]

2010-01-24 22:59:10.473: \[ default\]\[1373676464\] OLR initalization failured, rc=26

2010-01-24 22:59:10.474: \[ default\]\[1373676464\]Created alert : (:OHAS00106

![5f60efcaccfa40b26873a6a9cb266119.gif][] :  Failed to initialize Oracle Local Registry

2010-01-24 22:59:10.474: \[ default\]\[1373676464\]\[PANIC\] OHASD exiting; Could not init OLR

2010-01-24 23:01:46.275: \[  OCROSD\]\[1228334000\]utread:3: Problem reading buffer 1907f000 buflen 4096 retval 0 phy\_offset 102400 retry 5

2010-01-24 23:01:46.275: \[  OCRRAW\]\[1228334000\]propriogid:1\_1: Failed to read the whole bootblock. Assumes invalid format.

2010-01-24 23:01:46.275: \[  OCRRAW\]\[1228334000\]proprioini: all disks are not OCR/OLR formatted

2010-01-24 23:01:46.275: \[  OCRRAW\]\[1228334000\]proprinit: Could not open raw device

2010-01-24 23:01:46.275: \[  OCRAPI\]\[1228334000\]a\_init:16!: Backend init unsuccessful : \[26\]

2010-01-24 23:01:46.276: \[  CRSOCR\]\[1228334000\] OCR context init failure.  Error: PROCL-26: Error while accessing the physical storage

2010-01-24 23:01:46.276: \[ default\]\[1228334000\] OLR initalization failured, rc=26

2010-01-24 23:01:46.276: \[ default\]\[1228334000\]Created alert : (:OHAS00106

![5f60efcaccfa40b26873a6a9cb266119.gif][] :  Failed to initialize Oracle Local Registry

2010-01-24 23:01:46.277: \[ default\]\[1228334000\]\[PANIC\] OHASD exiting; Could not init OLR

2010-08-04 13:13:11.102: \[   CRSPE\]\[35\] Resources parsed

2010-08-04 13:13:11.103: \[   CRSPE\]\[35\] Server \[\] has been registered with the PE data model

2010-08-04 13:13:11.103: \[   CRSPE\]\[35\] STARTUPCMD\_REQ = false:

2010-08-04 13:13:11.103: \[   CRSPE\]\[35\] Server \[\] has changed state from \[Invalid/unitialized\] to \[VISIBLE\]

2010-08-04 13:13:11.103: \[  CRSOCR\]\[31\] Multi Write Batch processing...

2010-08-04 13:13:11.103: \[ default\]\[35\] Dump State Starting ...

2010-08-04 13:13:11.112: \[   CRSPE\]\[35\] SERVERS:

:VISIBLE:address\{ \{Absolute|Node:0|Process:-1|Type:1\}\}; recovered state:VISIBLE. Assigned to no pool

\------------- SERVER POOLS:

Free \[min:0\]\[max:-1\]\[importance:0\] NO SERVERS ASSIGNED

2010-08-04 13:13:11.113: \[   CRSPE\]\[35\] Dumping ICE contents...:ICE operation count: 0

2010-08-04 13:13:11.113: \[ default\]\[35\] Dump State Done.

The solution is to restore a good backup of OLR with "ocrconfig -local -restore ocr\_backup\_name". By default, OLR will be backed up to $GRID\_HOME/cdata/$HOST/backup\_$TIME\_STAMP.olr once installation is complete.

7. ohasd.bin is able to access network socket files:

2010-06-29 10:31:01.570: \[ COMMCRS\]\[1206901056\]clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=procr\_local\_conn\_0\_PROL))

2010-06-29 10:31:01.571: \[  OCRSRV\]\[1217390912\]th\_listen: CLSCLISTEN failed clsc\_ret= 3, addr= \[(ADDRESS=(PROTOCOL=ipc)(KEY=procr\_local\_conn\_0\_PROL))\]

2010-06-29 10:31:01.571: \[  OCRSRV\]\[3267002960\]th\_init: Local listener did not reach valid state

In Grid Infrastructure cluster environment, ohasd related socket files should be owned by root, but in Oracle Restart environment, they should be owned by grid user, refer to "Network Socket File Location, Ownership and Permission" section for example output.

[5f60efcaccfa40b26873a6a9cb266119.gif]: /images/20221021/ff3f245a66024c688952b620ce87572c.png