I was working at a customer database environment, setting up nearly 30 new databases with oracle 12.1 and 12.2 Standard Edition and Enterprise Edition version. For the datafiles we used ASM with 12.2 Grid Infrastructure for Single Instances (with Restart) and the newest version we got(April 2018).
One of the things I was really going mad was, that some entries at oratab were gone after stopping and/or starting the database with srvctl or by rebooting the nodes. .
First I was thinking about a bug at ASM, because there was also one at 12.1.
But after a while I found a nice note at https://support.oracle.com which states, that this is a new default behavior and not a bug.
In detail, starting with 12.2.0.1.171017 GI RU/PSU (or any version later), the oratab entries are removed automatically, if they have been "added by agent".
The note "Applying 12.2.0.1.171017 RU (patch 26737266) To 12.2 Cluster Removes Oratab Entries (Doc ID 2329359.1)" tells us, that "The agent deletes the oratab entries that have comment "# line added by Agent". For example, when the GI is recycled (including after a node reboot), then the agent will remove the oratab entries that have the comment "# line added by Agent"".
Unfortunately, it was not enough to just remove the comment "# line added by Agent", like it is written at the document - we needed to put own comments "# manually added" at the end of the lines.
Why is Oracle doing this as new default behavior? "With the introduction of the flex ASM (asm may not run on all nodes) and policy managed databases (no instances is tied to a server), the mapping of the asm and RAC database instances to a specific node is no longer guaranteed to be valid, so this is the reason that the oratab entries are being removed."
So if you run into this problem with GI version 12.2 or later (I think this will also happen with Oracle Database Grid Infrastructure 18c) and you want to keep the entries, just add your own comment at the end of every line you want to keep.
If you don't know it, you are going mad, if you know it - life can be easy.
P.S. If you need to find the names of the databases, this blog entry could be helpful: https://fritshoogland.wordpress.com/tag/recreate-oratab/
Excellent article that helped me solve this problem in our latest 12.2.0.1.180717 upgraded databases.
ReplyDeleteWe add comments to help us. I cannot believe that a very basic comment like "# line added by Agent"
can delete oratab entries for 12.2 databases. Oracle has easy ways of saying "This is expected behavior"
but, in the IT world, this is NOT an expected behavior with simple comments. I am surprised Oracle
developers can use comments to manipulate entries in files !!!
Great article. Also wanted to mention that dbshut and dbstart scripts in oracle_home/bin directory take the last parameter of the oratab line which is
ReplyDeleteN # line added by Agent
The scripts don't work anymore without modification. Manualy removing comment in oratab help until database restart and oratab gets restored which is apparently a "feature".
Thanks for your comment, Rok. Dbshut and dbstart are deprecated since 11g R2 I think,so the better solution is to use Oracle Restart.
DeleteHi,
ReplyDeleteI tried adding a different comment at the end of each lines but lines still get modified after a reboot.
Before reboot :
+ASM2:/u01/app/12.2.0.1/grid:N # RQ
AX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
AX42:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
AX62:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
DX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
DXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
EX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
EXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
FX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
FXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
PX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
TVX2:/u01/app/oracle/product/12.1.0.2/dbhome_1:Y # RQ
UNXA:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UX42:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UX62:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UXA2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
After reboot :
AX42:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
AX62:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
DXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
EX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
EXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
FXX2:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
TVX2:/u01/app/oracle/product/12.1.0.2/dbhome_1:Y # RQ
UNXA:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
UX42:/u01/app/oracle/product/12.2.0.1/dbhome_1:Y # RQ
AX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
FX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
DX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
UX62:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
PX12:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
UXA2:/u01/app/oracle/product/12.2.0.1/dbhome_1:N # line added by Agent
How can I fix this ?
Regards,
Well, it is not so easy to contact you, if you don't leave an email address or a name...
DeleteYou may can try with a longer comment, I have seen that (but it was not possible to re-test this with every RU/PSU) short own comments are sometimes not working (I don't know why).
What I do wonder is, why the oratab entries have changed from "Y" to "N" only due to reboot and what happened with +ASM2.
Frits is having a solution, how to use a script to build up the ORATAB from cluster registry, maybe this can help you. (https://fritshoogland.wordpress.com/2014/06/19/reconstructing-oratab-from-the-cluster-registry/)
Wondering if added longer comments will work ? @UnknownJuly 20, 2020 at 8:08 PM
ReplyDeleteJust give it a try - and if you have find out, with how many characters minimum it works, please post it here.
Delete