Showing posts with label Oracle Database Appliance. Show all posts
Showing posts with label Oracle Database Appliance. Show all posts

ODABR - Usage update for the Backup and Restore Utility for Patching Oracle Database Appliances

 ODABR (just like purgelog) is probably familiar to anyone who works with Oracle Database Appliances (ODA) or applies patches to them regularly. Until recently, the odabr tool was downloaded via this link from Oracle Support and installed as a RPM.

Newer versions had to be manually copied to the ODA, and the RPM had to be updated accordingly on the ODA. 

The ODABR tool then allowed users to manually create backups of the system areas and provided the option to restore them or use the backup to reset the system areas.

Presumably, almost everyone used the tool this way before applying the patch: they created a backup as a snapshot using ODABR, and then manually deleted that snapshot using ODABR after applying the patch.

If an error occurred on the ODA during patching, one was able to restore the backup using 'odabr restore --snap' and rebooting the ODA - which restored the system area (in very rare cases where booting was no longer possible, on had to switch to the rescue disk, but that has never happened to me personally) and one had a working ODA again.

Starting with Release 19.30, the odabr backup and restore utility is now an integrated part of the ODA installation itself and is installed automatically.

As one can see this from the log, the system checks whether odabr is already installed on the ODA and, if so, which version is installed.

Log

Since ODABR is now always available on the server, Oracle has also updated the syntax and checks for odacli accordingly.
Starting with version 19.30, when one generates the prepatch report or perform the upgrade for the server components, there is now a new switch which can be used: “-rf” or “--rollback-onfailure.” This switch ensures that the ODABR utility is automatically checked and used during a patch.
So there are now two different approaches: 
The "classical" way:
1. Create an ODABR snapshot manually
2. Precheck and patch the server components
3. Delete the ODABR snapshot manually

The new approach (example with version 19.32.0.2.0) is:
1. odacli create-prepatchreport -sc -v 19.32.0.2.0 -rf 
2. odacli update-servercomponents -v 19.32.0.2.0 -rf

Let's have a look behind the curtain of the jobs and check the job output. 
When generating the prepatch report, the prepatch check may fail while verifying the rollback behavior.


Why did this happen? The image below (details for the Pre-Checks) shows that one has already created the odabr snapshot here (presumably out of habit or as a safety precaution). This causes the check to fail. When running the command with the “-rf” switch, no snapshot should be present.


The manually created snapshot needs to be deleted. Afterwards, the Pre-Checks all succeed. 


If the "-rf" switch was used during the prepatch check, it must also be used when updating the server components! If one misses that, the patching is done without a valid ODABR backup!

Using "-rf", a separate snapshot is created right at the start of the update. 
 

The patching procedure is checking the validity of the snap, which can be found at the log:


After the upgrade has been completed successful, the backup snapshot is deleted automatically and the server reboots a minute later.


This would mean - in theory - a scenario is possible in which an error is only detected AFTER the reboot. However, since the snapshot has already been deleted, a restore would no longer be possible at this stage. With the traditional, manual approach, you can decide for yourself when the snapshot is deleted. Typically, this always happens only after the reboot - nobody deletes a snapshot while the backup job is running (but one could forget to delete the snapshot).

This raises the question of the usefulness of the new switch—who trusts the update mechanism and uses the "-rf" switch, and who is more type "safety first" and sticks with the traditional use of ODABR? Both approaches benefit from the tool's automatic installation. Both approaches are possible, but one should never try to mix them. Use the approach you like and stick on it. If you use "-rf" only on the prepatchreport but don't specify it for the upgrade, you don't have an odabr snapshot!












Install RLWRAP on Oracle Database Appliance without internet connection

 If you need to install RLWRAP on an Oracle Database Appliance (ODA) with actual releases (19.26-19.28 ) you need to install the following rpms. As we use rlwrap a lot also in our Robotron DBA tool set, we install it nearly everywhere. 

While it is easy to install rlwarp with a working internet connection and the ODAs yum/dnf repository configuration, most of our customers don't allow the ODAs access to the public internet (except of ASR). 

Here is the list of the rpms you need to install (ordered) from the public yum base repository:
- ncurses-c++-libs: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/ncurses-c++-libs-6.1-10.20180224.el8.x86_64.rpm
- ncurses-devel: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/ncurses-devel-6.1-10.20180224.el8.x86_64.rpm
- readline-devel: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/readline-devel-7.0-10.el8.x86_64.rpm
- readline: should be already installed, if not: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/readline-7.0-10.el8.x86_64.rpm
- m4: https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackage/m4-1.4.18-7.el8.x86_64.rpm

The following must be installed using the rpms from the Appstream folder. 
- autoconf: https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackage/autoconf-2.69-29.el8_10.1.noarch.rpm
- automake: https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackage/automake-1.16.1-8.el8.noarch.rpm
- perl-file-slurp: https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackage/perl-File-Slurp-9999.19-19.el8.noarch.rpm

rlwrap can either be installed from oracle yum epel oder fedoraproject repository
- rlwrap from oracle yum epel : https://yum.oracle.com/repo/OracleLinux/OL8/developer/EPEL/x86_64/getPackage/rlwrap-0.46.2-3.el8.x86_64.rpm
- rlwrap from fedoraproject: https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/r/rlwrap-0.46.2-3.el8.x86_64.rpm  

If you install rlwrap from the fedoraproject und you get an error "GPG check FAILED" you need to download the public key RPM-GPG-KEY-EPEL-8  from https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8, save it, e.g. as RPM-GPG-KEY-EPEL-8.txt and import it with "rpm --import RPM-GPG-KEY-EPEL-8.txt" before installing the rpm.
  
If you patch an ODA you might get an error at the prepatchreport for the server. If so, uninstall automake, autoconf and m4 with dnf remove <package> and re-install them after you finished the ODA patch (server and storage). 
  
If you download everything locally and upload/put all rpms into the /tmp folder of the ODA, you can run dnf install /tmp/<name-of-the-rpm> and install them one after the other. 

Oracle Database Appliance Patch Update error when default gateway is not pingable - DCS-10001 and PRGO-1664

When setting up an Oracle Database Appliance (ODA) the configuration of the network with odacli configure-firstnet does check, if the default gateway is pingable. If it is not, the configuration is not possible. 

But one of our customers (or better the network administrators) later configured the environment (router), that pings are not responded, which means, they time out (this is part of the security strategy of this company). The network does work, but the ODA can not ping the default gateway anymore (no one can).

As we tried to patch an ODA from ODA Release 19.21 to 19.24 we therefore run into an ugly error while the Grid Infrastructure was upgraded from 19.21 to 19.24, leaving us with two unzipped home directories:

"DCS-10001:Internal error encountered: Failed to set ping target on public network."

If you take a look into the dcs-agent.log, you can find a more detailled error message:

DEBUG [Server patching : JobId=<something>] [] c.o.d.a.r.s.g.GiOperations: ping failed: unknown gateway <IP-Address of Gateway>

The creation of the prepatch report for the server was fine ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.24.0.0.0). It finished successful without error, so we didn't expect that we run into an error later on. 

But while running the server patch ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-server -v 19.24.0.0.0) we got the above error. 

We asked the network team to fix the ping problem, tested the ping manually and restarted the odacli update-server -v 19.24.0.0.0 a second time. Sometimes, when an ODA update stucks, it is fine to just restart the update-server again, but this time we got another error:

 “DCS-10001:Internal error encountered: Failed to patch GI with RHP : DCS-10001:Internal error encountered: PRGO-1664 : The specified source working copy \"OraGrid192100\" is a software-only working copy...”

Thanks god we did an "odabr backup -snap" before we started the update and by running "odabr restore -snap -force" we were able to restore the "before" image of the operating system stuff. If you start odabr restore without the force option, you will get a warning "Clusterware patchlevel at backup time is not what you have now. ODABR can't be considered a tool to perform Grid Infrastructure downgrade" and the snap will not be restored.

Unfortunately, the grid infrastructure is not part of the odabr backup with newer ODA software releases, which means, we still had two Grid Infrastructure homes and the problem with the working copy as a software only working copy. 

Searching in support.oracle.com does not really help, searching for PRGO-1664 does not result in useful hits. But there is a small section in the note for the odabr tool (2466177.1). "if a GI update was in progress and you have done a restore", it may happen that the executable permissions are set wrong. 

To fix this, set the GRID_HOME to the old grid environment as user root and run [root@oda1]# GRID_HOME/crs/install/rootcrs.pl -lock

This will change the executable permissions - in our case in the 19.21 grid directory back and after this change patching including the odacli update-server -v 19.24.0.0.0 was running fine. 

Depending on the time you did the odabr backup, you need to start with patching the dcs stack or running the prepatch-report creation again. 

For the next ODA software versions, there is an enhancement request placed for the development, so the gateway ping check is then part of the prepatch report and will not lead to a corrupted GI update.

In the meantime it could be helpful to test manually, if the default gateway is pingable. To find out the gateways ip address you can run a "ip route show" command. It outputs a line "default via <IP Address>". This ip address is your gateway - a "ping <IP Address>" will (hopefully) work. If so, you will not run into this problem, we got. We will add the ping test to our manual check list, it is a 10 second work and it will save an hour work later on.


 

Starting an Oracle Database Appliance (ODA) using a ssh connection to ILOM (with/without SSH Keys)

The question is - why does someone want to do this? Well, the business case behind this is a shortage in the power supply of a company. They do have a small battery running for some minutes which is enough time to shut down all servers. After a couple of minutes, a diesel emergency generator should have took over the power supply until the public power supply is back in a stable condition. 

If this status is reached, all servers should be restarted automatically by a software - in the right order! This means, first all infrastructure servers (like AD, exchange, DNS, ...) are started, next the database servers and last, after some minutes of wait to bring up all databases, the application servers can start. 

The customer therefore asked for a possibility to stop and start the Oracle Database Appliance using ssh and some kind of script. Well, shutting down the server is easy - one can just follow the note from Oracle Support (Note 2479508.1) for a graceful shutdown and power off of the ODA - as long as there is enough time on the battery left to do this. This may can run several minutes.

But there is no chance to connect to the ODA itself and start the server up. This means, one needs to use an ILOM connection to the ODA to start it (and in case of emergency, one can also shut it down using the same approach, even if it is not recommended and may can lead to OS file system corruption in rare cases). 

The steps:

Optional Step 1 - Create a new user

In the ILOM UI at "ILOM Administration" use "User Management" to create a new user account. The new user should get the rights "Reset and Host Control" and "Read only" (ro). If one want to use the connection as root user, one can skip this.

 
 
As example, I created a user "bootman" (Boot Manager).
Afterwards, your User Management looks like

 
 
The following steps do work also with another user as "root", but I have tested it with "root", so all screenshots later will only contain the root user.

 

Optional Step 2 - Create Keys for passwordless connections to ILOM 

One can create the keys for the ssh connection e.g. with the ssh-keygen on Linux. Nothing special at all.
 
 
Check the id_rsa.pub file after creation and copy it to the local computer (for later upload).

 

Optional Step 3 - Add Keys for passwordless connections in ILOM

The certificate now must be installed at the ILOM. Again, one uses the "ILOM Administration"-"User Management" with the "User Accounts" Tab to add the SSH Key. 

Add SSH Key in ILOM

Now one can specify the user and the id_rsa.pub file for the passwordless connections.

Select User and pub ssh key file

After uploading the file, one can check the successfull upload. There is an entry at the SSH Key section. The user name column corresponds to the user specified (root or the new created user like "bootman"). 

Check SSH Key upload

Step 4 - Connect to the ILOM

If one has done Step two and three the connection can just be done using ssh <ilom_user>@<ilom-ip-address/ilom-dns>. If there is no passwordless ssh connection configured, the password for the <ilom_user> must be entered/specified at the ssh connection. 
Connect to ILOM using SSH

 

Step 5 - Stop/Start the System via ILOM

Unfortunately, if one want to stop/start the ILOM, a question is raised. "Are you sure you want to start/stop /System (y/n)?". This can be overwritten by using the "-script" parameter. 
 
Stop/Start System using -script parameter

From a batch or second system, it is now possible to use a "one-liner" to shutdown or startup the ODA. With SSH Keys it can be done without specifying any password.
 
Starting ODA using SSH one line

By the way - if it is enough to power on the ODA if the power is established again after a power loss, one can set the ILOM parameter "HOST_AUTO_POWER_ON"  to enabled.

Nice.