How to start preparing your database for the upgrade to Oracle database 23ai - even without an on-premises released database software!

 As many of you already know, Oracle's autoupgrade tool will be the only supported option for database upgrades in the future. This fact can also be found in the support notes of the 23ai database.

However, the autoupgrade tool - and most people don't know this - already supports the upgrade to Oracle database version 23ai in Release 24 (available since roundabout March of this year). The question therefore arises, how can you TODAY use the autoupgrade tool to optimally prepare your existing databases for the upgrade to 23ai ? Especially now, when the perhaps quiet time of the year is approaching, but the official release date of 23ai has been postponed to “sometime in calendar year 2025” (postponed once again)?

The preparation

First of all, the latest version of the autoupgrade.jar must be downloaded from Oracle Support and can be found at Document ID 2485457.1.
This is currently a build from October 2024 (24.7.241021). The autoupgrade.jar can next be copied to the server and tested for functionality. If no Java is installed on the server, the JDK from the highest Oracle Home release can be used. This has always worked so far.
Oracle's support for direct upgrades to 23ai is limited to 19c and 21c databases, this means older database versions cannot be prepared for a direct upgrade to 23ai using the autoupgrade tool. An intermediate upgrade (to 19c) is therefore necessary beforehand or, for example, data migration with Datapump Export-Import is a possible way to move from older database versions to 23ai. However, no pre-test can be one with datapump.

If the execution of autoupgrade.jar (java -jar autoupgrade.jar -version) works on the server, the next step is to create a config file for the database that is to be prepared for the 23ai upgrade.

The following information is important here:

• the log directories (global and per DB)
• Which SID should be tested/analyzed
  
• The source Oracle Home - this needs to be a 19c or 21c database home
• The target Oracle Home - is set, but this directory does not exist - the database is not yet available on-premises
• Start_time and upgrade_node can be set by default (NOW/localhost)
• Important - the autoupgrade tool normally takes the target_version from the target home. However, as this does not exist, it is important to set 23 fixed as target_version here.

 

 

Config file

 Analyzing the existing database

Once this preparation has been completed, the autoupgrade tool can now be run in Analyze mode against the source database. NO changes are made to the source database, this means this procedure is completely safe.
However, it is important that all pluggable databases (PDB) are open. PDBs that are in mount status are not analyzed.

If all PDBs are open, there is no longer a warning and the corresponding autoupgrade job starts analyzing the existing database.

The “status” command can be used to display the progress of the analysis of all jobs on the Autoupgrade CLI. It shows 4 databases (1 CDB + Seed PDB + 2 user PDBs).

To see the details of the individual databases, just add the job with number to the “status” command. Now one can see the stage progress per container (the 4 databases).

The job should be successfully completed after a few minutes. There are typically no errors in the analysis phase.

The analysis result (in various formats, including JSON, text and HTML) is now available for the analyzed database in the Prechecks subdirectory of the log directories for the job. The next picture does show the HTML.

The first overview already shows that there are components in the 19c database that would be removed during an upgrade to 23ai. These are remnants of Oracle Multimedia - marked yellow.

If one now looks in the detailed areas of the databases (CDB and PDBs), one will find valuable information on how to deal with such changed or removed components. The Autoupgrade tool recommends taking a look at various support notes, for example.

In the database analyzed here (which was already upgraded from a 12c environment), the autoupgrade tool also found a configuration of traditional auditing. Traditional auditing was first supplemented by Oracle some time ago with Unified Auditing and then replaced - or is now officially desupported for the first time, so that Unified Auditing must be switched to.
As the configuration of auditing can be a very time-consuming process, it is advisable to adapt the configuration on the existing database even before the upgrade planning officially starts after 23ai.

The autoupgrade tool may already find things as part of the POST check, things which then need to be  done on the 23ai database, such as changing the LOB type at an audit table. Of course, this cannot yet be prepared in the existing environment, but it does provide an indication of any manual work that may need to be done (and maybe any additional downtime) during a later upgrade.

Incidentally, even though the screenshots are now all from a Linux environment, the analysis method was also tested with a 19.8 database under MS Windows. It should therefore be possible to use this procedure on all supported operating systems.

Now start prepare your database for a successful upgrade to database version 23ai.

Oracle database client installation "The java.library.path system variable is missing or invalid"

As I heard it more than once in the last months, it is the right time to write a blog post about it. I still don't know, why it happens so often at the moment (except of a lack of reading the documentation), but I know, how it will (hopefully) work for you, if you run in this problem (without reading the documentation).

The typical environment where this happen is a Red Hat Linux (mostly 8.x and 9.x), but it can happen on any Linux distribution.

The customers of mine all tried to install a 19c 64-bit Client on a Red Hat Linux system as the installer stopped somewhere with this message:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall20XX-XX-XX_XX-XX-XXPM. Please wait ... The java.library.path system variable is missing or invalid. Please set java.library.path with a correct value and retry the operation.

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader

First to check is always, if the java environment is set up/available correctly, but typically, it is. Second is to check the documentation for the operating systems requirements of the product you want to install - in case of 19c clients on Linux, the documentation can be found here.

In the above case, the error message  "The java.library.path system variable is missing or invalid" is just wrong. Java is fine and available and the java.library.path is correct.

In this case and with the documentation you can then check the supported (kernel) version and afterwards the Linux system for the required packages, this is e.g. the list for Red Hat Linux 8.x:
bc, binutils, elfutils-libelf, elfutils-libelf-devel, fontconfig-devel, glibc, glibc-devel, ksh, libaio, libaio-devel, libXrender, libX11, libXau, libXi, libXtst, libgcc, libnsl, librdmacm, libstdc++, libstdc++-devel, libxcb, libibverbs, make, policycoreutils, policycoreutils-python-utils, smartmontools, sysstat.

There is also a section in the documentation which allows to check the optional packages required for some products, e.g. ipmiutil (for Intelligent Platform Management Interface), libnsl2 (for Oracle Database Client only), libnsl2-devel (for Oracle Database Client only), net-tools (for Oracle RAC and Oracle Clusterware), nfs-utils (for Oracle ACFS).

Now, what happend to all the customers with the above error message while installing the client? As we found out step by step, one of the required packages were not installed as X86 64bit version: The libnsl. But this isn't told by the installer - it complains about the java.library.path system variable.

All customers just installed libnsl.x86_64 as newest version using 

yum install libnsl.x86_64

and then they retried the installation of the client succesfully. All other packages, except libnsl, were setup with the Red Hat installation, only libnsl was missing.

If you run an Oracle Enterprise Linux you can use the preinstall-rpm for the database (oracle-database-preinstall-19c) - by the way even if you only want to install the client - to get all the libs installed automatically. If you are on Red Hat, you really should check the libraries according to the documentation.

Yes, it is a (mostly) stupid work to compare libs, but if it prevents from a whole day searching for an error with a misleading error message... 

Starting an Oracle Database Appliance (ODA) using a ssh connection to ILOM (with/without SSH Keys)

The question is - why does someone want to do this? Well, the business case behind this is a shortage in the power supply of a company. They do have a small battery running for some minutes which is enough time to shut down all servers. After a couple of minutes, a diesel emergency generator should have took over the power supply until the public power supply is back in a stable condition. 

If this status is reached, all servers should be restarted automatically by a software - in the right order! This means, first all infrastructure servers (like AD, exchange, DNS, ...) are started, next the database servers and last, after some minutes of wait to bring up all databases, the application servers can start. 

The customer therefore asked for a possibility to stop and start the Oracle Database Appliance using ssh and some kind of script. Well, shutting down the server is easy - one can just follow the note from Oracle Support (Note 2479508.1) for a graceful shutdown and power off of the ODA - as long as there is enough time on the battery left to do this. This may can run several minutes.

But there is no chance to connect to the ODA itself and start the server up. This means, one needs to use an ILOM connection to the ODA to start it (and in case of emergency, one can also shut it down using the same approach, even if it is not recommended and may can lead to OS file system corruption in rare cases). 

The steps:

Optional Step 1 - Create a new user

In the ILOM UI at "ILOM Administration" use "User Management" to create a new user account. The new user should get the rights "Reset and Host Control" and "Read only" (ro). If one want to use the connection as root user, one can skip this.

 

 

 

As example, I created a user "bootman" (Boot Manager).
Afterwards, your User Management looks like

 

 

The following steps do work also with another user as "root", but I have tested it with "root", so all screenshots later will only contain the root user.

 

Optional Step 2 - Create Keys for passwordless connections to ILOM 

One can create the keys for the ssh connection e.g. with the ssh-keygen on Linux. Nothing special at all.

 

 

Check the id_rsa.pub file after creation and copy it to the local computer (for later upload).

 

Optional Step 3 - Add Keys for passwordless connections in ILOM

The certificate now must be installed at the ILOM. Again, one uses the "ILOM Administration"-"User Management" with the "User Accounts" Tab to add the SSH Key. 

Now one can specify the user and the id_rsa.pub file for the passwordless connections.

After uploading the file, one can check the successfull upload. There is an entry at the SSH Key section. The user name column corresponds to the user specified (root or the new created user like "bootman"). 

Step 4 - Connect to the ILOM

If one has done Step two and three the connection can just be done using ssh <ilom_user>@<ilom-ip-address/ilom-dns>. If there is no passwordless ssh connection configured, the password for the <ilom_user> must be entered/specified at the ssh connection. 

 

Step 5 - Stop/Start the System via ILOM

Unfortunately, if one want to stop/start the ILOM, a question is raised. "Are you sure you want to start/stop /System (y/n)?". This can be overwritten by using the "-script" parameter. 

 

From a batch or second system, it is now possible to use a "one-liner" to shutdown or startup the ODA. With SSH Keys it can be done without specifying any password.

 

By the way - if it is enough to power on the ODA if the power is established again after a power loss, one can set the ILOM parameter "HOST_AUTO_POWER_ON"  to enabled.

Nice.