Oracle database client installation "The java.library.path system variable is missing or invalid"

As I heard it more than once in the last months, it is the right time to write a blog post about it. I still don't know, why it happens so often at the moment (except of a lack of reading the documentation), but I know, how it will (hopefully) work for you, if you run in this problem (without reading the documentation).

The typical environment where this happen is a Red Hat Linux (mostly 8.x and 9.x), but it can happen on any Linux distribution.

The customers of mine all tried to install a 19c 64-bit Client on a Red Hat Linux system as the installer stopped somewhere with this message:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall20XX-XX-XX_XX-XX-XXPM. Please wait ... The java.library.path system variable is missing or invalid. Please set java.library.path with a correct value and retry the operation.

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader

First to check is always, if the java environment is set up/available correctly, but typically, it is. Second is to check the documentation for the operating systems requirements of the product you want to install - in case of 19c clients on Linux, the documentation can be found here.

In the above case, the error message  "The java.library.path system variable is missing or invalid" is just wrong. Java is fine and available and the java.library.path is correct.

In this case and with the documentation you can then check the supported (kernel) version and afterwards the Linux system for the required packages, this is e.g. the list for Red Hat Linux 8.x:
bc, binutils, elfutils-libelf, elfutils-libelf-devel, fontconfig-devel, glibc, glibc-devel, ksh, libaio, libaio-devel, libXrender, libX11, libXau, libXi, libXtst, libgcc, libnsl, librdmacm, libstdc++, libstdc++-devel, libxcb, libibverbs, make, policycoreutils, policycoreutils-python-utils, smartmontools, sysstat.

There is also a section in the documentation which allows to check the optional packages required for some products, e.g. ipmiutil (for Intelligent Platform Management Interface), libnsl2 (for Oracle Database Client only), libnsl2-devel (for Oracle Database Client only), net-tools (for Oracle RAC and Oracle Clusterware), nfs-utils (for Oracle ACFS).

Now, what happend to all the customers with the above error message while installing the client? As we found out step by step, one of the required packages were not installed as X86 64bit version: The libnsl. But this isn't told by the installer - it complains about the java.library.path system variable.

All customers just installed libnsl.x86_64 as newest version using 

yum install libnsl.x86_64

and then they retried the installation of the client succesfully. All other packages, except libnsl, were setup with the Red Hat installation, only libnsl was missing.

If you run an Oracle Enterprise Linux you can use the preinstall-rpm for the database (oracle-database-preinstall-19c) - by the way even if you only want to install the client - to get all the libs installed automatically. If you are on Red Hat, you really should check the libraries according to the documentation.

Yes, it is a (mostly) stupid work to compare libs, but if it prevents from a whole day searching for an error with a misleading error message... 

Starting an Oracle Database Appliance (ODA) using a ssh connection to ILOM (with/without SSH Keys)

The question is - why does someone want to do this? Well, the business case behind this is a shortage in the power supply of a company. They do have a small battery running for some minutes which is enough time to shut down all servers. After a couple of minutes, a diesel emergency generator should have took over the power supply until the public power supply is back in a stable condition. 

If this status is reached, all servers should be restarted automatically by a software - in the right order! This means, first all infrastructure servers (like AD, exchange, DNS, ...) are started, next the database servers and last, after some minutes of wait to bring up all databases, the application servers can start. 

The customer therefore asked for a possibility to stop and start the Oracle Database Appliance using ssh and some kind of script. Well, shutting down the server is easy - one can just follow the note from Oracle Support (Note 2479508.1) for a graceful shutdown and power off of the ODA - as long as there is enough time on the battery left to do this. This may can run several minutes.

But there is no chance to connect to the ODA itself and start the server up. This means, one needs to use an ILOM connection to the ODA to start it (and in case of emergency, one can also shut it down using the same approach, even if it is not recommended and may can lead to OS file system corruption in rare cases). 

The steps:

Optional Step 1 - Create a new user

In the ILOM UI at "ILOM Administration" use "User Management" to create a new user account. The new user should get the rights "Reset and Host Control" and "Read only" (ro). If one want to use the connection as root user, one can skip this.

 

 

 

As example, I created a user "bootman" (Boot Manager).
Afterwards, your User Management looks like

 

 

The following steps do work also with another user as "root", but I have tested it with "root", so all screenshots later will only contain the root user.

 

Optional Step 2 - Create Keys for passwordless connections to ILOM 

One can create the keys for the ssh connection e.g. with the ssh-keygen on Linux. Nothing special at all.

 

 

Check the id_rsa.pub file after creation and copy it to the local computer (for later upload).

 

Optional Step 3 - Add Keys for passwordless connections in ILOM

The certificate now must be installed at the ILOM. Again, one uses the "ILOM Administration"-"User Management" with the "User Accounts" Tab to add the SSH Key. 

Now one can specify the user and the id_rsa.pub file for the passwordless connections.

After uploading the file, one can check the successfull upload. There is an entry at the SSH Key section. The user name column corresponds to the user specified (root or the new created user like "bootman"). 

Step 4 - Connect to the ILOM

If one has done Step two and three the connection can just be done using ssh <ilom_user>@<ilom-ip-address/ilom-dns>. If there is no passwordless ssh connection configured, the password for the <ilom_user> must be entered/specified at the ssh connection. 

 

Step 5 - Stop/Start the System via ILOM

Unfortunately, if one want to stop/start the ILOM, a question is raised. "Are you sure you want to start/stop /System (y/n)?". This can be overwritten by using the "-script" parameter. 

 

From a batch or second system, it is now possible to use a "one-liner" to shutdown or startup the ODA. With SSH Keys it can be done without specifying any password.

 

By the way - if it is enough to power on the ODA if the power is established again after a power loss, one can set the ILOM parameter "HOST_AUTO_POWER_ON"  to enabled.

Nice.