Creating KVM network bridge on ODA - Not able to connect to database anymore.

A lot of people are using Oracles KVM solution on ODA (Oracle Database Appliance). My company e.g. runs something like application servers in a Linux VM on the ODA lite models for our own software solutions (like communication server, etc.) if a customer runs ODAs (we call it "Solution-in-a-Box"). But there are also other customers, where we just act as system integrator, who want to use KVM on the ODA.

There is a really nice blog how to enable and use KVM on ODA, the starting point is this blog post by Tammy: kvm-on-oda.

It is straight forward, but one of the things are a little bit crucial. It is how to make the network configuration for the KVM on ODA. The best solution is to BRIDGE the network. Ruggero has written a blog post as part of Tammys blog, how you can enable all the different network types.
Don't use NAT or MacVTap - just follow the configuration steps for "Bridged networking (aka "shared physical device")".

Be sure you have access to the iloms host redirection function, because if there is any misconfiguration with the bridge, you lose the network connectivity und you are not able to connect internal (like you could do with ODA HA).

If you have followed the configuration steps and if you can connect to the ODA host again with the bridge configuration, you are not able to connect to the database(s) on that host anymore. Why? Because one mandatory step is missing at Ruggeros example: The configuration of the clusterware (as grid user)! Some tried to stop and start the listener, but the listener will not be started and errors out.

What you need to do as a last configuration step is to modify your clusterware network configuration. The listener is bound to Network 1, which can easily seen by issueing

$ srvctl config listener
Name: LISTENER
Type: Database Listener
Network: 1, Owner: grid
Home: <CRS home>
End points: TCP:1521
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:

To check the configuration, run the following command:
$ srvctl config network
Network 1 exists
Subnet IPv4: 10.214.0.0/255.255.248.0/btbond1, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

As you can see, the network is still configured to btbond1 instead of pubbr0.
The syntax (I use the same subnet in my example as Ruggero in his blog) to change this is:
$ srvctl modify network -netnum 1 -subnet 10.214.0.0/255.255.248.0/pubbr0 

Now it is best to either restart the ODA to check if everything is running fine also after an reboot or at least to restart the listener.
$ srvctl stop listener
$ srvctl start listener

Then you are done folks, you can use the bridged device now for your KVM and the databases are reachable again.

How to deploy ODA lite models with (active-active) VLAN configuration easily

This blog entry is a description of deploying an Oracle Database Appliances X6-2 S in a VLAN environment, but it is valid for all models of the X6-2 and X7-2 lite versions (X6-2 S, X6-2 M, X6-2 L, X7-2 S, X7-2 M). The release of the ODA software we used to deploy was 12.1.2.12 - the newest version at the time of the installation.

Re-imaging the ODA

The easiest way to start with the deployment (if the delivered machine does not already have the latest version installed) is to just re-image the ODA following the instructions that can be found at the manual for this software version. So just configure the ILOMs and start with re-imaging. You can also use the already installed ODA software and upgrade it to the newest release. Then you do test the different steps to upgrade an ODA, but as there are more steps included it's easier to re-image and the time used for both approaches doesn't differ much.

Preparation

After re-imaging, the VLAN creation can take part. We have taken the fiber bond to configure our VLAN, but one can use this article also to create VLANs using copper (then use btbond1 or btbond2 instead of sfpbond1). We didn't used the copper interface for our deployment, but on ODA X6-2 machines both network interfaces can be used in parallel (on X7-2 you have to decide either using copper OR fibre).. 
To create a VLAN, Oracle kindly provides a script called ODA_vlan_config.sh which can be found at the support.oracle.com note 2216567.1.
At this note, at the manual and even on the ODAs itself one can also find a template of a json file, which is needed to deploy the ODA. VLAN is supported on the ODA light models (since software release 12.1.2.8), but the graphical deployment user interface doesn't work in VLAN environments.
Taken the provided json template file one has to change everything that is needed to deploy the ODA silent for the own environment. We saved the json template as file /tmp/create-appliance-ODA.json before we changed the entries.
The points to change are especially the name of the instance/database, the database edition, if one want to use standard edition (actually you can't mix SE and EE deployments on the same machine), the time zone, all ip-addresses and domain names. Do NOT change the nodeNumber as it is always "0" on the ODA light models S and M. To use the right VLAN for the public interface the nicName must be changed to match the VLAN configuration.

"instance" : {
"name" : "ODA1",
"instanceBaseName" : "PROD",
"dbEdition" : "EE",
"timeZone" : "UTC",
"ntpServers" : ["10.0.3.14"],
"dnsServers" : ["10.0.4.10","10.0.4.11","10.0.4.12"],
"domainName" : "example.com",
 ...
"nodes" : [ {
"nodeNumber" : "0",
"nodeName" : "ODA1",
"network" : [ {
"nicName" : "sfpbond1.200",
"ipAddress" : "10.24.128.243",
"subNetMask" : "255.255.255.0",
"gateway" : "10.24.128.1",
"networkType" : [ "Public" ],
"isDefaultNetwork" : true

One needs to remove the whole ILOM tree from the json file as the ODA deployment otherwise can fail (and we did had the right names / ip addresses put in there but there may be a routing problem between the different VLANs and the management network). Also one needs to remove the Auto Service Request "asr" tree from the json file, if the ASR should be configured later on. If a non existing account is used at the ASR part or the user/password combination is not right, the whole deployment fails and one needs to start over again (undeploying the ODA first and re-start with the deployment). 
If needed, the disk group redundancy can be changed (NORMAL equals mirroring, HIGH equals triple mirroring which is not available with only 2 NVMe cards) and the disk percentage. The documentation tells:

Storage Configuration Options
When Oracle Database Appliance X6-2 is deployed, you can select one of the following configuration options to divide the storage capacity between DATA diskgroup and RECO diskgroup:

• External: Storage capacity is split between 80% for DATA and 20% for RECO.
• Internal: Storage capacity is split between 40% for DATA and 60% for RECO.
• Custom: Storage capacity is configurable from 10% to 90% for DATA and the remainder for RECO.

Therefore, the deployment was done with a 50% split between DATA and RECO as we needed some kind of "internal" for storing the FRA (backups, archivelogs, flashback logs) locally on the ODA but kept some more space for "DATA".

Do not forget to change the entries inside the database tree at the json file. dbType can only be single instance on ODA lite models, but everything else can be set. The admin password is not only used by the database, but is set as the new "master password", e.g. also for the oracle user at linux (everything can be set after the deployment, you don't need to keep a master password for all).
 
Enable dbConsole if you want to use Enterprise Manager Express on 12c! Provide the right database shape you want to use (db shapes documentation). Keep ACFS as dbStorage, so you are able to use e.g. standard backup agents to backup files from the disks. The character set can't be changed later on, so it is also needed to change it before the deployment starts.

"database" : {
"dbName" : "PROD",
"dbVersion" : "12.1.0.2",
"instanceOnly" : false,
"isCdb" : false,
"pdBName" : "pdb1",
"pdbAdminuserName" : "pdbuser",
"adminPassword" : "welcome1",
"dbType" : "SI",
"dbTargetNodeNumber" : "0",
"dbClass" : "OLTP",
"dbShape" : "odb2",
"dbStorage" : "ACFS",
"dbCharacterSet" : {
"characterSet" : "AL32UTF8",
"nlsCharacterset" : "AL16UTF16",
"dbTerritory" : "AMERICA",
"dbLanguage" : "AMERICAN"
},
"dbConsoleEnable" : false
}

 

Creating the VLAN

After finishing the preparation, the creation of the VLAN can be done. In our deployment we have created two different VLANs on the spfbond1.  So we connected as root and provided the vlan, the bondname, the ip address, network mask and the gateway as parameters to the script we have downloaded from the support.oracle.com note.

./ODA_vlan_config.sh add -v 3001 -i sfpbond1 -ip 10.24.30.243 -n 255.255.255.0 -g 10.24.30.1
./ODA_vlan_config.sh add -v 200 -i sfpbond1 -ip 10.24.128.243 -n 255.255.255.0 -g 10.24.128.1

In our case, we needed to change the bond to run active-active instead of active-failover (the default):
vi /etc/sysconfig/network-scripts/ifcfg-sfpbond1
BONDING_OPTS="mode=8.2.3ad miimon=100 lacp_rate=0" 

If one has used configure-firstnet, it may is necessary to remove ip-address, netmask and gateway from the icfg-sfpbond1 file. 
After the configuration has been changed the network service and the dcs agent must be restarted:

service network restart
initctl stop initdcsagent
initctl start initdcsagent 

Best practice is to check  the network configuration afterwards with:
odacli list-networkinterfaces

#/opt/oracle/dcs/bin/odacli list-networkinterfaces
#
#ID Name NIC Type
#---------------------------------------- -------------------- ------------- #----------
#b324f474-627e-4b15-af83-2ed21eda845a btbond1 btbond1 Bond
#595dfda9-67a6-4fc9-96af-afe966c287a7 em1 em1 Physical
#2033de33-99fb-4d53-8235-6ef088db54fe em2 em2 Physical
#62a09165-d9a5-4b8d-8eaa-45022988215b em3 em3 Physical
#40070762-6e87-42b4-8eb2-b8f9aa642849 em4 em4 Physical
#d70bbb83-e7de-462a-8892-42bdb687d6dc sfpbond1.3001 sfpbond1.2001 Bond
#26a0c940-5705-460a-b51d-ab8723d06f35 priv0 priv0 Dummy
#49cf2709-3c75-4c2c-bf53-718757385335 sfpbond1.200 sfpbond1.200 Bond

Deploying the ODA

As the VLANs are now fully functionable, one can start the deployment (still connected as root):
/opt/oracle/dcs/bin/odacli create-appliance -r /tmp/create-appliance-ODA.json

As you can see, the previously created json file is used as an input parameter for the odacli create-appliance statement. After waiting a couple of minutes, the ODA is now deployed with spfbond1.200 as standard network interface.
So one is now able to use both VLANs for different tasks. As the listener is configured by default only with the VLAN 200 (specified at the json file), one needs to configure the second ip-address as listener endpoint manually, if needed (ssh as grid user and run netmgr). So in our case the public interface for the application is the VLAN 200 and the private interface for the data guard and the backup is VLAN 3001. 

Conclusion

Even if the deployment with the GUI is not working for VLAN environments, the ODA lite models using bare metal installation without OVM (so called virtualized platform) can easily be deployed with VLAN configuration and, if you keep in mind to remove the entries from the json files you don't need, it is fast and straightforward and works on from ODA light software release 12.1.2.9.0 (until at least 12.1.2.12, but I think this is also available for later software releases). If needed, configure additional network things (like permanent routes and rule sets in /etc/sysconfig/network-scripts).