SOA/AIA Deployment Script (AID) issue in AIA Cluster environment with error/exception as ‘java.io.IOException’

Tried to deploy the code in AIA Cluster environment and getting the below error.

[UpdateDeployedComposites] Connecting to: service:jmx:t3://ADMIN_HOST:7001/jndi/weblogic.management.mbeanservers.domainruntime
[UpdateDeployedComposites] Connecting to: service:jmx:t3://SOA_HOST1:8001/jndi/weblogic.management.mbeanservers.runtime

BUILD FAILED
/mnt_fmw/app/oracle/middleware/aia/Infrastructure/Install/AID/AIAExecuteDriver.xml:119: The following error occurred while executing this line:
/mnt_fmw/app/oracle/middleware/aia/Infrastructure/Install/AID/AIAExecuteDriver.xml:62: java.io.IOException

In a SOA/AIA Cluster environment & AID assumes that all managed servers are up and running when executing the task UpdateDeployedComposites in order to determine if a composite is already deployed.

As AID is connecting to Admin server and tries to get the list of servers in SOA/AIA cluster environment (i.e. SOA servers). Then AID will connect to first  SOA server for the deployment.

If first SOA server in the cluster is in shut down STATE and as a result, will get this kind of  error on the deployment script.

On validating the UpdateDeployedComposites task in AID, it has the code designed to catch situations where some of the servers are down and to try another ones if it fails to connect to the first one. However, such exceptions are not caught and so AID stops without attempting to connect to other managed servers that are up and running.

As a temporary work around & solution, all the cluster members in the SOA/AIA cluster environment should be up & running and then try the deployment.

Any questions & comments are Welcome.

SOA 12c- Applying patch to SOA environment using opatch utility.

 1. Download the corresponding patch from Oracle My Oracle Support (MOS) site

 2. Extract into some stage directory. In my case, I have extracted to below directory.

• /opt/app/oracle/soa_bp_12.1.3.2/20423408
• This patch (number: 20423408) is for upgrade the SOA12.1.3.0 to 12.1.3.2 bundle patch.

 3. Follow the below instructions.

1. Stop all services which are associated to SOA Oracle home. Below is the Oracle home in my SOA 12c environment.
• ORACLE_HOME = /opt/app/oracle/middleware
2. Set Java Home

•   export JAVA_HOME=/opt/app/oracle/jdk
•   export PATH=$JAVA_HOME/bin:$PATH 

           3.  Set Oracle Home

•  ORACLE_HOME=/opt/app/oracle/middleware
•  export PATH=$ORACLE_HOME/OPatch:$PATH

           4.  Verify the Java_Home & Oracle_Home pointing to right directory and also check the opatch location pointing        to correct ORACLE_HOME.

o   $ which opatch

o   /opt/app/oracle/middleware/OPatch/opatch

          5.    Then go to downloaded directory (/opt/app/oracle/soa_bp_12.1.3.2/20423408) and apply the patch using        opatch. Before running check the ‘lsinventory’.

[oracle@mySOAHost01 oracle]$ opatch lsinventory -jre /opt/app/oracle/jdk/jre

Oracle Interim Patch Installer version 13.2.0.0.0

Copyright (c) 2014, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/app/oracle/middleware

Central Inventory : /opt/app/oracle/oraInventory

   from           : /opt/app/oracle/middleware/oraInst.loc

OPatch version    : 13.2.0.0.0

OUI version       : 13.2.0.0.0

Log file location : /opt/app/oracle/middleware/cfgtoollogs/opatch/opatch2015-06-20_05-54-49AM_1.log

OPatch detects the Middleware Home as "/opt/app/oracle/middleware"

Jun 20, 2015 5:55:08 AM oracle.sysman.oii.oiii.OiiiInstallAreaControl initAreaControl

INFO: Install area Control created command with access level  0

Lsinventory Output file location : /opt/app/oracle/middleware/cfgtoollogs/opatch/lsinv/lsinventory2015-06-20_05-54-49AM.txt

--------------------------------------------------------------------------------

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

          6. Run the opatch.

[oracle@mySOAHost01 oracle]$ cd soa_bp_12.1.3.2/20423408/

[oracle@mySOAHost01 20423408]$ opatch apply

Oracle Interim Patch Installer version 13.2.0.0.0

Copyright (c) 2014, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/app/oracle/middleware

Central Inventory : /opt/app/oracle/oraInventory

   from           : /opt/app/oracle/middleware/oraInst.loc

OPatch version    : 13.2.0.0.0

OUI version       : 13.2.0.0.0

Log file location : /opt/app/oracle/middleware/cfgtoollogs/opatch/20423408_Jun_20_2015_06_00_47/apply2015-06-20_06-00-32AM_1.log

OPatch detects the Middleware Home as "/opt/app/oracle/middleware"

Jun 20, 2015 6:00:47 AM oracle.sysman.oii.oiii.OiiiInstallAreaControl initAreaControl

INFO: Install area Control created with access level  0

Applying interim patch '20423408' to OH '/opt/app/oracle/middleware'

Verifying environment and performing prerequisite checks...

Patch 20423408: Optional component(s) missing : [ oracle.integration.bpm, 12.1.3.0.0 ] , [ oracle.mft.apache, 12.1.3.0.0 ] , [ oracle.bpm.processspaces, 12.1.3.0.0 ] , [ oracle.bpm.addon, 12.1.3.0.0 ] , [ oracle.mft, 12.1.3.0.0 ] , [ oracle.soa.workflow.wc, 12.1.3.0.0 ] , [ oracle.bpm.plugins, 12.1.3.0.0 ]

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/opt/app/oracle/middleware')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Patching component oracle.rcu.soainfra, 12.1.3.0.0...

Patching component oracle.rules, 12.1.3.0.0...

Patching component oracle.soa.mgmt, 12.1.3.0.0...

Patching component oracle.integration.bam, 12.1.3.0.0...

Patching component oracle.soa.common.adapters, 12.1.3.0.0...

Patching component oracle.soacommon.plugins, 12.1.3.0.0...

Patching component oracle.integration.soainfra, 12.1.3.0.0...

Verifying the update...

Patch 20423408 successfully applied

Log file location: /opt/app/oracle/middleware/cfgtoollogs/opatch/20423408_Jun_20_2015_06_00_47/apply2015-06-20_06-00-32AM_1.log

OPatch succeeded.

[oracle@mySOAHost01 20423408]$

          7.  Now run lsinventory command again and verify the patch is installed to corresponding Oracle home and start            all services. Apply the same patch to other node(s) of SOA Cluster.

Oracle AIA for Communications Electronic Document Uploader (EDU) Implementation Setup

The EDU mainly used in Communication industry where Oracle RODOD (Rapid Offer Design and Order Delivery) setup implemented.

Refer my Previous post on  EDU Introduction and information (EDU Introduction). 

·        EDU is part of Oracle AIA media pack & it is delivered in Oracle AIA Siebel CRM Integration Pack for Oracle Communications Billing and Revenue Management (BRM) for Agent Assisted Billing Care (AABC) 11.3.

·        Download the EDU required bundle (17760006_1of2.zip)

·        Unzip the above bundle and navigate to $STAGE/17760006/files/services/industry/Communications/JavaServices/ebp/ directory. This directory  contains eduapp.zip file, WCCProvisioningForEDU folder & other files.

·        WCCProvisioningForEDU is required for WCC integrations with EDU system.

·        EDU setup requires, Oracle Weblogic cluster setup (if require EDU high availability).

o   Install Weblogic & create a cluster (edu_cluster) setup with 2 managed servers (WLS_EDU1/WLS_EDU2) .

o   Note that no DB required and no need to create any Data Sources.

o   As I stated in my previous post, EDU uses JMS Queue to stores the invoices retrieved by Document Retriever. So, we have to create JMS server & required Modules (Refer below info & DO NOT change the names defined below as these value(s) are used in edu.ear application code).

o   The eduapp.zip file [extracted from EDU bundle] contains the edu.ear file and this file will be deployed to Weblogic cluster environment and target it to edu_cluster.

o   If OHS setup included, then we have to add below context root of edu application (/edu) to mod_wls_ohs file located at (<ORACLE_WebTier_Home>/instances/<instance_name>/config/OHS/ohs1).

<< 

                       <Location /edu>

SetHandler weblogic-handler

WebLogicCluster WLS_EDU_Host1:7010, WLS_EDU_Host2:8010

</Location>  

>> 

·        As deployment is finished, we need to check the servers & deployment status. Verify & validate the Weblogic servers section & deployment page. Refer the below screens.

·        As a post configuration steps, access the edu app using the below URL and update the required information on below shown screen.

http://OHS_host:7777/edu/configedu   (supply weblogic credentials)

§  Update the EDU JMS Server URL on the configuration UI screen with below details.

·        t3:// WLS_EDU_Host1:7010, WLS_EDU_Host2:8010

§  Update BIP URL(s) on the configuration UI screen with below details.

§  Update WCC RIDC protocol URL on the configuration UI screen with below details.

Note: BI Publisher & WCC post configurations are not covered in this post.

Oracle AIA for Communications Electronic Document Uploader (EDU) - A short Introduction and Information

The EDU mainly used in Communication industry where Oracle RODOD (Rapid Offer Design and Order Delivery) setup implemented.

The EDU component is part of AIA Electronic Billing presentation (EBP). The  EBP is the process of generating, storing, publishing, and presenting customer invoices digitally. It help us to provide a Quality Customer Experience.

EDU is the Oracle AIA EBP document publisher component. It automatically retrieves customer invoices in PDF format from BI Publisher and publishes them to WebCenter Content.

The main EDU integrations are with applications BRM, BIP & WCC

o   BRM – Oracle Billing & Revenue Management.

o   BIP   - Oracle Business Intelligence Publisher

o   WCC – Oracle WebCenter Content

Before configuring/using EDU, we have to have  below systems ready and integrated.

o   BRM & BIP system’s are ready and integrated successfully.

o   Required EDU post configurations are implemented @ BIP & WCC level.

EDU Process flow:

•            The EDU Document Retriever fetches the invoices from BI Publisher
•            A JMS Queue used to stores the invoices retrieved by Document Retriever
•            One message-driven bean (as part of edu.ear) has deployed and it reads messages from the JMS Queue and invokes the Document Publisher.
•            The EDU Document Publisher will fetch the invoices from the JMS Queue and publishes them to WebCenter Content.
•            The EDU Republish Utility: Reruns failed publications.

 Note: How to implement the AIA Communications EDU setup on Linux environment will be posted in my next post.

OUD11g (Oracle Unified Directory) Upgrade to 11g Release 2 (11.1.2.3) [OUD R2 PS3]

Follow below steps  for OUD11g (Oracle Unified Directory) Upgrade.

1.      Stop the OUD running instances using

/u01/app/oracle/middleware/asinst_1/OUD/bin/stop-ds

 [15/Jun/2015:16:54:45 +0530] category=CORE severity=NOTICE msgID=458955 msg=The Directory Server is now stopped

2.      Goto downloaded 11.1.2.3 software location and run runinstaller to invoke upgrade installer software.

      Follow below screens & give your existing OUD Home info.

               $STAGE/Disk1/runInstaller -jreLoc /u01/app/oracle/middleware/jdk1.7.0_51

      

              

3.      Verify the existing OUD Version by running below command.

[oracle@myOUDHost01 config]$ more /u01/app/oracle/middleware/asinst_1/OUD/config/buildinfo

11.1.2.2.0.1311150604

4.      Run the below upgrade command for OUD Instance.

/u01/app/oracle/middleware/asinst_1/OUD/bin/start-ds --upgrade

[15/Jun/2015:17:43:01 +0530] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerShutdown, alert ID 458893):  The Directory Server has started the shutdown process.  The shutdown was initiated by an instance of class org.opends.server.core.DirectoryServer and the reason provided for the shutdown was The server instance /u01/app/oracle/middleware/asinst_1/OUD has been successfully upgraded

5.      Verify the OUD Version by running below command.

[oracle@myOUDHost01 bin]$ more /u01/app/oracle/middleware/asinst_1/OUD/config/buildinfo

11.1.2.3.0.1504140602

6.      Then startup the OUD instance.

/u01/app/oracle/middleware/asinst_1/OUD/bin/start-ds

[15/Jun/2015:17:44:10 +0530] category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID 458887):  The Directory Server has started successfully

Finally, upgraded OUD instance will come into active state…

OUD11g: Update the Java version on Oracle Unified Directory (OUD 11.1.2.2) Instance.

1.      Delete the file  /Opt/Oracle/Middleware/asinst_1/OUD/lib/set-java-home file  if it exists.

2.      Install the Java (updated version)

3.      The OUD Java properties file is located in:  /Opt/Oracle/Middleware/asinst_1/OUD/config/java.properties and update the path & version of new Java

4.      Run the command-line /Opt/Oracle/Middleware/asinst_1/OUD/bat/dsjavaproperties file.

The script output looks like below.

<< 

The operation was successful.  The server commands will use the java arguments and java home specified in the properties file located in

/Opt/Oracle/Middleware/asinst_1/OUD/config/java.properties

>> 

5.      Then OUD Instance will take the new java version.

WCC: WebCenter Spaces roles missing under WCC/UCM Admin Applet view

Unable to see the below WebCenter Roles from UCM system ( Admin Applet section of Security Tab).

1. PersonalSpacesRole
2. WebCenterSpacesUserRole

Tried to verify communication from WebCenter Spaces  to Content Server and the required configuration is not available. 

Also when we tries to check Documents tab from WebCenter Spaces server from Oracle EM, getting below error.

"Document Service is unavailable as there is no default content repository connection"

To fix this issue, followed below steps.

1. Connect to EM --> WebCenter > WebCenter Spaces > WC_Spaces1 (&2) > Settings > Service configuration > Content Repository >

2. Add the Required data of UCM RIDC Info. Refer the below info.

Name and Type:

- Connection Name - UCM

- Repository Type - Oracle Content Server

- Active Connection - selected

WebCenter Spaces Content Respository:

- Content Administrator - weblogic

- Root Folder - /webcenter

- Application Name - webcenter

Connection Details:

- RIDC Soctet Type - Socket

- Server Host - RIDC_HOST ( Load Balancer Host Name of  UCM RIDC Host)

- Server Port - 4444  ( UCM RIDC Port)

- Connection Timeout (ms) - 30000

- Authentication Method - Identity Propagation

- Web Server Context Root - /cs

- Administrator User Name - sysadmin

Cache Details:

- Cache Invalidation Interval (minutes) - 0

- Maximum Cached Document Size (bytes) - 1024

Once this configuration is done, then we are able to see the required WebCenter Spaces role(s) at UCM side ( Admin Applet section under /cs URL).

OHS12c: Different ways of start/stop OHS instance services

We can start/stop the ohs instance in different ways. Here are the few ways...

1. By using WLST

• Connect to WLST
• Connect to nmConnect()
• Run the below command(s).
• START -    start('ohs1')  [or]   nmStart(serverName='ohs1', serverType='OHS')
• STOP    -    shutdown('ohs1')
• RESTART -  softRestart('ohs1')
• STATUS      - state('ohs1')

2. By using command line

•     Go to Domain location & run the below .sh scripts.
• START - $DOMAIN_HOME/bin/startComponent.sh ohs1
• STOP -    $DOMAIN_HOME/bin/stopComponent.sh ohs1

Also we can start/stop from EM (Enterprise Manager). 

• Login to EM & then select corresponding Weblogic Domain name. 
• Select machine host to be associated to corresponding OHS instance and then start/stop the corresponding instance.

Note: There are so many WLST locations available in your setup depending up on installed FMW components. Make sure to start WLST script from $ORACLE_HOME/ohs/common/bin directory. 

OHS12c - Update the ohs instance level configurations

As OHS12c is associated with Weblogic server and if the corresponding Weblogic setup as per EDG (Enterprise Deployment Guide) then you have to follow below guidelines if you are updating OHS configurations.

1. Make sure to update the configurations at aserver path (i.e. Admin Server location) and restart Admin Server. Then the changes will be propagated to mserver path (i.e. Managed Server/OHS instance locations)
2. If you directly apply any change at mserver path, then those changes will available till you restart the Admin Server. 
3. After that your Admin Serer will propagate the Admin Server copies of httpd.conf, mod_wl_ohs.conf/other all configurations to mserver path. As a result your local changes which are available at mserver path will be lost.

Conclusion: Make sure to update the configurations at aserver path.

OHS12c: Admin port conflict issue while configuring the new instance.

• There are 2 options available while installing OHS.
• Collocated HTTP Server
• Standalone HTTP Server
• At the time of configure OHS, below screen appear in the middle of configurations. 

• In the above screen, we should not use Weblogic Server Admin & Port information. This is specific to OHS Configurations and have to use local sever info (other than Weblogic Admin Server Port). 
• If we use Admin server info & if already Admin Sever is running then will get below error stack ( If we start OHS Instance).
• [2015-06-12T04:22:23.3124-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: My_OHS_Server1] [host_addr: 10.25.11.6] [pid: 25178] [tid: 140568266618688] [e_sock: could not bind to address 10.25.11.10:7001
• [2015-04-12T04:22:23.3124-04:00] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [worker.c] [host_id: My_OHS_Server1] [host_addr: 10.25.11.6] [pid: 25178] [tid: 140568266618688] [ shutting down
• [2015-04-12T04:22:23.3124-04:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: My_OHS_Server1] [host_addr: 10.25.11.6] [pid: 25178] [tid: 140568266618688] [user: oracle ..
• Make sure have other port info other than Admin server info. 
• If you have already used Admin info then try to modify the configuration at OHS configuration files and start OHS instance.

Finally OHS instance will come into normal state.

SOA/AIA - AID Script ( AIA Deployment script) inputting user credentials while deploying

If we try to deploy the custom code on AIA and AID script will look for user credentials to be input at time of invoke.

 This is because of existing SOA/AIA environment credentials are updated some of the places manually and it is not updated in proper way.

To correct this, we have followed below approach.

1. Take the backup of AIAInstallProperties.xml file located under  $AIA_HOME/aia_instances/aia_inst1/config directory.

2. source $AIA_HOME/aia_instances/aia_inst1/bin/aiaenv.sh

3. Go to $AIA_HOME/util and Run the following command
Note: Make sure to have VNC/X-Manager/X-ming or a X11 Server Display has to be set as the below UpdateStore Utility opens a GUI window

ant -f updateStore.xml updateStore -DAdminUsername=weblogic -DAdminPassword=xxxxx

Once utility opens up, it will as below details.
Enter Existing Username, Password, New Username, New Password and XPATH
This XPATH reference should be exactly the same as suggested in the AIAInstallProperties.xml  (i.e. /properties/fp/server/password)

4. Once this process completed, it will update the credentials in AIAInstallProperties.xml file.

5. Also create Aid.Properties (under $AIA_HOME/aia_instances/aia_inst1/.configuration directory) with below details.
oracle.aia.security.username=weblogic
oracle.aia.security.password=xxxxxxxx

6.    Upload the  AIAInstallProperties.xml file to MDS repository.

7. Then try to run sample deployment as shown below and AID will not prompt for any user credentials.
ant -f $AIA_HOME/Infrastructure/Install/AID/AIAInstallDriver.xml -DPropertiesFile=$AIA_HOME/aia_instances/aia_inst1/config/AIAInstallProperties.xml -DDeploymentPlan=dp_test.xml

Note: The same solution can be used for changing the password for AIA environment.

Any suggestions/comments are welcome.