How to distribute websphere server configuration (datasources,jms,...) to multiple instances? - websphere

For a team of developers it is essential that everybody sets up and configures the application server. In our case we are using websphere 8.5.
I'm looking for an easy way to do this. Normally you do it using the profile management tool located in WAS_HOME/bin/ProfileManagement and this tool works quiet well. But after the installation of the websphere server one still needs to configure the server profile - creating datasources, JMS queues, buses, variables and so on. So I thought it would be nice if there is a way to apply these configurations to an existing profile.
My first try was to just configure one profile and then make a configuration backup using
%WAS_HOME%/bin/backupConfig.bat
But the configuration contains e.g. the hostname and other host dependent configurations. So the backupConfig.bat tool is not what I'm looking for.
The next thought that came in my mind is that I might could create a special profileTemplate. So that others can use the profile management tool and use this template. But the template structure does not seem to be made for customization. A lot of files and nearly no documentation can be found of how to create an own profile template.
So I came across the augment templates. These template are used (as the name implies) to add specific configuration to an existing profile. I found a lot of documentation of how to apply an augmentation to an existing profile but no documentation of how to create an augmentation.
At last I think that there must be some way of exporting websphere datasource, buses, jms etc. configuration and apply them to other profiles, because in very big installations the operations team must have this ability.
I know that I can add container-specific descriptors to the EAR. E.g. META-INF/ibmconfig/cells/defaultCell/applications/defaultApp/resources.xml. But I don't want to build environment specific EAR files, because it couples our builds to the infrastructure and thus we have to build and redeploy when ever operations changes the infrastructure, e.g. hostnames, IPs, passwords.
Does anyone know how to manage the distribution of datasource, buses, jms, etc. to multiple websphere installations?

In addition to wsadmin scripts - which are very good for these kind of tasks, I'd suggest Properties-based configuration. It might be more useful for you, since it allows to export many configuration objects at one time and then apply it to different environments. It is also might a bit easier, since you work on plain text files instead of jython scripts.
Properties file based configuration enables you to:
Extract data out of the configuration repository to create properties
files.
Update a properties file to manipulate the configuration, as
needed.
Apply the updated data in the properties file to a target
configuration repository.
See more details here:
Properties-based configuration
Infocenter documentation
Education assistant

I suggest you to use wsadmin shell scripting and create a script for resource creation. A bonus is that you can run it directly from RAD (right click Run As->Administrative Script).
Here is the complete example written in Jython for JDBC resource creation along with JAAS login information (note: I'm using Oracle Database, your setup could be different depending on database you are using):
cell=AdminConfig.showAttribute(AdminConfig.list("Cell"), "name")
node=AdminConfig.showAttribute(AdminConfig.list("Node"), "name")
#Add JAAS credentials
print "Adding JAAS credentials"
security = AdminConfig.getid('/Cell:'+cell+'/Security:/')
alias = ['alias', node+'/dbUser']
userid = ['userId', 'DBUSER']
password = ['password', 'PASSWORD']
jaasAttrs = [alias, userid, password]
AdminConfig.create('JAASAuthData', security, jaasAttrs)
AdminConfig.save()
#Add JDBC jar path
print "Adding JDBC jar path"
AdminTask.setVariable('[-variableName ORACLE_JDBC_DRIVER_PATH -variableValue ${WAS_INSTALL_ROOT}/lib/ext -scope Cell='+cell+',Node='+node+']')
AdminConfig.save()
#JDBC Provider print "Adding JDBC Provider"
AdminTask.createJDBCProvider('[-scope Node='+node+',Server=server1 -databaseType Oracle -providerType "Oracle JDBC Driver" -implementationType "Connection pool data source" -name "Oracle JDBC Driver" -description "Oracle JDBC Driver-compliant Provider." -classpath ${ORACLE_JDBC_DRIVER_PATH}/ojdbc6.jar]')
AdminConfig.save()
#JDBC Datasources print "Creating Datasource"
AdminJDBC.createDataSourceAtScope("Node="+node+",Server=server1", "Oracle JDBC Driver", "test", "jdbc/test", "com.ibm.websphere.rsadapter.Oracle11gDataStoreHelper", "jdbc:oracle:thin:#10.0.0.1:1521:TEST", [['componentManagedAuthenticationAlias', node+'/test'], ['containerManagedPersistence', 'true']])
AdminConfig.save()

I just remebered the wsadmin tool and guess it is the best way to implement my requirements.
Fortunately IBM provides sample scripts that show you how to create datasources or modify them using jython or jacl scripts.
An example of how to create datasources can be found for example in the Administration scripts (1-12) -- Jython version (file ex7.py in the zip)
Hope this helps others who have the same or similay question.

Related

Is remote partitioning available with JSR-352 under Spring Batch?

I would like to configure a JSR-352 batch job for remote partitioning, behind the scenes through configuration, without having to explicitly define the controller/worker in the job definition (which is not supported by the JSR-352 specification anyway). IBM WebSphere Liberty provides this capability with their "multi-JVM" feature, where you define remote partitioning within the configuration file (server.xml).
I have seen that legacy Spring Batch has the ability to support remote partitioning, but only through an explicit job definition. I don't want to use legacy Spring Batch. Instead, I want to build a solution that is portable to Java EE, including IBM Liberty.
I don't want to use legacy Spring Batch. Instead, I want to build a solution that is portable to Java EE, including IBM Liberty.
If you really want a portable solution, you need to define your job in the Job Specification Language of the JSR-352 and launch it with:
javax.batch.runtime.BatchRuntime.getJobOperator().start(jobName, properties)
Now if JSR-352 does not specify how to define a remote partitioning setup, then you can't get a portable solution for that.
Please note that there are some differences between Spring Batch and JSR-352 which are detailed in the JSR-352 Support section. The differences regarding partitioning are documented here.

Need advice on deploying on Liberty and managing sensitive user/passwords data *for db, mq etc.)

We are looking to move our dev+prod WebSphere full profile app to Liberty.
Currently, we build only once (using Ant scripts) and deploy the same package (i.e. EAR) to our functional, UAT and production environments.
Database and MQ connections (and related sensitive data like usernames and passwords) are directly set via the WAS admin console for each environment, so there is no such data in our EAR.
A few non sensitive settings that change per environment (mail server address etc), are kept in a file suffixed with the (e.g configuration_.properties). All these files are bundled within the EAR. Each WebSphere defines a JVM property to specify the environment they are running (prd, uat, fnc, lab etc). When the application starts, it reads the files that is associated with the environment. That works great.
Now with Liberty, the connection/MQ pools, LDAP users etc are defined in server.xml.
Questions:
how to manage the server.xml file(s) that replace the job done via the WAS console by the authorized admins?
how to define the database name/port/host/user/password needed for those access per environnment? keep one server.xml file per environnement?
is there a way to have a "base" server.xml file and "override" the database name/port/host/user/password etc at startup on runtime?
or maybe there are more clever strategies?
We don't know yet if we will run Liberty in a traditional ND/Cluster way or into a docker infrastructure (this is all very new to us..).
How do you handle this?
Thanks in advance.
You can do the same thing in Liberty, just using different methods.
1) in your server.xml files, use variables wherever needed:
${this.style} for referencing system/bootstrap properties or server.xml defined variables, or ${env.ENV_VAR} for referencing Environment variables
2) add in a per-environment server.env file, or use configDropins/overrides to define environment-specific server.xml snippets (this answers one of your questions: yes, you can have a base server.xml and use environment-specific overrides)
More information here: http://www.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.core.doc/ae/cwlp_config.html?lang=en
And here (specifically includes and dropins):
http://www.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.core.doc/ae/twlp_setup_basics.html?lang=en

WAS Liberty: DEV and UAT Configuration

I am wondering what is the best way to configure an WAS Liberty installation, allowing to to switch from a DEV environment configuration to an UAT(testing) environment configuration dynamically.
To elaborate, we have a similar setup with our glassfish servers, we simply configure system properties for both in the Glassfish console. For example
hostname.uat="some uat value"
hostname.dev="some dev value"
Dropping the ".uat" or ".dev" in the system property configuration in Glassfish makes that property active. In Glassfish, this can be done dynamically and while the application is running (no need to reboot).
Is there or can someone elaborate how I could achieve a similar setup in WAS Liberty?
Thank-you kindly
You can create a server.env file in two possible places:
${wlp.install.dir}/etc/server.env (properties are applied to all servers) or
${server.config.dir}/server.env (properties applied only to one server)
and specify any environment variables in that file.
For example:
# Specify properties and values
admin.email=dev.admin#domain.com
admin.email.uat=uat.admin#domain.com
To access these properties in an application environment (such as a Servlet) do the following:
System.getenv("admin.email"); // returns "dev.admin#domain.com"
Other useful properties can be specified in the server.env file as well such as JAVA_HOME, WLP_USER_DIR, WLP_OUTPUT_DIR, and WLP_DEBUG_ADDRESS.
For IBM's full doc on this, see: Customizing the Liberty Environment.
What we do is generate the Liberty server with Ansible, where the variables can be added to an ansible inventory based on environment.
So, our deployments essentially drop and recreate the Liberty server by using ansible templates and roles to stamp it out as needed.
Lastly, we make use of Hasicorp Vault (you can also use ansible-vault) for credentials or secrets at deploy time to fetch credentials. This is then injected into Ansible as JSON and used to stamp out the server.xml and other related configuration files.

WSO2 Identity Server - Custom JDBC User Store Manager - JDBC Pools

WSO2 Identity Server 5.0.0 (and some patches ;))
It does not appear that custom JDBC user store managers (child of JDBCUserStoreManager) use a JDBC pool. I'm noticing that I can end up session closed errors and sql exceptions whereas the Identity Server itself is still operating OK with its separate database connection (a configured pool).
So I guess I have two questions about this:
Somewhere up the chain, is there a JDBC pool for the JDBCUserStoreManager? If so, are there means to configure that guy more robustly?
Can I create another JDBC datasource in master-datasources.xml which my custom JDBC user store manage could reference?
Instead of using your own datasources/connections, you can import Carbon Datasources and use those (they come with inbuilt pooling and no need to worry about any configurations etc). You can either access these programmatically by directly calling ndatasource component or access them via JNDI.
To access them directly from ndatasource component:
The dependency:
<dependency>
<groupId>org.wso2.carbon</groupId>
<artifactId>org.wso2.carbon.ndatasource.core</artifactId>
<version>add_correct_version_here</version>
</dependency>
(You can check repository/components/plugins to find out the correct version for above dependency)
You can inject DataSourceService as in this code (the #scr.reference tag refers to the service you need to inject, this uses maven scr plugin to parse these dependencies when building the bundle).
Note that when you follow this approach you'll have to build the jar as an OSGi bundle as it uses declarative services (and have to place it in repository/components/dropins). Otherwise the dependencies won't be injected at runtime.
Next, you can access all the data sources as:
List<CarbonDataSource> dataSources = dataSourceService.getAllDataSources();
Rajeev's answer was really insightful and helped with investigating and evaluating what I should do. But, I didn't end up using that route. :)
I ended up looking through the Identity Server and Carbon source code and found out that the JDBCUserStoreManager does end up creating a JDBC pool configured by the properties you set for that manager. I had a class called CustomUserStoreConstants for my custom user store manager which had setMandatoryProperty called by default to set:
JDBCRealmConstants.DRIVER_NAME
JDBCRealmConstants.URL
JDBCRealmConstants.USER_NAME
JDBCRealmConstants.PASSWORD
So the pool was configured with these values, BUT that was it...nothing else. So no wonder it wasn't surviving the night!
It turned out that the code setting this up, if it found a value for the JDBCRealmConstants.DATASOURCE in the config params, it would just load up that datasource and ignore any other params set. Seeing that, I got rid of those 4 params listed above and forced my custom user store to only allow having a DATASOURCE and I set it in code with the default JNDI name that I would name that datasource always. With that, I was able to configure my JDBC pool for this datasource with all params such as testOnBorrow, validationQuery, validationInterval, etc in master-datasources.xml. Now the only thing that would ever need to change is the datasource's configuration in that file.
The other reason I went with the datasource in the master-datasources.xml is that I didn't have to decided in my custom user store's code which parameters I would want to have or not have and just manage it all in the xml file easily. This really has advantages with portability of configs and IT involvement for deployments and debugging. I already have other datasources in this file for the IS deployment.
All said, my user store is now living through the night and weekends. :)

How to specify JDBC setting in a struts project?

I am trying to setup a struts project locally. One way I know to set up JDBC settings as to go to administrative console of websphere and create JDBC provider and JNDI and all. But is there any other way to do in the code itself?
There is some resource reference in web.xml. I am totally new to struts.Please help.
DataSourceAlias
javaxsql.Data...... etc etc
If you configured for WAS 6.1and configuration is good you need to stop and start nodeAgents for the changes to get propagated and test the jdbc connection after restarting.....if it was WAS 8 they will be propagated automatically that means you configured improperly

Resources