Using the WebSphere Integrated Solutions Console, a large (18,400 file) web application is updated by specifying a war file name and going through the update screens and finally saving the configuration. The Solutions Console web UI spins a while, then it returns, at which point the user is able to start the web application.
If the application is started after the "successful update", it fails because the files that are the web application have not been exploded out to the deployment directory yet.
Experimentation indicates that it takes on the order of 12 minutes for the files to appear!
One more bit of background that may be significant: There are 19 application servers on this one WebSphere instance. WebSphere insists that there be a lot of chatter between them, even though they don't need anything from each other. I wondered if this might be slowing things down when it comes to deployment. Or if there's some timer in the bowels of WebSphere that is just set wrong (usual disclaimers apply...I'm just showing up and finding this situation...I didn't configure this installation).
Additional Information:
This is a Network Deployment configuration, and it's all on one physical host.
* ND 6.1.0.23
Is this a standalone or a ND set up? I am guessing it is ND set up considering you have stated that they are 19 app servers. The nodes should be synchronized with the deployment manager so that the updated files are available to the respective nodes.
After you update and save the changes, try and synchronize the nodes with the dmgr (or alternatively as part of the update process, click on review and the check the box which says synchronize nodes) and this would distribute the changes to the various nodes.
The default interval, i believe is 1 minute.
12 minutes certainly sounds a lot. Is there any possibility of network being an issue here?
HTH
Manglu
Related
While looking into the resource balancer and dynamic load metrics on Service Fabric, we ran into some questions (Running devbox SDK GA 2.0.135).
In the Service Fabric Explorer (the portal and the standalone application) we can see that the balancing is ran very often, most of the time it is done almost instantly and this happens every second. While looking at the Load Metric Information on the nodes or partitions it is not updating the values as we report load.
We send a dynamic load report based on our interaction (a HTTP request to a service), increasing the reported load data of a single partition by a large amount. This spike becomes visible somewhere in 5 minutes at which point the balancer actually starts balancing. This seems to be an interval in which the load data gets refreshed. The last reported time gets updated all the time but without the new value.
We added the metrics to applicationmanifest and the clustermanifest to make sure it gets used in the balancing.
This means the resource balancer uses the same data for 5 minutes. Is this a configurable setting? Is it constraint because it is running on a devbox?
We tried a lot of variables in the clustermanifest but none seem to be affecting this refreshtime.
If this is not adaptable, can someone explain why would you run the balancer with stale data? and why this 5 minute interval was chosen?
This is indeed a configurable setting, and the default is 5 minutes. The idea behind it is that in prod you have tons of replicas all reporting load all the time, and so you want to batch them up so you don't spam the Cluster Resource Manager with all those as independent messages.
You're probably right in that this value is way too long for local development. We'll look into changing that for the local clusters, but in the meantime you can add the following to your local cluster manifest to change the amount of time we wait by default. If there are other settings already in there, just add the SendLoadReportInterval line. The value is in seconds and you can adjust it accordingly. The below would change the default load reporting interval from 5 minutes (300 seconds) to 1 minute (60 seconds).
<Section Name="ReconfigurationAgent">
<Parameter Name="SendLoadReportInterval" Value="60" />
</Section>
Please note that doing so does increase load on some of the system services (TANSTAAFL), and as always if you're operating on a generated or complete cluster manifest be sure to Test-ServiceFabricClusterManifest before deploying it. If you're working with a local development cluster the easiest way to get it deployed is probably just to modify the cluster manifest template (by default here: "C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\NonSecure\ClusterManifestTemplate.xml") and just add the line, then right click on the Service Fabric Local Cluster Manager in your system tray and select "Reset Local Cluster". This will regenerate the local cluster with your changes to the template.
I have a Digital Ocean droplet (512MB RAM, 20GB SSD Disk, Ubuntu 13.10 x64) on which
a MongoDB instance and
a Tomcat 7 server
run.
On the Tomcat server, following applications are installed
Apache CXF-based application, which takes processes web service requests, interacts with the database and executes scheduled jobs,
Vaadin application,
JSF (Primefaces) application and
Psi Probe.
When I
restart Tomcat,
use the Vaadin and/or JSF application,
then for several weeks do nothing on that machine (it basically is idle during that time),
then try to open the JSF and/or Vaadin application,
I find the site unresponsive (nothing is displayed after I enter the URL in the browser).
When I restart Tomcat (sudo service tomcat7 restart), everything works again. I don't see any obvious problems in the Tomcat logs.
How can I find out,
whether the problem is on the Tomcat side (one of the applications consumes too many resources even if idle) or on the OS side (nothing happens on the machine and therefore the OS puts itself into a "hibernating" mode) and
if the problem is with Tomcat, exactly which of the application is causing it?
Please start from top to bottom.
then try to open the JSF and/or Vaadin application,
I find the site unresponsive (nothing is displayed after I enter the
URL in the browser).
Check if the service is still running before restarting sudo service tomcat7 status and/or ps -ef | grep tomcat
Check with netstat -patune | grep <portnumber, e.g. 443> if the server is listening on the configured ports
Check your httpd/Apache/Tomcat access logs if the request reaches the server and if yes, check if there are errors or timeouts related to the requests
Check if the DB connection is still possible
To force some error logs, try to change your maxIdle, maxActive and maxWait attributes of your Tomcat's Connection Pool configuration. maxWait default is -1, connections created sometimes during these weeks will wait forever.
#Patrick provided some excellent basic tests.
I notice that you only have 512 MB of RAM. With some fairly beefy software such as tomcat, plus MongoDB on top of that, your machine may simply be overloaded.
Based on that, I would propose a couple additional things to check:
sudo free
Should tell you how much memory is being used, and how much swap space you use.
sudo top
Will tell you which process is using the most memory. You may want to sort the output of top by memory (default is usually by CPU utilization).
Most importantly, check the log files in /var/log (in particular /var/log/messages). You may find indications that the kernel killed one of your processes (probably tomcat).
I am trying to automate the startup of my Selenium Grid.
I have the Hub registered as a service, so that starts when the machine starts, but
the literature tells me I can't do the same with the node, because it won't be in a User context, and so I would not be able to get screenshots etc.
I've seen vague hints that you can add something to the registry to start a program, but I'm not really convinced thats what I want.
IT pulls down the servers for upgrades at intervals, and sessions are set to time out after X amount of inactivity, so its a tedious and silly process to open remote desktops to all 6 nodes, in order to log in, then start the process every time.
How do you best manage this?
- Configure the machines to auto-login, and place startSeleniumNode.bat in that users start-folder?
- Add some kind of commandline entry in the jenkins build script that launches the test, to call each of the 6 nodes in turn to start the selenium node (and how would you do that?)
Take a look at AlwaysUp - it allows you to run almost any application as a Windows service including Selenium Grid hubs and nodes.
I've previously created a fairly large Grid infrastructure using AlwaysUp for node management. It's very useful for starting up Grid on boot and lets you specify a user account to run as, schedule restarts at regular intervals and a lot more.
Our requirement from our next web app is that we will be able to deploy a new version of the web app without a downtime.
how is it possible to achieve such task?
does it mean we need to run 2 different servers (tomcats) ? and redirect users to each one when needed?
are there tools that are doing this specific task? in what category these tools in?
Thanks
Just use Tomcat's parallel deployment feature. It is available from Tomcat 7 onwards.
Don't forget, 100% availability is impossible - it may happen for a certain period, but no one can guarantee it, no matter what setup you have.
But since you're looking for a smooth change from one version to another, then the best you can do is update one node and switch nodes then. Of course, since you likely have sessions which shouldn't disconnected, you'll need to make sure that an instance (e.g. load balancer) directs all new requests to the new node, whereas old session requests stay on the old node until no one uses it anymore, after which you can upgrade the second node and finally, balance load again to both nodes.
How do I start WebSphere 6.x admin service without starting my apps? Does anyone know how to do this (if its even possible)?
Since my app takes about 5+ minutes to start, I am hoping I can start the admin first , then login to the GUI and then manually start my app from the GUI. Another reason I want to do this is because the .ear update/upgrade process goes much faster if the app is turned off first and then I update the code and then I restart the app. If I try to update on the fly it seems to take much longer.
Change the default start up properties for the application to "Stopped".
The default is to start the applications as part of server start up.
Enterprise Applications --> Target Specific Application Status --> Disable Auto Start
I am assuming that you are a stand alone set up. If you are on ND environment, administration is essentially performed using the deployment manager and your application servers need not run for the activities to be performed.