Error occurred during initialization of VM

Error occurred during initialization of VM
Could not reserve enough space for object heap

This is an unusual case of Java heap size. At first, we thought that Java is eating up a lot of memory but is not.

According to the application owner the problem happen all the time time, and they had to open up a ticket once in a few month.

Initially I'm not sure if this is actually capacity problem since the initial setting of the java heap size is very small.

My first option is to restart the Weblogic application to free up some memory and see if it’s really eating up a lot of memory.
The application owner agreed to restart the weblogic application.

I also asked if they are you using java console. I had a chance to trace the slope of the increase in memory usage via BMC perform. In the beginning of the slope I found a java console process start-up. I’m almost convinced that it’s the problem maker.

So I asked the application owner to stop that admin console after he restarted the application. But He told me that they are doing everything via command line. So maybe its not the one.

It took about 10 min to come up. But still the application couldn't start up. It’s still saying:

Error occurred during initialization of VM
Could not reserve enough space for object heap

Then I run top command to check what is still eating up the memory.

And I found out that there are sec.pl processes consuming memory.

It is actually the sec.pl which doing a lot of damage with the capacity. As per conversation with the application owner, sec.pl is a monitoring tool they used for the application. As those jobs are very small, and shouldn't take up much we opt to kill these processes.

I found 5 instances of sec.pl eating up more than 500MB each. So upon, killing these processes we got almost 3GB of memory available.

The application was able to run the script after.

Now we are still investigating why these sec.pl processes are eating up a lot of memory.