[omniORB] Resize strategy for POA not reclaiming memory when servants are deactivated for omniORB 4.1.7

[Jason Hein]

We are seeing an issue when using a POA to capture servants as they are
activated and deactivated. The memory size of the server program that
manages the POA is growing as servants are being activated by the POA.
Conversely, when we are deactivating those same servants and removing all
references to the object to initiate object destruction, we are not seeing
a reduction in memory size for the server program. Instead, the memory
size remains consistent when the last object was activated by the POA.

It looks like the containers that manage the servants and identifiers are
not re-sizing as we would expect. Are there policy settings or orb
configuration settings that we could alter to affect the resize activity on
these containers? When creating a child POA we are using the following
policy:

policy_list.length(pstart+1);
policy_list[pstart] = parent->create_request_
processing_policy(PortableServer::USE_ACTIVE_OBJECT_MAP_ONLY);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_lifespan_policy(PortableServer::TRANSIENT);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_id_assignment_policy(PortableServer::SYSTEM_ID);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_thread_policy(PortableServer::ORB_CTRL_MODEL);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_servant_retention_policy(PortableServer::RETAIN);
pstart++;?

Cheers,

Jason

[Duncan Grisby]

Post by Jason Hein
We are seeing an issue when using a POA to capture servants as they
are activated and deactivated. The memory size of the server program
that manages the POA is growing as servants are being activated by
the POA. Conversely, when we are deactivating those same servants and
removing all references to the object to initiate object destruction,
we are not seeing a reduction in memory size for the server program.
Instead, the memory size remains consistent when the last object was
activated by the POA.

Are you certain that you are getting your servant reference counting
right? Have you added logging to your servant classes' destructors to
confirm that they are running?

How are you measuring the memory usage? On some platforms, the OS often
does not reclaim freed process memory. When the process frees memory, it
doesn't give it back to the OS but instead holds onto it for future
allocations. Do you see your process grow continuously over time, or
does it just get big and then plateau?

Duncan.

[Jason Hein]

We put a log message in the destructor of the servant class to ensure that
the object clean-up was happening correctly. We even put additional
_remove_refs in the code to force a segfault condition to verify that
reference counting was triggering correctly. We ran a test to place 100000
NamedCounterImpl servants into a child POA using the AddNamedCounter
method. We then followed up with calling the RemoveNamedCounter method with
the same name sequence.

..... Output from server process, 100000 calls to AddNamedCounter

Counter goober_99995 added with a value of 0
Counter goober_99996 added with a value of 0
Counter goober_99997 added with a value of 0
Counter goober_99998 added with a value of 0
Counter goober_99999 added with a value of 0
omniORB: Deinitialising omniDynamic library.


..... Output from server process, 100000 calls to RemovedNamedCounter

2014-02-03 15:55:36 INFO NamedCounterFactory:84 - START .....
RemoveNamedCounter goober_99995
2014-02-03 15:55:36 INFO NamedCounterFactory:95 - RemoveNamedCounter:
NamedCount goober_99995
2014-02-03 15:55:36 INFO NamedCounter:23 - DTOR NamedCounter:
goober_99995" = 0
2014-02-03 15:55:36 INFO NamedCounterFactory:105 - RemoveNamedCounter:
Remove from List:goober_99995
2014-02-03 15:55:36 INFO NamedCounterFactory:108 - END .....
RemoveNamedCounter goober_99995
2014-02-03 15:55:36 INFO NamedCounterFactory:84 - START .....
RemoveNamedCounter goober_99996
2014-02-03 15:55:36 INFO NamedCounterFactory:95 - RemoveNamedCounter:
NamedCount goober_99996
2014-02-03 15:55:36 INFO NamedCounter:23 - DTOR NamedCounter:
goober_99996" = 0
2014-02-03 15:55:36 INFO NamedCounterFactory:105 - RemoveNamedCounter:
Remove from List:goober_99996
2014-02-03 15:55:36 INFO NamedCounterFactory:108 - END .....
RemoveNamedCounter goober_99996
2014-02-03 15:55:36 INFO NamedCounterFactory:84 - START .....
RemoveNamedCounter goober_99997
2014-02-03 15:55:36 INFO NamedCounterFactory:95 - RemoveNamedCounter:
NamedCount goober_99997
2014-02-03 15:55:36 INFO NamedCounter:23 - DTOR NamedCounter:
goober_99997" = 0
2014-02-03 15:55:36 INFO NamedCounterFactory:105 - RemoveNamedCounter:
Remove from List:goober_99997
2014-02-03 15:55:36 INFO NamedCounterFactory:108 - END .....
RemoveNamedCounter goober_99997
2014-02-03 15:55:36 INFO NamedCounterFactory:84 - START .....
RemoveNamedCounter goober_99998
2014-02-03 15:55:36 INFO NamedCounterFactory:95 - RemoveNamedCounter:
NamedCount goober_99998
2014-02-03 15:55:36 INFO NamedCounter:23 - DTOR NamedCounter:
goober_99998" = 0
2014-02-03 15:55:36 INFO NamedCounterFactory:105 - RemoveNamedCounter:
Remove from List:goober_99998
2014-02-03 15:55:36 INFO NamedCounterFactory:108 - END .....
RemoveNamedCounter goober_99998
2014-02-03 15:55:36 INFO NamedCounterFactory:84 - START .....
RemoveNamedCounter goober_99999
2014-02-03 15:55:36 INFO NamedCounterFactory:95 - RemoveNamedCounter:
NamedCount goober_99999
2014-02-03 15:55:36 INFO NamedCounter:23 - DTOR NamedCounter:
goober_99999" = 0
2014-02-03 15:55:36 INFO NamedCounterFactory:105 - RemoveNamedCounter:
Remove from List:goober_99999
2014-02-03 15:55:36 INFO NamedCounterFactory:108 - END .....
RemoveNamedCounter goober_99999



The child POA was created using the following policy list:

policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_request_processing_policy(PortableServer::USE_ACTIVE_OBJECT_MAP_ONLY);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_lifespan_policy(PortableServer::TRANSIENT);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_id_assignment_policy(PortableServer::SYSTEM_ID);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_thread_policy(PortableServer::ORB_CTRL_MODEL);
pstart++;
policy_list.length(pstart+1);
policy_list[pstart] =
parent->create_servant_retention_policy(PortableServer::RETAIN);
pstart++;


For our tests we used the top command and isolated the server process and
then recorded the RES value for each step of test. We are running on
CentOS 6.3 x86_64, with omniORB 4.1.7.

/***
.... Snipet from NamedCounterFactory

**/


void NamedCounterFactoryImpl::AddNamedCounter(
const char* name,
CORBA::Long initialValue)
{

LOG4CXX_INFO(logger, "START ..... AddNamedCounter " << name );
CounterContainer::const_iterator it = m_counters.find(string(name));

if (it != m_counters.end())
{
// found
throw NamedCounterFactory::CounterExists(name);
}

LOG4CXX_INFO(logger, "AddNamedCount Create NamedCounter " << name );x
PortableServer::Servant_var<NamedCounterImpl> svt = new
NamedCounterImpl(name, initialValue);
try {

LOG4CXX_INFO(logger, "AddNamedCount Add to POA, NamedCounter " <<
name );x
PortableServer::POA_var poa =
corba::RootPOA()->find_POA("NamedCounterImpl::POA", 1);
if ( !CORBA::is_nil(poa) ) {

PortableServer::ObjectId_var oid = poa->activate_object(svt);
svt.in()->m_objectID = oid;
}
else {
LOG4CXX_WARN(logger, "AddNamedCounter, Could not get POA,
NamedCounterImpl::POA" );
}
}
catch(...) {

// save off object
m_counters.insert(CounterContainer::value_type(name, svt.in()));

LOG4CXX_INFO(logger, "END ..... AddNamedCounter " << name );

}


void NamedCounterFactoryImpl::RemoveNamedCounter(const char* name)
{

LOG4CXX_INFO(logger, "START ..... RemoveNamedCounter " << name );

CounterContainer::iterator it = m_counters.find(string(name));

if (it == m_counters.end())
{
throw NamedCounterFactory::NoCounterExists(name);
}

NamedCounterImpl *nc_ptr = it->second;

LOG4CXX_INFO(logger, "RemoveNamedCounter: NamedCount " << name );
PortableServer::POA_var poa =
corba::RootPOA()->find_POA("NamedCounterImpl::POA", 0);

try {
poa->deactivate_object(nc_ptr->m_objectID);
}
catch(...) {
LOG4CXX_ERROR(logger, "RemoveNamedCounter: Deactivate NamedCount
Failed:" << name );
}

LOG4CXX_INFO(logger, "RemoveNamedCounter: Remove from List:" << name );
m_counters.erase(it);

LOG4CXX_INFO(logger, "END ..... RemoveNamedCounter " << name );
}

Post by Duncan Grisby

Post by Jason Hein
We are seeing an issue when using a POA to capture servants as they
are activated and deactivated. The memory size of the server program
that manages the POA is growing as servants are being activated by
the POA. Conversely, when we are deactivating those same servants and
removing all references to the object to initiate object destruction,
we are not seeing a reduction in memory size for the server program.
Instead, the memory size remains consistent when the last object was
activated by the POA.

Are you certain that you are getting your servant reference counting
right? Have you added logging to your servant classes' destructors to
confirm that they are running?
How are you measuring the memory usage? On some platforms, the OS often
does not reclaim freed process memory. When the process frees memory, it
doesn't give it back to the OS but instead holds onto it for future
allocations. Do you see your process grow continuously over time, or
does it just get big and then plateau?
Duncan.
--
-- Duncan Grisby --
-- duncan at grisby.org --
-- http://www.grisby.org --

[Duncan Grisby]

Post by Jason Hein
We put a log message in the destructor of the servant class to ensure
that the object clean-up was happening correctly. We even put
additional _remove_refs in the code to force a segfault condition to
verify that reference counting was triggering correctly. We ran a test
to place 100000 NamedCounterImpl servants into a child POA using the
AddNamedCounter method. We then followed up with calling the
RemoveNamedCounter method with the same name sequence.
..... Output from server process, 100000 calls to AddNamedCounter
Counter goober_99995 added with a value of 0
Counter goober_99996 added with a value of 0
Counter goober_99997 added with a value of 0
Counter goober_99998 added with a value of 0
Counter goober_99999 added with a value of 0
omniORB: Deinitialising omniDynamic library.

I find that concerning. Why is omniORB deinitialising the omniDynamic
library at that stage? It should only do that when the system is
shutting down. Are you dynamically loading and unloading libraries?

[...]

Post by Jason Hein
For our tests we used the top command and isolated the server process
and then recorded the RES value for each step of test. We are running
on CentOS 6.3 x86_64, with omniORB 4.1.7.

Looking at the resident size of the process doesn't tell you a huge
amount. You don't know how the kernel and C library are interacting to
share memory management responsibility. You should use valgrind:

valgrind --tool=memcheck --leak-check=full myServerCommand

That will tell you if it's really leaking memory. Don't worry about
warnings from valgrind about omniORB transmitting uninitialised memory
-- that's just GIOP padding bytes that don't need to be initialised.

Duncan.