Discussion:
[omniORB] OMNIORB_ASSERT(num_active_oas == 0); with omniORB 4.1.2 and 4.1.4
Martin B.
2011-01-10 17:40:34 UTC
Permalink
Hi all, Hi Duncan,

Today I got OMNIORB_ASSERT(num_active_oas == 0) for the 3rd time within
one week on my development machine. Sadly logging isn't upped, but at
least I have configured VS to immediately stop at any
omniORB::fatalException ... so I'll try to give as much details as possible:

* Last week it happened with 4.1.2 but I'm now using 4.1.4
* Debug Build with Visual C++ 2005
* Pre-Built Binaries omniORB414_vc8_rtd.dll
* Windows XP sp3
* local connection
* Multithreaded MFC GUI app.
* ORB_init and shutdown(true) are called from the main (=GUI) thread
* The ORB::run() function is called from a separate thread (not the main
thread)

The error occurs during ORB shutdown which is initiated by the main
(GUI) thread which is the same thread that also called ORB_init.
All CORBA connections are already terminated during shutdown (that is,
the only client I had connected was already disconnected several seconds
before ORB shutdown is called.

objectadapter.cc : 509 - OMNIORB_ASSERT(num_active_oas == 0);

Find all omniORB related threads+callstacks at the end!

My observations:
* I call shutdown(true) from the main thread (#380), while the run()
function is running from a separate thread (#2588)
* There seem to be several worker threads: omniAsyncWorker::run : #3068,
#1372, #456, #4608
* Thread #5840 shows the destruction of one of the POAs I used. (Which
is correct, as I shut down this POA by calling destroy(true, false)
+ Note that there should've been 2 POAs involved in this test run:
the omniINS POA, and a newly created POA that used the RootPOA a parent.
+ Inspecting the POA object under destruction confirms it's the POA I
created and not the INS POA.
* From the state my process is in at the moment, it is clear the the
ASSERT is triggered when:
+ omniObjAdapter::shutdown() (called from
omniOrbORB::actual_shutdown()) is holding the lock sync(oa_lock) and the
POA shutdown thread #5840 is waiting for this lock to shut down the POA.
* What is weird is that omniOrbPOA::shutdown() (in
omniOrbORB::actual_shutdown()) should have waited for all object
adapters to shut down, but apparently this did not work.
* Maybe it's a similar timing problem to the one I asked about in my
mail from 05.01.2011 ->
http://www.omniorb-support.com/pipermail/omniorb-list/2011-January/030868.html

Note that my shutdown sequence (in thread 380) goes like this:
1) deactivate_object(..) of my main object in the omniINS POA
2) PortableServer::POAManager_var pman = other_poa->the_POAManager();
pman->deactivate(true, /*wait=*/false);
other_poa->destroy(true, /*wait=*/false);
other_poa = PortableServer::POA::_nil();
the_orb->shutdown(true); <<< assert is triggered here

The other_poa POA was created with these options:
root_poa->create_lifespan_policy(TRANSIENT);
root_poa->create_id_assignment_policy(SYSTEM_ID);
root_poa->create_id_uniqueness_policy(UNIQUE_ID);
root_poa->create_implicit_activation_policy(IMPLICIT_ACTIVATION); //
!= default
root_poa-> create_request_processing_policy(USE_ACTIVE_OBJECT_MAP_ONLY);
root_poa->create_servant_retention_policy(RETAIN);
root_poa->create_thread_policy(ORB_CTRL_MODEL);

***
Since this is pretty non-reproducible (read: I started and stopped the
app under the debugger dozens and dozens of times during the last week)
it would be great if I could get some tips how to influence omniORBs
timing behaviour to repro this behaviour.
***

cheers,
Martin

Calls stack follow for reference:

# 380 wWinMainCRTStartup _CxxThrowException Normal 0
- - - - -
omniORB414_vc8_rtd.dll!omni::assertFail(const char * file=0x67c809dc,
int line=509, const char * expr=0x67c809c8) Line 1300 + 0x46 bytes C++
omniORB414_vc8_rtd.dll!omni::omniObjAdapter::shutdown() Line 509 +
0x1d bytes C++
omniORB414_vc8_rtd.dll!omniOrbORB::actual_shutdown() Line 1020 C++
omniORB414_vc8_rtd.dll!omniOrbORB::do_shutdown(bool
wait_for_completion=true) Line 1078 C++
omniORB414_vc8_rtd.dll!omniOrbORB::shutdown(bool
wait_for_completion=true) Line 871 C++
my.dll!ShutdownDmORB() Line 214 C++
my.dll!StopDatamanager() Line 2477 + 0xd bytes C++
my.exe!CMainFrame::OnClose() Line 856 C++
mfc80ud.dll!CWnd::OnWndMsg(unsigned int message=16, unsigned int
wParam=0, long lParam=0, long * pResult=0x0013ee14) Line 2028 C++
...
mfc80ud.dll!CWinThread::PumpMessage() Line 896 C++
mfc80ud.dll!CWinThread::Run() Line 625 + 0xd bytes C++
mfc80ud.dll!CWinApp::Run() Line 894 C++
mfc80ud.dll!AfxWinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ *
hPrevInstance=0x00000000, wchar_t * lpCmdLine=0x00020d8e, int
nCmdShow=1) Line 47 + 0xd bytes C++
my.exe!wWinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ *
hPrevInstance=0x00000000, wchar_t * lpCmdLine=0x00020d8e, int
nCmdShow=1) Line 33 C++
my.exe!__tmainCRTStartup() Line 589 + 0x35 bytes C
my.exe!wWinMainCRTStartup() Line 414 C
kernel32.dll!***@4() + 0x23 bytes
- - - - -

# 2588 _threadstartex omni_condition::wait Normal 0
- - - - -
omnithread34_vc8_rtd.dll!omni_condition::wait() Line 187 + 0x16 bytes C++
omniORB414_vc8_rtd.dll!omniOrbORB::run() Line 823 + 0x16 bytes C++
my.dll!run_thread_fn(void * param=0x02e991b8) Line 315 + 0x23 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -

# 3068 _threadstartex omni::SocketCollection::Select Normal 0
- - - - -
omniORB414_vc8_rtd.dll!omni::SocketCollection::Select() Line 929 +
0x19 bytes C++
omniORB414_vc8_rtd.dll!omni::tcpEndpoint::AcceptAndMonitor(void (void *,
omni::giopConnection *)* func=0x67b64f02, void * cookie=0x02ea22e0)
Line 721 + 0xb bytes C++
omniORB414_vc8_rtd.dll!omni::giopRendezvouser::execute() Line 96 +
0x1e bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::real_run() Line 236 + 0x14
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorkerInfo::run() Line 285 C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::run(void * __formal=0x00000000)
Line 164 C++
omnithread34_vc8_rtd.dll!omni_thread_wrapper(void * ptr=0x02ea2338)
Line 502 + 0x16 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -

# 1372 _threadstartex omni_condition::timedwait Normal 0
- - - - -
omnithread34_vc8_rtd.dll!omni_condition::timedwait(unsigned long
abs_sec=1294654111, unsigned long abs_nsec=322000000) Line 230 + 0x18
bytes C++
omniORB414_vc8_rtd.dll!omni::Scavenger::execute() Line 736 + 0x16 bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::real_run() Line 236 + 0x14
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorkerInfo::run() Line 285 C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::run(void * __formal=0x00000000)
Line 164 C++
omnithread34_vc8_rtd.dll!omni_thread_wrapper(void * ptr=0x0a0f6df0)
Line 502 + 0x16 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -

# 456 _threadstartex omni_condition::timedwait Normal 0
- - - - -
omnithread34_vc8_rtd.dll!omni_condition::timedwait(unsigned long
abs_sec=1294654115, unsigned long abs_nsec=760000000) Line 230 + 0x18
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::real_run() Line 197 + 0x16
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorkerInfo::run() Line 285 C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::run(void * __formal=0x00000000)
Line 164 C++
omnithread34_vc8_rtd.dll!omni_thread_wrapper(void * ptr=0x0b62c468)
Line 502 + 0x16 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -

# 4608 _threadstartex omni_condition::timedwait Normal 0
- - - - -
omnithread34_vc8_rtd.dll!omni_condition::timedwait(unsigned long
abs_sec=1294654115, unsigned long abs_nsec=760000000) Line 230 + 0x18
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::real_run() Line 197 + 0x16
bytes C++
omniORB414_vc8_rtd.dll!omniAsyncWorkerInfo::run() Line 285 C++
omniORB414_vc8_rtd.dll!omniAsyncWorker::run(void * __formal=0x00000000)
Line 164 C++
omnithread34_vc8_rtd.dll!omni_thread_wrapper(void * ptr=0x0b635d70)
Line 502 + 0x16 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -

# 5840 _threadstartex omni_mutex::lock Normal 0
- - - - -
omnithread34_vc8_rtd.dll!omni_mutex::lock() Line 237 + 0x1b bytes C++
omnithread34_vc8_rtd.dll!omni_mutex_lock::omni_mutex_lock(omni_mutex &
m={...}) Line 281 + 0x20 bytes C++
omniORB414_vc8_rtd.dll!omni::omniObjAdapter::adapterInactive() Line 562
+ 0x10 bytes C++
omniORB414_vc8_rtd.dll!omni::omniOrbPOA::do_destroy(bool
etherealize_objects=true) Line 2513 + 0xf bytes C++
omniORB414_vc8_rtd.dll!destroyer_thread_fn(void * args=0x09edfed0)
Line 4169 C++
omnithread34_vc8_rtd.dll!omni_thread_wrapper(void * ptr=0x0a0f5fc8)
Line 492 + 0x11 bytes C++
msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
- - - - -
Duncan Grisby
2011-02-04 23:40:30 UTC
Permalink
Post by Martin B.
The error occurs during ORB shutdown which is initiated by the main
(GUI) thread which is the same thread that also called ORB_init.
All CORBA connections are already terminated during shutdown (that is,
the only client I had connected was already disconnected several seconds
before ORB shutdown is called.
objectadapter.cc : 509 - OMNIORB_ASSERT(num_active_oas == 0);
The problem is that a POA is considered destroyed slightly before it is
really destroyed, meaning there is a race condition if you manually
destroy a POA then shutdown the ORB while the POA is still being
destroyed.

[...]
Post by Martin B.
1) deactivate_object(..) of my main object in the omniINS POA
2) PortableServer::POAManager_var pman = other_poa->the_POAManager();
pman->deactivate(true, /*wait=*/false);
other_poa->destroy(true, /*wait=*/false);
other_poa = PortableServer::POA::_nil();
the_orb->shutdown(true); <<< assert is triggered here
That's a really complex shutdown sequence. Why not just call
orb->shutdown(true) ? The problem is that other_poa->destroy() is still
running in a separate thread at the time orb->shutdown() starts. The
situation is made even more complex because pman->deactivate() and
other_poa->destroy() are busy clashing with each other as they try to do
some of the same things.

The best way to avoid the problem is to remove the concurrent complexity
of your shutdown sequence.

It's infeasible to avoid the race that's causing the assertion failure
without making the concurrency control much more complex than it already
is, so the best way to avoid the assertion will be to make it loop with
a small sleep if num_active_oas != 0.

Cheers,

Duncan.
--
-- Duncan Grisby --
-- ***@grisby.org --
-- http://www.grisby.org --
Loading...