[DynInst_API:] Dyninst abort on running GEMM on the Power9 cluster

Discussion:

Jim Galarowicz

2018-10-26 15:58:50 UTC

Hi all,

Posting again without the library attached - the first email went to the
moderator list.

Jim G

Hi Dyninst team,
Here is the abort and output from running GEMM (mpi shoc cuda
benchmark) with our CUDA experiment osscuda.
It seems like Dyninst is trying to get a Dwarf handle on this file:Â
/usr/lib64/libcuda.so.410.48 when we abort.
This is from using Dyninst master top of tree from October 23, 2018.Â
When I go back to the September 15, 2018 top of tree I do not see the
abort.
That seems to indicate that it is a Dyninst issue, so I'm sending this
in because our spack build of O|SS depends on your top of tree, until
10.0 is released.
Let me know if you need more information.Â Â I attached the file above
- gzipped.
Thanks,
Jim G
Here is the output from the run and the core traceback information.
unlimited
[openss]: cuda counting all instructions for CPU and GPU.
[openss]: cuda using default periodic sampling rate (10 ms).
"interval=10000000,PAPI_TOT_INS,inst_executed"
Creating topology file for frontend host p9-4V100-01
Generated topology file: ./cbtfAutoTopology
Running cuda collector.
Program: mpirun -np 2 ./GEMM
Number of mrnet backends: 2
Topology file used: ./cbtfAutoTopology
executing mpi program: mpirun -np 2Â cbtfrunÂ --mpiÂ --mrnet -c cuda
./GEMM
MPI Task 0/1 starting....
MPI Task 1/1 starting....
Chose device: name='Tesla V100-SXM2-16GB' index=0
Chose device: name='Tesla V100-SXM2-16GB' index=1
Running single precision test
Running single precision test
Running double precision test
Running double precision test
testÂ Â Â attsÂ Â Â unitsÂ Â Â medianÂ Â Â meanÂ Â Â stddevÂ Â Â min max
DGEMM-N(max)Â Â Â 128Â Â Â GFlopsÂ Â Â 9.69694Â Â Â 9.69694 2.91311Â Â Â
6.78383Â Â Â 12.61
DGEMM-N(mean)Â Â Â 128Â Â Â GFlopsÂ Â Â 6.3096Â Â Â 6.3096 0.64253Â Â Â
5.66707Â Â Â 6.95213
DGEMM-N(median)Â Â Â 128Â Â Â GFlopsÂ Â Â 6.44043Â Â Â 6.44043 0.954308Â Â Â
5.48612Â Â Â 7.39473
DGEMM-N(min)Â Â Â 128Â Â Â GFlopsÂ Â Â 2.74011Â Â Â 2.74011 1.78164Â Â Â
0.958474Â Â Â 4.52175
DGEMM-N(stddev)Â Â Â 128Â Â Â GFlopsÂ Â Â 2.12004Â Â Â 2.12004 1.32167Â Â Â
0.798364Â Â Â 3.44171
DGEMM-N_PCIe(max)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.986637Â Â Â 0.986637 0.382143Â Â Â
0.604494Â Â Â 1.36878
DGEMM-N_PCIe(mean)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.883682Â Â Â 0.883682
0.290739Â Â Â 0.592943Â Â Â 1.17442
DGEMM-N_PCIe(median)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.931726 0.931726Â Â Â
0.339714Â Â Â 0.592012Â Â Â 1.27144
DGEMM-N_PCIe(min)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.584403Â Â Â 0.584403
0.00570615Â Â Â 0.578697Â Â Â 0.590109
DGEMM-N_PCIe(stddev)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.119548 0.119548Â Â Â
0.110633Â Â Â 0.00891468Â Â Â 0.230181
DGEMM-N_Parity(max)Â Â Â 128Â Â Â NÂ Â Â 9.21747Â Â Â 9.21747 1.00485Â Â Â
8.21262Â Â Â 10.2223
DGEMM-N_Parity(mean)Â Â Â 128Â Â Â NÂ Â Â 6.53364Â Â Â 6.53364 2.00588Â Â Â
4.52775Â Â Â 8.53952
DGEMM-N_Parity(median)Â Â Â 128Â Â Â NÂ Â Â 6.54143Â Â Â 6.54143 1.72542Â Â Â
4.81601Â Â Â 8.26685
DGEMM-N_Parity(min)Â Â Â 128Â Â Â NÂ Â Â 3.71895Â Â Â 3.71895 3.09472Â Â Â
0.624231Â Â Â 6.81368
DGEMM-N_Parity(stddev)Â Â Â 128Â Â Â NÂ Â Â 1.72226Â Â Â 1.72226 0.519237Â Â Â
1.20303Â Â Â 2.2415
DGEMM-T(max)Â Â Â 128Â Â Â GFlopsÂ Â Â 8.73437Â Â Â 8.73437 1.89898Â Â Â
6.83539Â Â Â 10.6334
DGEMM-T(mean)Â Â Â 128Â Â Â GFlopsÂ Â Â 5.84512Â Â Â 5.84512 0.200883Â Â Â
5.64424Â Â Â 6.04601
DGEMM-T(median)Â Â Â 128Â Â Â GFlopsÂ Â Â 6.43896Â Â Â 6.43896 0.641068Â Â Â
5.7979Â Â Â 7.08003
DGEMM-T(min)Â Â Â 128Â Â Â GFlopsÂ Â Â 2.19582Â Â Â 2.19582 2.05793Â Â Â
0.137885Â Â Â 4.25375
DGEMM-T(stddev)Â Â Â 128Â Â Â GFlopsÂ Â Â 2.08324Â Â Â 2.08324 1.28537Â Â Â
0.797875Â Â Â 3.36861
DGEMM-T_PCIe(max)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.973304Â Â Â 0.973304 0.368403Â Â Â
0.604901Â Â Â 1.34171
DGEMM-T_PCIe(mean)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.837045Â Â Â 0.837045
0.244427Â Â Â 0.592618Â Â Â 1.08147
DGEMM-T_PCIe(median)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.928633 0.928633Â Â Â
0.333162Â Â Â 0.59547Â Â Â 1.2618
DGEMM-T_PCIe(min)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.350296Â Â Â 0.350296 0.223772Â Â Â
0.126523Â Â Â 0.574068
DGEMM-T_PCIe(stddev)Â Â Â 128Â Â Â GFlopsÂ Â Â 0.186293 0.186293Â Â Â
0.176799Â Â Â 0.00949395Â Â Â 0.363091
DGEMM-T_Parity(max)Â Â Â 128Â Â Â NÂ Â Â 8.61263Â Â Â 8.61263 1.68739Â Â Â
6.92524Â Â Â 10.3
DGEMM-T_Parity(mean)Â Â Â 128Â Â Â NÂ Â Â 6.22137Â Â Â 6.22137 2.28375Â Â Â
3.93762Â Â Â 8.50512
DGEMM-T_Parity(median)Â Â Â 128Â Â Â NÂ Â Â 6.67385Â Â Â 6.67385 2.0628Â Â Â
4.61105Â Â Â 8.73666
DGEMM-T_Parity(min)Â Â Â 128Â Â Â NÂ Â Â 3.24982Â Â Â 3.24982 3.16002Â Â Â
0.0898014Â Â Â 6.40984
DGEMM-T_Parity(stddev)Â Â Â 128Â Â Â NÂ Â Â 1.69809Â Â Â 1.69809 0.495801Â Â Â
1.20229Â Â Â 2.19389
SGEMM-N(max)Â Â Â 256Â Â Â GFlopsÂ Â Â 62.1606Â Â Â 62.1606 7.55859Â Â Â
54.602Â Â Â 69.7191
SGEMM-N(mean)Â Â Â 256Â Â Â GFlopsÂ Â Â 47.9714Â Â Â 47.9714 1.68053Â Â Â
46.2909Â Â Â 49.652
SGEMM-N(median)Â Â Â 256Â Â Â GFlopsÂ Â Â 54.0888Â Â Â 54.0888 6.28268Â Â Â
47.8061Â Â Â 60.3715
SGEMM-N(min)Â Â Â 256Â Â Â GFlopsÂ Â Â 20.6876Â Â Â 20.6876 6.84074Â Â Â
13.8469Â Â Â 27.5284
SGEMM-N(stddev)Â Â Â 256Â Â Â GFlopsÂ Â Â 14.7462Â Â Â 14.7462 6.70088Â Â Â
8.0453Â Â Â 21.4471
SGEMM-N_PCIe(max)Â Â Â 256Â Â Â GFlopsÂ Â Â 10.3257Â Â Â 10.3257 1.85213Â Â Â
8.47362Â Â Â 12.1779
SGEMM-N_PCIe(mean)Â Â Â 256Â Â Â GFlopsÂ Â Â 9.66712Â Â Â 9.66712 1.9606Â Â Â
7.70653Â Â Â 11.6277
SGEMM-N_PCIe(median)Â Â Â 256Â Â Â GFlopsÂ Â Â 10.0591Â Â Â 10.0591 1.74278Â Â Â
8.31632Â Â Â 11.8019
SGEMM-N_PCIe(min)Â Â Â 256Â Â Â GFlopsÂ Â Â 7.83632Â Â Â 7.83632 2.1509Â Â Â
5.68542Â Â Â 9.98722
SGEMM-N_PCIe(stddev)Â Â Â 256Â Â Â GFlopsÂ Â Â 0.840406 0.840406Â Â Â
0.206688Â Â Â 0.633718Â Â Â 1.04709
SGEMM-N_Parity(max)Â Â Â 256Â Â Â NÂ Â Â 5.35575Â Â Â 5.35575 1.87205Â Â Â
3.4837Â Â Â 7.22779
SGEMM-N_Parity(mean)Â Â Â 256Â Â Â NÂ Â Â 4.05043Â Â Â 4.05043 1.09699Â Â Â
2.95344Â Â Â 5.14743
SGEMM-N_Parity(median)Â Â Â 256Â Â Â NÂ Â Â 4.65442Â Â Â 4.65442 1.6043Â Â Â
3.05012Â Â Â 6.25872
SGEMM-N_Parity(min)Â Â Â 256Â Â Â NÂ Â Â 1.59593Â Â Â 1.59593 0.160425Â Â Â
1.43551Â Â Â 1.75636
SGEMM-N_Parity(stddev)Â Â Â 256Â Â Â NÂ Â Â 1.36836Â Â Â 1.36836 0.855058Â Â Â
0.513304Â Â Â 2.22342
SGEMM-T(max)Â Â Â 256Â Â Â GFlopsÂ Â Â 63.4329Â Â Â 63.4329 10.5459Â Â Â
52.8869Â Â Â 73.9788
SGEMM-T(mean)Â Â Â 256Â Â Â GFlopsÂ Â Â 45.9749Â Â Â 45.9749 0.620777Â Â Â
45.3541Â Â Â 46.5957
SGEMM-T(median)Â Â Â 256Â Â Â GFlopsÂ Â Â 49.8315Â Â Â 49.8315 4.88943Â Â Â
44.942Â Â Â 54.7209
SGEMM-T(min)Â Â Â 256Â Â Â GFlopsÂ Â Â 19.9912Â Â Â 19.9912 18.8349Â Â Â
1.15635Â Â Â 38.8261
SGEMM-T(stddev)Â Â Â 256Â Â Â GFlopsÂ Â Â 13.8137Â Â Â 13.8137 9.37892Â Â Â
4.43478Â Â Â 23.1926
SGEMM-T_PCIe(max)Â Â Â 256Â Â Â GFlopsÂ Â Â 10.3119Â Â Â 10.3119 1.77855Â Â Â
8.53333Â Â Â 12.0904
SGEMM-T_PCIe(mean)Â Â Â 256Â Â Â GFlopsÂ Â Â 9.42068Â Â Â 9.42068 2.20647Â Â Â
7.2142Â Â Â 11.6272
SGEMM-T_PCIe(median)Â Â Â 256Â Â Â GFlopsÂ Â Â 9.90605Â Â Â 9.90605 1.71467Â Â Â
8.19137Â Â Â 11.6207
SGEMM-T_PCIe(min)Â Â Â 256Â Â Â GFlopsÂ Â Â 6.09928Â Â Â 6.09928 5.06671Â Â Â
1.03257Â Â Â 11.166
SGEMM-T_PCIe(stddev)Â Â Â 256Â Â Â GFlopsÂ Â Â 1.24072Â Â Â 1.24072
0.950884Â Â Â 0.289832Â Â Â 2.1916
SGEMM-T_Parity(max)Â Â Â 256Â Â Â NÂ Â Â 5.52184Â Â Â 5.52184 2.14756Â Â Â
3.37428Â Â Â 7.66939
SGEMM-T_Parity(mean)Â Â Â 256Â Â Â NÂ Â Â 3.86213Â Â Â 3.86213 0.968453Â Â Â
2.89367Â Â Â 4.83058
SGEMM-T_Parity(median)Â Â Â 256Â Â Â NÂ Â Â 4.27015Â Â Â 4.27015 1.40277Â Â Â
2.86738Â Â Â 5.67292
SGEMM-T_Parity(min)Â Â Â 256Â Â Â NÂ Â Â 1.29853Â Â Â 1.29853 1.17865Â Â Â
0.119879Â Â Â 2.47717
SGEMM-T_Parity(stddev)Â Â Â 256Â Â Â NÂ Â Â 1.34366Â Â Â 1.34366 1.06072Â Â Â
0.282947Â Â Â 2.40438
Note: results marked with (*) had missing values such as
might occur with a mixture of architectural capabilities.
Detecting outliers based on per-process mean values.
None.
None.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig for a
thread.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig for a
thread.
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscuda: line 1914:Â 1803
Segmentation faultÂ Â Â Â Â (core dumped) osscollect $topology_opt
$cbtf_offline_opt --program "$1" --collector $collector
232290224.754090: Message.c[278] Message_send - MRN_send() failed
232290224.754194: PeerNode.c[176] PeerNode_sendDirectly -
Message_send() failed
BE: receive failure
232290224.754211: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754228: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
232290224.754328: Message.c[278] Message_send - MRN_send() failed
232290224.754400: PeerNode.c[176] PeerNode_sendDirectly -
Message_send() failed
232290224.754416: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754432: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
^C
total 105636
Â Â Â 0 drwxr-xr-x. 3 jeg openssÂ Â Â Â Â Â 18 Oct 25 09:46 ..
Â 2312 -rwxr-xr-x. 1 jeg openssÂ 2366680 Oct 25 10:14 BusSpeedDownload
Â 2476 -rwxr-xr-x. 1 jeg openssÂ 2533880 Oct 25 10:14 DeviceMemory
Â 2316 -rwxr-xr-x. 1 jeg openssÂ 2368472 Oct 25 10:14 BusSpeedReadback
Â 3736 -rwxr-xr-x. 1 jeg openssÂ 3824448 Oct 25 10:14 MaxFlops
Â 2948 -rwxr-xr-x. 1 jeg openssÂ 3017920 Oct 25 10:14 BFS
Â 2792 -rwxr-xr-x. 1 jeg openssÂ 2858208 Oct 25 10:14 FFT
Â 2620 -rwxr-xr-x. 1 jeg openssÂ 2681720 Oct 25 10:14 GEMM
Â 2608 -rwxr-xr-x. 1 jeg openssÂ 2669440 Oct 25 10:14 MD
Â 2392 -rwxr-xr-x. 1 jeg openssÂ 2445944 Oct 25 10:14 Reduction
Â 2488 -rwxr-xr-x. 1 jeg openssÂ 2544208 Oct 25 10:14 Scan
Â 2356 -rwxr-xr-x. 1 jeg openssÂ 2410744 Oct 25 10:14 Sort
Â 2820 -rwxr-xr-x. 1 jeg openssÂ 2883792 Oct 25 10:14 Spmv
Â 2436 -rwxr-xr-x. 1 jeg openssÂ 2491968 Oct 25 10:14 Triad
Â 2504 -rwxr-xr-x. 1 jeg openssÂ 2560960 Oct 25 10:14 MD5Hash
Â 2444 -rwxr-xr-x. 1 jeg openssÂ 2500432 Oct 25 10:14 NeuralNet
Â 4512 -rwxr-xr-x. 1 jeg openssÂ 4617400 Oct 25 10:14 S3D
Â 2452 -rwxr-xr-x. 1 jeg openssÂ 2508096 Oct 25 10:14 BusCont
Â 2496 -rwxr-xr-x. 1 jeg openssÂ 2551920 Oct 25 10:14 MTBusCont
Â 2588 -rwxr-xr-x. 1 jeg openssÂ 2649928 Oct 25 10:14 Stability_mpi
Â 444 -rw-r--r--. 1 jeg openssÂ Â 451584 Oct 25 11:54 GEMM-cuda-0.openss
Â Â Â 4 -rw-r--r--. 1 jeg openssÂ Â Â Â Â Â 34 Oct 25 12:02 cbtfAutoTopology
Â Â Â 4 -rw-r--r--. 1 jeg openssÂ Â Â Â Â Â 44 Oct 25 12:02 attachBE_connections
Â 524 -rw-r--r--. 1 jeg openssÂ Â 533504 Oct 25 12:03 GEMM-cuda-1.openss
Â Â Â 4 drwxr-xr-x. 2 jeg openssÂ Â Â Â 4096 Oct 25 12:03 .
53360 -rw-------. 1 jeg openss 97779712 Oct 25 12:03 core.1803
core.1803: ELF 64-bit LSB core file 64-bit PowerPC or cisco 7500,
version 1 (SYSV), SVR4-style, from 'osscollect --program mpirun -np 2
./GEMM --collector cuda', real uid: 1001, effective uid: 1001, real
'/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect', platform: 'power9'
15536 -rwxr-xr-x. 1 root root 15906448 SepÂ 6
05:20*/usr/lib64/libcuda.so.410.48*
15536 -rwxr-xr-x. 1 root root 15906448 SepÂ 6 05:20
/usr/lib64/libcuda.so.410.48
Â Â Â 0 lrwxrwxrwx. 1 root rootÂ Â Â Â Â Â 17 OctÂ 9 15:49
/usr/lib64/libcuda.so.1 -> libcuda.so.410.48
Â Â Â 0 lrwxrwxrwx. 1 root rootÂ Â Â Â Â Â 17 OctÂ 9 15:49
/usr/lib64/libcuda.so -> libcuda.so.410.48
Â 544 -rw-rw-rw-. 1 root rootÂ 554512 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
1152 -rw-rw-rw-. 1 root root 1178786 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart_static.a
Â 680 -rw-rw-rw-. 1 root rootÂ 694236 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudadevrt.a
Â Â 0 lrwxrwxrwx. 1 root rootÂ Â Â Â Â 21 OctÂ 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so.10.0 -> libcudart.so.10.0.130
Â Â 0 lrwxrwxrwx. 1 root rootÂ Â Â Â Â 17 OctÂ 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so -> libcudart.so.10.0
Â Â Â linux-vdso64.so.1 =>Â (0x00007fff87e50000)
Â Â Â libm.so.6 => /lib64/libm.so.6 (0x00007fff86de0000)
Â Â Â libc.so.6 => /lib64/libc.so.6 (0x00007fff86bf0000)
Â Â Â libdl.so.2 => /lib64/libdl.so.2 (0x00007fff86bc0000)
Â Â Â libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fff86b80000)
Â Â Â librt.so.1 => /lib64/librt.so.1 (0x00007fff86b50000)
Â Â Â libnvidia-fatbinaryloader.so.410.48 =>
/lib64/libnvidia-fatbinaryloader.so.410.48 (0x00007fff86ae0000)
Â Â Â /lib64/ld64.so.2 (0x00007fff87e70000)
/usr/lib64/libcuda.so.410.48: ELF 64-bit LSB shared object, 64-bit
PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, stripped
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.Â Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect...done.
[New LWP 1909]
[New LWP 1828]
[New LWP 1820]
[New LWP 1830]
[New LWP 1803]
[New LWP 1826]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun -np 2 ./GEMM
--collector cuda '.
Program terminated with signal 11, Segmentation fault.
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716Â Â Â Â Â Â Â Â { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1Â compare (__str=..., this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2Â operator< <char, std::char_traits<char>, std::allocator<char> >
(__rhs=..., __lhs=<error reading variable: Cannot access memory at
address 0xffffffffffffffe8>)
Â Â Â at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=..., __x=<error reading
variable: Cannot access memory at address 0xffffffffffffffe8>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff89c283f0) at /usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_map.h:822
*#7 Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="**/usr/lib64/libcuda.so.410.48**", file_=0x7fff89e07740)*
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fffa65d29e8 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff89e06ef0, mf_=0x7fff89c24f10, err_func=<optimized out>,
alloc_syms=<optimized out>,
Â Â Â st=<optimized out>) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Object-elf.C:2986
#9Â 0x00007fffa6577f24 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff89e06060, filename="", defensive_bin=<optimized out>,
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:1261
#10 0x00007fffa657838c in Dyninst::SymtabAPI::Symtab::openFile
def_binary=<optimized out>)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:2077
#11 0x00007fffa673cdfc in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols
(this=<optimized out>, addresses=..., linkedobject=..., stm=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fffa680d188 in OpenSpeedShop::Framework::Callbacks::finalize ()
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fffa680eab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fffa68331e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:118
#15 0x00007fffa685f7c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, 0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (
value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&, 1ul> (
value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (connectionBody=..., this=0x7fffa10dd7a0)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
---Type <return> to continue, or q <return> to quit---
#19 dereference (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (f=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/optional_last_value.hpp:57
#23 operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., combiner=...,
this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&),
boost::signals2::optional_last_value<void>, int, std::less<int>,
boost::function<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::function<void (boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>,
Â Â Â args#0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fffa685f974 in operator() (args#0=..., this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::valueHandler (this=<optimized out>, value=...)

Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fffa6833734 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,

boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (a1=...,
this=<optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>,

boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

*>, boost::arg<1> > >, void,

boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_obj_ptr=...,
---Type <return> to continue, or q <return> to quit---
Â Â Â a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#31 0x00007fffa68432ec in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::operator() (

Â Â Â this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fffa643ec48 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fffa63daf38 in call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const,
boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized
out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&,
const boost::any&>, boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable: value has been
optimized out>,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >,
void, boost::any const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#41 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fffa643ec48 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fffa643f308 in call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const,
boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized
out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
---Type <return> to continue, or q <return> to quit---
#47 operator()<KrellInstitute::CBTF::Impl::Network*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&,
const boost::any&>, boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable: value has been
optimized out>,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >,
void, boost::any const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#51 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fffa2bd0fa0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> >
(value=..., name="LinkedObjectGroup", this=<optimized out>)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff9c02e380, message=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fffa2bd1c94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void, StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> (a1=...,
this=<optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>::invoke
(function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#61 0x00007fffa2bd342c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>

::operator() (

Â Â Â this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fffa643ecc8 in emitOutput (value=..., type=...,
name="value", this=0x7fff9c02a700)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff9c02a700, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fffa643db94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (
Â Â Â a=<synthetic pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, KrellInstitute::CBTF::Impl::InputMediator,
boost::any const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any const&>::invoke
(function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#71 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fffa2be0f44 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> >
(value=<synthetic pointer>, name="out", this=0x7fff9c0076f0)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff9c0076f0, in=...)
Â Â Â at /opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fffa2bd67a4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#79 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet>
const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#81 0x00007fffa63e357c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>

::operator() (this=<error reading variable: value has been optimized

out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fffa63c9494 in emitOutput<boost::shared_ptr<MRN::Packet> >
(value=..., name="value", this=0x7fff9c0072d0)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff9c0072d0, packet=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fffa63dafe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=...,
Â Â Â this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

(a1=..., u=..., this=<optimized out>)

Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator, const
boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#90 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

, boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet>

const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#92 0x00007fffa63d1460 in operator() (a0=..., this=0x7fff8802c6a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fffa63bab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump (this=0x7fff9c002a80)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fffa63bcea4 in operator() (p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0>
(a=<synthetic pointer>, f=..., this=<optimized out>)
Â Â Â at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void, KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>

::run (this=<optimized out>) at

/opt/tools/OSS/krellroot_v2.4.0/include/boost/thread/detail/thread.hpp:117
#99 0x00007fffa73323b0 in thread_proxy () from
/opt/tools/OSS/krellroot_v2.4.0/lib/libboost_thread.so.1.67.0
#100 0x00007fffa66f8af4 in start_thread (arg=0x7fffa10df170) at
pthread_create.c:310
#101 0x00007fffa5e78814 in clone () at
../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109
(gdb)
(gdb)

Xiaozhu Meng

2018-10-26 17:17:09 UTC

Permalink

Hi Jim,

Can you try with the latest Dyninst master? It looks like the crash is
during Symtab::openFile based your stack trace. But, on my end with the
latest master branch, I can open the library file provided by you using
Symtab and perform other analysis.

By the way, the master branch now has the parallel parsing merged. So, it
will need two more dependencies (tbb and openmp). Dyninst will
automatically download and install tbb and will use openmp by default. The
master branch also has the two planed API breaking change:

1. InstructionAPI::Instruction is now passed by value rather than boost
smart pointer. So, all Instruction::Ptr i should be replaced as
Instruction i.
2. Operation.h is renamed as Operation_impl.h.

Let me know if you encounter any problem with the latest master branch.

Thanks,
--Xiaozhu

Post by Jim Galarowicz
Hi all,
Posting again without the library attached - the first email went to the
moderator list.
Jim G
Hi Dyninst team,
Here is the abort and output from running GEMM (mpi shoc cuda benchmark)
with our CUDA experiment osscuda.
/usr/lib64/libcuda.so.410.48 when we abort.
This is from using Dyninst master top of tree from October 23, 2018. When
I go back to the September 15, 2018 top of tree I do not see the abort.
That seems to indicate that it is a Dyninst issue, so I'm sending this in
because our spack build of O|SS depends on your top of tree, until 10.0 is
released.
Let me know if you need more information. I attached the file above -
gzipped.
Thanks,
Jim G
Here is the output from the run and the core traceback information.
unlimited
[openss]: cuda counting all instructions for CPU and GPU.
[openss]: cuda using default periodic sampling rate (10 ms).
"interval=10000000,PAPI_TOT_INS,inst_executed"
Creating topology file for frontend host p9-4V100-01
Generated topology file: ./cbtfAutoTopology
Running cuda collector.
Program: mpirun -np 2 ./GEMM
Number of mrnet backends: 2
Topology file used: ./cbtfAutoTopology
executing mpi program: mpirun -np 2 cbtfrun --mpi --mrnet -c cuda
./GEMM
MPI Task 0/1 starting....
MPI Task 1/1 starting....
Chose device: name='Tesla V100-SXM2-16GB' index=0
Chose device: name='Tesla V100-SXM2-16GB' index=1
Running single precision test
Running single precision test
Running double precision test
Running double precision test
test atts units median mean stddev min max
DGEMM-N(max) 128 GFlops 9.69694 9.69694 2.91311
6.78383 12.61
DGEMM-N(mean) 128 GFlops 6.3096 6.3096 0.64253
5.66707 6.95213
DGEMM-N(median) 128 GFlops 6.44043 6.44043 0.954308
5.48612 7.39473
DGEMM-N(min) 128 GFlops 2.74011 2.74011 1.78164
0.958474 4.52175
DGEMM-N(stddev) 128 GFlops 2.12004 2.12004 1.32167
0.798364 3.44171
DGEMM-N_PCIe(max) 128 GFlops 0.986637 0.986637 0.382143
0.604494 1.36878
DGEMM-N_PCIe(mean) 128 GFlops 0.883682 0.883682 0.290739
0.592943 1.17442
DGEMM-N_PCIe(median) 128 GFlops 0.931726 0.931726
0.339714 0.592012 1.27144
DGEMM-N_PCIe(min) 128 GFlops 0.584403 0.584403
0.00570615 0.578697 0.590109
DGEMM-N_PCIe(stddev) 128 GFlops 0.119548 0.119548
0.110633 0.00891468 0.230181
DGEMM-N_Parity(max) 128 N 9.21747 9.21747 1.00485
8.21262 10.2223
DGEMM-N_Parity(mean) 128 N 6.53364 6.53364 2.00588
4.52775 8.53952
DGEMM-N_Parity(median) 128 N 6.54143 6.54143 1.72542
4.81601 8.26685
DGEMM-N_Parity(min) 128 N 3.71895 3.71895 3.09472
0.624231 6.81368
DGEMM-N_Parity(stddev) 128 N 1.72226 1.72226 0.519237
1.20303 2.2415
DGEMM-T(max) 128 GFlops 8.73437 8.73437 1.89898
6.83539 10.6334
DGEMM-T(mean) 128 GFlops 5.84512 5.84512 0.200883
5.64424 6.04601
DGEMM-T(median) 128 GFlops 6.43896 6.43896 0.641068
5.7979 7.08003
DGEMM-T(min) 128 GFlops 2.19582 2.19582 2.05793
0.137885 4.25375
DGEMM-T(stddev) 128 GFlops 2.08324 2.08324 1.28537
0.797875 3.36861
DGEMM-T_PCIe(max) 128 GFlops 0.973304 0.973304 0.368403
0.604901 1.34171
DGEMM-T_PCIe(mean) 128 GFlops 0.837045 0.837045 0.244427
0.592618 1.08147
DGEMM-T_PCIe(median) 128 GFlops 0.928633 0.928633
0.333162 0.59547 1.2618
DGEMM-T_PCIe(min) 128 GFlops 0.350296 0.350296 0.223772
0.126523 0.574068
DGEMM-T_PCIe(stddev) 128 GFlops 0.186293 0.186293
0.176799 0.00949395 0.363091
DGEMM-T_Parity(max) 128 N 8.61263 8.61263 1.68739
6.92524 10.3
DGEMM-T_Parity(mean) 128 N 6.22137 6.22137 2.28375
3.93762 8.50512
DGEMM-T_Parity(median) 128 N 6.67385 6.67385 2.0628
4.61105 8.73666
DGEMM-T_Parity(min) 128 N 3.24982 3.24982 3.16002
0.0898014 6.40984
DGEMM-T_Parity(stddev) 128 N 1.69809 1.69809 0.495801
1.20229 2.19389
SGEMM-N(max) 256 GFlops 62.1606 62.1606 7.55859
54.602 69.7191
SGEMM-N(mean) 256 GFlops 47.9714 47.9714 1.68053
46.2909 49.652
SGEMM-N(median) 256 GFlops 54.0888 54.0888 6.28268
47.8061 60.3715
SGEMM-N(min) 256 GFlops 20.6876 20.6876 6.84074
13.8469 27.5284
SGEMM-N(stddev) 256 GFlops 14.7462 14.7462 6.70088
8.0453 21.4471
SGEMM-N_PCIe(max) 256 GFlops 10.3257 10.3257 1.85213
8.47362 12.1779
SGEMM-N_PCIe(mean) 256 GFlops 9.66712 9.66712 1.9606
7.70653 11.6277
SGEMM-N_PCIe(median) 256 GFlops 10.0591 10.0591 1.74278
8.31632 11.8019
SGEMM-N_PCIe(min) 256 GFlops 7.83632 7.83632 2.1509
5.68542 9.98722
SGEMM-N_PCIe(stddev) 256 GFlops 0.840406 0.840406
0.206688 0.633718 1.04709
SGEMM-N_Parity(max) 256 N 5.35575 5.35575 1.87205
3.4837 7.22779
SGEMM-N_Parity(mean) 256 N 4.05043 4.05043 1.09699
2.95344 5.14743
SGEMM-N_Parity(median) 256 N 4.65442 4.65442 1.6043
3.05012 6.25872
SGEMM-N_Parity(min) 256 N 1.59593 1.59593 0.160425
1.43551 1.75636
SGEMM-N_Parity(stddev) 256 N 1.36836 1.36836 0.855058
0.513304 2.22342
SGEMM-T(max) 256 GFlops 63.4329 63.4329 10.5459
52.8869 73.9788
SGEMM-T(mean) 256 GFlops 45.9749 45.9749 0.620777
45.3541 46.5957
SGEMM-T(median) 256 GFlops 49.8315 49.8315 4.88943
44.942 54.7209
SGEMM-T(min) 256 GFlops 19.9912 19.9912 18.8349
1.15635 38.8261
SGEMM-T(stddev) 256 GFlops 13.8137 13.8137 9.37892
4.43478 23.1926
SGEMM-T_PCIe(max) 256 GFlops 10.3119 10.3119 1.77855
8.53333 12.0904
SGEMM-T_PCIe(mean) 256 GFlops 9.42068 9.42068 2.20647
7.2142 11.6272
SGEMM-T_PCIe(median) 256 GFlops 9.90605 9.90605 1.71467
8.19137 11.6207
SGEMM-T_PCIe(min) 256 GFlops 6.09928 6.09928 5.06671
1.03257 11.166
SGEMM-T_PCIe(stddev) 256 GFlops 1.24072 1.24072 0.950884
0.289832 2.1916
SGEMM-T_Parity(max) 256 N 5.52184 5.52184 2.14756
3.37428 7.66939
SGEMM-T_Parity(mean) 256 N 3.86213 3.86213 0.968453
2.89367 4.83058
SGEMM-T_Parity(median) 256 N 4.27015 4.27015 1.40277
2.86738 5.67292
SGEMM-T_Parity(min) 256 N 1.29853 1.29853 1.17865
0.119879 2.47717
SGEMM-T_Parity(stddev) 256 N 1.34366 1.34366 1.06072
0.282947 2.40438
Note: results marked with (*) had missing values such as
might occur with a mixture of architectural capabilities.
Detecting outliers based on per-process mean values.
None.
None.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig for a thread.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig for a thread.
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscuda: line 1914: 1803 Segmentation
fault (core dumped) osscollect $topology_opt $cbtf_offline_opt
--program "$1" --collector $collector
232290224.754090: Message.c[278] Message_send - MRN_send() failed
232290224.754194: PeerNode.c[176] PeerNode_sendDirectly - Message_send()
failed
BE: receive failure
232290224.754211: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754228: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
232290224.754328: Message.c[278] Message_send - MRN_send() failed
232290224.754400: PeerNode.c[176] PeerNode_sendDirectly - Message_send()
failed
232290224.754416: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754432: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
^C
total 105636
0 drwxr-xr-x. 3 jeg openss 18 Oct 25 09:46 ..
2312 -rwxr-xr-x. 1 jeg openss 2366680 Oct 25 10:14 BusSpeedDownload
2476 -rwxr-xr-x. 1 jeg openss 2533880 Oct 25 10:14 DeviceMemory
2316 -rwxr-xr-x. 1 jeg openss 2368472 Oct 25 10:14 BusSpeedReadback
3736 -rwxr-xr-x. 1 jeg openss 3824448 Oct 25 10:14 MaxFlops
2948 -rwxr-xr-x. 1 jeg openss 3017920 Oct 25 10:14 BFS
2792 -rwxr-xr-x. 1 jeg openss 2858208 Oct 25 10:14 FFT
2620 -rwxr-xr-x. 1 jeg openss 2681720 Oct 25 10:14 GEMM
2608 -rwxr-xr-x. 1 jeg openss 2669440 Oct 25 10:14 MD
2392 -rwxr-xr-x. 1 jeg openss 2445944 Oct 25 10:14 Reduction
2488 -rwxr-xr-x. 1 jeg openss 2544208 Oct 25 10:14 Scan
2356 -rwxr-xr-x. 1 jeg openss 2410744 Oct 25 10:14 Sort
2820 -rwxr-xr-x. 1 jeg openss 2883792 Oct 25 10:14 Spmv
2436 -rwxr-xr-x. 1 jeg openss 2491968 Oct 25 10:14 Triad
2504 -rwxr-xr-x. 1 jeg openss 2560960 Oct 25 10:14 MD5Hash
2444 -rwxr-xr-x. 1 jeg openss 2500432 Oct 25 10:14 NeuralNet
4512 -rwxr-xr-x. 1 jeg openss 4617400 Oct 25 10:14 S3D
2452 -rwxr-xr-x. 1 jeg openss 2508096 Oct 25 10:14 BusCont
2496 -rwxr-xr-x. 1 jeg openss 2551920 Oct 25 10:14 MTBusCont
2588 -rwxr-xr-x. 1 jeg openss 2649928 Oct 25 10:14 Stability_mpi
444 -rw-r--r--. 1 jeg openss 451584 Oct 25 11:54 GEMM-cuda-0.openss
4 -rw-r--r--. 1 jeg openss 34 Oct 25 12:02 cbtfAutoTopology
4 -rw-r--r--. 1 jeg openss 44 Oct 25 12:02 attachBE_connections
524 -rw-r--r--. 1 jeg openss 533504 Oct 25 12:03 GEMM-cuda-1.openss
4 drwxr-xr-x. 2 jeg openss 4096 Oct 25 12:03 .
53360 -rw-------. 1 jeg openss 97779712 Oct 25 12:03 core.1803
core.1803: ELF 64-bit LSB core file 64-bit PowerPC or cisco 7500, version
1 (SYSV), SVR4-style, from 'osscollect --program mpirun -np 2 ./GEMM
--collector cuda', real uid: 1001, effective uid: 1001, real gid: 1001,
'/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect', platform: 'power9'
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20*
/usr/lib64/libcuda.so.410.48*
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20
/usr/lib64/libcuda.so.410.48
0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/lib64/libcuda.so.1 -> libcuda.so.410.48
0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49 /usr/lib64/libcuda.so
-> libcuda.so.410.48
544 -rw-rw-rw-. 1 root root 554512 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
1152 -rw-rw-rw-. 1 root root 1178786 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart_static.a
680 -rw-rw-rw-. 1 root root 694236 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudadevrt.a
0 lrwxrwxrwx. 1 root root 21 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so.10.0 -> libcudart.so.10.0.130
0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so -> libcudart.so.10.0
linux-vdso64.so.1 => (0x00007fff87e50000)
libm.so.6 => /lib64/libm.so.6 (0x00007fff86de0000)
libc.so.6 => /lib64/libc.so.6 (0x00007fff86bf0000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fff86bc0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fff86b80000)
librt.so.1 => /lib64/librt.so.1 (0x00007fff86b50000)
libnvidia-fatbinaryloader.so.410.48 =>
/lib64/libnvidia-fatbinaryloader.so.410.48 (0x00007fff86ae0000)
/lib64/ld64.so.2 (0x00007fff87e70000)
/usr/lib64/libcuda.so.410.48: ELF 64-bit LSB shared object, 64-bit PowerPC
or cisco 7500, version 1 (SYSV), dynamically linked, stripped
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html> <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
<http://www.gnu.org/software/gdb/bugs/>
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect...done.
[New LWP 1909]
[New LWP 1828]
[New LWP 1820]
[New LWP 1830]
[New LWP 1803]
[New LWP 1826]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun -np 2 ./GEMM
--collector cuda '.
Program terminated with signal 11, Segmentation fault.
#0 size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716 { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0 size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1 compare (__str=..., this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2 operator< <char, std::char_traits<char>, std::allocator<char> >
(__rhs=..., __lhs=<error reading variable: Cannot access memory at address
0xffffffffffffffe8>)
at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3 operator() (this=<optimized out>, __y=..., __x=<error reading
variable: Cannot access memory at address 0xffffffffffffffe8>)
at /usr/include/c++/4.8.2/bits/stl_function.h:235
#4 _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff89c283f0) at /usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5 find (__k="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6 find (__x="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
at /usr/include/c++/4.8.2/bits/stl_map.h:822
*#7 Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle (filename_="*
*/usr/lib64/libcuda.so.410.48**", file_=0x7fff89e07740)*
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/dwarf/src/dwarfHandle.C:255
#8 0x00007fffa65d29e8 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff89e06ef0, mf_=0x7fff89c24f10, err_func=<optimized out>,
alloc_syms=<optimized out>,
st=<optimized out>) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Object-elf.C:2986
#9 0x00007fffa6577f24 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff89e06060, filename="", defensive_bin=<optimized out>,
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:1261
#10 0x00007fffa657838c in Dyninst::SymtabAPI::Symtab::openFile
def_binary=<optimized out>)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:2077
#11 0x00007fffa673cdfc in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols (this=<optimized
out>, addresses=..., linkedobject=..., stm=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fffa680d188 in OpenSpeedShop::Framework::Callbacks::finalize ()
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fffa680eab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fffa68331e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:118
#15 0x00007fffa685f7c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, 0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (
value has been optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&, 1ul> (
value has been optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (connectionBody=..., this=0x7fffa10dd7a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
---Type <return> to continue, or q <return> to quit---
#19 dereference (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (f=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/optional_last_value.hpp:57
#23 operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., combiner=...,
this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&),
boost::signals2::optional_last_value<void>, int, std::less<int>,
boost::function<void (boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>, boost::function<void (boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>,
args#0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fffa685f974 in operator() (args#0=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::valueHandler (this=<optimized out>, value=...)

at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fffa6833734 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,

boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#29 operator()<const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(a1=..., this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>,

boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

*>, boost::arg<1> > >, void,

boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_obj_ptr=...,
---Type <return> to continue, or q <return> to quit---
a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#31 0x00007fffa68432ec in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::operator() (

this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fffa643ec48 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fffa63daf38 in call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const, boost::any
const> (
u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has been optimized out>,
this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
value has been optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string const&,
KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >, void,
boost::any const&>::invoke (function_obj_ptr=..., a0=...)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#41 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fffa643ec48 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fffa643f308 in call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const, boost::any
const> (
u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
---Type <return> to continue, or q <return> to quit---
#47 operator()<KrellInstitute::CBTF::Impl::Network*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has been optimized out>,
this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
value has been optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string const&,
KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >, void,
boost::any const&>::invoke (function_obj_ptr=..., a0=...)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#51 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fffa2bd0fa0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> > (value=...,
name="LinkedObjectGroup", this=<optimized out>)
at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff9c02e380, message=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fffa2bd1c94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void, StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> (a1=...,
this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>, boost::arg<1>

, void, boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>

const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#61 0x00007fffa2bd342c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>

::operator() (

this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fffa643ecc8 in emitOutput (value=..., type=..., name="value",
this=0x7fff9c02a700)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff9c02a700, value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fffa643db94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (
a=<synthetic pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, KrellInstitute::CBTF::Impl::InputMediator,
boost::any const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any const&>::invoke (function_obj_ptr=...,
a0=...)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#71 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fffa2be0f44 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> >
(value=<synthetic pointer>, name="out", this=0x7fff9c0076f0)
at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff9c0076f0, in=...)
at /opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fffa2bd67a4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#79 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet> const&>::invoke
(function_obj_ptr=..., a0=...)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#81 0x00007fffa63e357c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#82 KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>

::operator() (this=<error reading variable: value has been optimized out>,

value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fffa6498ffc in KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fffa63c9494 in emitOutput<boost::shared_ptr<MRN::Packet> >
(value=..., name="value", this=0x7fff9c0072d0)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff9c0072d0, packet=...)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fffa63dafe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

(a1=..., u=..., this=<optimized out>)

at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator, const
boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#90 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

, boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet> const&>::invoke

(function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#92 0x00007fffa63d1460 in operator() (a0=..., this=0x7fff8802c6a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fffa63bab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump (this=0x7fff9c002a80)
at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fffa63bcea4 in operator() (p=<optimized out>, this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0> (a=<synthetic
pointer>, f=..., this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void, KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>

::run (this=<optimized out>) at

Jim Galarowicz

2018-10-26 21:38:51 UTC

Permalink

Hi Xiaozhu, all,

Thanks for your help getting past the API changes!

Unfortunately, with top of master, I still get the core dump:

Note: results marked with (*) had missing values such as
might occur with a mixture of architectural capabilities.

Detecting outliers based on per-process mean values.

Mild outliers (>1.5 IQR from 1st/3rd quartile):
None.

Extreme outliers (>3.0 IQR from 1st/3rd quartile):
None.
[p9-4V100-01:59338] 1 more process has sent help message
help-mpi-btl-openib.txt / no active ports found
[p9-4V100-01:59338] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
/opt/tools/OSS/osscbtf_v2.4.0dyn1026/bin/osscuda: line 1914: 59289
Segmentation faultÂ Â Â Â Â (core dumped) osscollect $topology_opt
$cbtf_offline_opt --program "$1" --collector $collector
[***@p9-4V100-01 CUDA]$ BE: receive failure
BE: receive failure
232389263.030495: Message.c[278] Message_send - MRN_send() failed
232389263.030495: Message.c[278] Message_send - MRN_send() failed
232389263.030562: PeerNode.c[176] PeerNode_sendDirectly -
232389263.030563: PeerNode.c[176] PeerNode_sendDirectly - Message_send()
failed
232389263.030576: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
Message_send() failed
232389263.030577: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232389263.030592: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
232389263.030588: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
^C

[***@p9-4V100-01 CUDA]$ gdb osscollect core.59289
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
..
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun -np 2 ./GEMM
--collector cuda '.
Program terminated with signal 11, Segmentation fault.
#0Â size (this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716Â Â Â Â Â Â Â Â { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le tbb-4.1-9.20130314.el7.ppc64le
xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0Â size (this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1Â compare (__str=..., this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2Â operator< <char, std::char_traits<char>, std::allocator<char> >
(__rhs=..., __lhs=<error reading variable: Cannot access memory at
address 0xffffffffffffffe8>)
Â Â Â at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=..., __x=<error reading
variable: Cannot access memory at address 0xffffffffffffffe8>) at
/usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff6d798330) at /usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48", this=0x7fff897602d0
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>) at
/usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48", this=0x7fff897602d0
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>) at
/usr/include/c++/4.8.2/bits/stl_map.h:822
#7Â Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="/usr/lib64/libcuda.so.410.48", file_=0x7fff6d98c910)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fff8c485988 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff6d98c0c0, mf_=0x7fff6d7a1ab0, err_func=<optimized out>,
alloc_syms=<optimized out>, st=<optimized out>)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Object-elf.C:2888
#9Â 0x00007fff8c42b280 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff6d98ae10, filename="", defensive_bin=<optimized out>,
err=@0x7fff827cc1b9: false)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Symtab.C:1273
#10 0x00007fff8c42b6dc in Dyninst::SymtabAPI::Symtab::openFile
(obj=@0x7fff827cc670: 0x0, filename="/usr/lib64/libcuda.so.410.48",
def_binary=<optimized out>)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Symtab.C:2091
#11 0x00007fff8c5ed8ac in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols (this=<optimized
out>, addresses=..., linkedobject=..., stm=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fff8c6bd188 in OpenSpeedShop::Framework::Callbacks::finalize ()
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fff8c6beab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fff8c6e31e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:118
---Type <return> to continue, or q <return> to quit---
#15 0x00007fff8c70f7c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, 0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (
Â Â Â args=std::tuple containing = {...}, func=<error reading variable:
value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&, 1ul> (
Â Â Â args=std::tuple containing = {...}, func=<error reading variable:
value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (connectionBody=...,
Â Â Â this=0x7fff827cd610) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
#19 dereference (this=0x7fff827cd628) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (f=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fff827cd628) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., this=<optimized out>)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/optional_last_value.hpp:57
#23 operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=...,
Â Â Â combiner=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&),
boost::signals2::optional_last_value<void>, int, std::less<int>,
boost::function<void (boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>, boost::function<void (boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>, args#0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fff8c70f974 in operator() (args#0=..., this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::valueHandler (this=<optimized out>, value=...)

Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fff8c6e3734 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,

boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>,

boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

*>, boost::arg<1> > >, void,

boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>::invoke
(function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#31 0x00007fff8c6f32ec in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::operator() (this=<error reading variable: value has been optimized

out>, value=...)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529

#34 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fff8c2eec68 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348

#36 0x00007fff8c28af38 in call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const,
boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has been optimized out>,
this=<error reading variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=..., this=<error reading variable:
value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >, void,
boost::any const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#41 0x00007fff8c2f42c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fff8c2eec68 in KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fff8c2ef328 in call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const,
boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>, b3=...,
b2=..., b1=..., this=<error reading variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:384
#47 operator()<KrellInstitute::CBTF::Impl::Network*> (u=<error reading
variable: value has been optimized out>, a3=..., a2=..., a1=...,
Â Â Â this=<error reading variable: value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
const std::basic_string<char>&, const KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has been optimized out>,
this=<error reading variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=..., this=<error reading variable:
value has been optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&, boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >, void,
boost::any const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#51 0x00007fff8c2f42c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fff886b0fc0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> >
(value=..., name="LinkedObjectGroup", this=<optimized out>)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff8402d820, message=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fff886b1cb4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void, StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> > (a=<synthetic
pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>::invoke
(function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#61 0x00007fff886b344c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>

::operator() (this=<error reading variable: value has been optimized

out>, value=...)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fff8c2eece8 in emitOutput (value=..., type=..., name="value",
this=0x7fff8402a470)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
---Type <return> to continue, or q <return> to quit---
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff8402a470, value=...) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fff8c2edbb4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (a=<synthetic pointer>, f=...,
Â Â Â this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, KrellInstitute::CBTF::Impl::InputMediator,
boost::any const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any const&>::invoke
(function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#71 0x00007fff8c2f42c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator() (this=<error
reading variable: value has been optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fff886c0f64 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> >
(value=<synthetic pointer>, name="out", this=0x7fff8401c3c0)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff8401c3c0, in=...)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fff886b67c4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#79 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
---Type <return> to continue, or q <return> to quit---
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet> const&>::invoke
(function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#81 0x00007fff8c29357c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>

::operator() (this=<error reading variable: value has been optimized

out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized out>,
name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fff8c279494 in emitOutput<boost::shared_ptr<MRN::Packet> >
(value=..., name="value", this=0x7fff84001c20)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff84001c20, packet=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fff8c28afe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

(a1=..., u=..., this=<optimized out>)

Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator, const
boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
Â Â Â (a=<synthetic pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#90 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

, boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet>

const&>::invoke (
---Type <return> to continue, or q <return> to quit---
Â Â Â function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#92 0x00007fff8c281460 in operator() (a0=..., this=0x7fff6c021a00) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
(this=0x7fff84017c68, tag=@0x7fff827ce500: 1124, packet=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fff8c26ab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump
(this=0x7fff84017c60) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fff8c26cea4 in operator() (p=<optimized out>, this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0> (a=<synthetic
pointer>, f=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void, KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>

::run (this=<optimized out>) at

/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/thread/detail/thread.hpp:117
#99 0x00007fff8d1e23b0 in thread_proxy () from
/opt/tools/OSS/krellroot_v2.4.0dyn1026/lib/libboost_thread.so.1.67.0
#100 0x00007fff8c5a8af4 in start_thread (arg=0x7fff827cefe0) at
pthread_create.c:310
#101 0x00007fff8bd28814 in clone () at
../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109

Hi Jim,
Can you try with the latest Dyninst master? It looks like the crash is
during Symtab::openFile based your stack trace. But, on my end with
the latest master branch, I can open the library file provided by you
using Symtab and perform other analysis.
By the way, the master branch now has the parallel parsing merged. So,
it will need two more dependencies (tbb and openmp). Dyninst will
automatically download and install tbb and will use openmp by default.
1. InstructionAPI::Instruction is now passed by value rather than
boost smart pointer. So, all Instruction::Ptr i should be replaced as
InstructionÂ i.
2. Operation.h is renamed as Operation_impl.h.
Let me know if you encounter any problem with the latest master branch.
Thanks,
--Xiaozhu
Hi all,
Posting again without the library attached - the first email went
to the moderator list.
Jim G

(__rhs=..., __lhs=<error reading variable: Cannot access memory

at address 0xffffffffffffffe8>)
Â Â Â at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=..., __x=<error reading
variable: Cannot access memory at address 0xffffffffffffffe8>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff89c283f0) at /usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48", this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
Â Â Â at /usr/include/c++/4.8.2/bits/stl_map.h:822
*#7 Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="**/usr/lib64/libcuda.so.410.48**", file_=0x7fff89e07740)*
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fffa65d29e8 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff89e06ef0, mf_=0x7fff89c24f10, err_func=<optimized
out>, alloc_syms=<optimized out>,
Â Â Â st=<optimized out>) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Object-elf.C:2986
#9Â 0x00007fffa6577f24 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff89e06060, filename="", defensive_bin=<optimized out>,
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:1261
#10 0x00007fffa657838c in Dyninst::SymtabAPI::Symtab::openFile
filename="/usr/lib64/libcuda.so.410.48", def_binary=<optimized out>)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:2077
#11 0x00007fffa673cdfc in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols
(this=<optimized out>, addresses=..., linkedobject=..., stm=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fffa680d188 in
OpenSpeedShop::Framework::Callbacks::finalize ()
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fffa680eab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fffa68331e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&),
void, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:118
#15 0x00007fffa685f7c4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, 0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (
Â Â Â args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&, 1ul> (
Â Â Â args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (connectionBody=...,
this=0x7fffa10dd7a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
---Type <return> to continue, or q <return> to quit---
#19 dereference (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (f=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/optional_last_value.hpp:57
#23 operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >, boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)> >,
boost::signals2::mutex> > > (last=..., first=..., combiner=...,
this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&),
boost::signals2::optional_last_value<void>, int, std::less<int>,
boost::function<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::function<void (boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>,
Â Â Â args#0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fffa685f974 in operator() (args#0=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::valueHandler (this=<optimized out>, value=...)

, const boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,

boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> >
(a=<synthetic pointer>, f=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&> (a1=...,
this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> const&>,

boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

*>, boost::arg<1> > >, void,

boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_obj_ptr=...,
---Type <return> to continue, or q <return> to quit---
Â Â Â a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#31 0x00007fffa68432ec in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>

::operator() (

Â Â Â this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fffa63daf38 in
call<KrellInstitute::CBTF::Impl::MRNet*, std::basic_string<char>
const, KrellInstitute::CBTF::Type const, boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>,
b3=..., b2=..., b1=..., this=<error reading variable: value has
been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*> (u=<error
reading variable: value has been optimized out>, a3=..., a2=...,
a1=...,
Â Â Â this=<error reading variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const std::basic_string<char>&,
const KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (a=<synthetic pointer>,
f=<error reading variable: value has been optimized out>,
Â Â Â this=<error reading variable: value has been optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&, boost::any
const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >,
void, boost::any const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#41 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized out>,
value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fffa643f308 in
call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type const,
boost::any const> (
Â Â Â u=<error reading variable: value has been optimized out>,
b3=..., b2=..., b1=..., this=<error reading variable: value has
been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
---Type <return> to continue, or q <return> to quit---
#47 operator()<KrellInstitute::CBTF::Impl::Network*> (u=<error
reading variable: value has been optimized out>, a3=..., a2=...,
a1=...,
Â Â Â this=<error reading variable: value has been optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const std::basic_string<char>&,
const KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (a=<synthetic pointer>,
f=<error reading variable: value has been optimized out>,
Â Â Â this=<error reading variable: value has been optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&, boost::any
const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>, boost::arg<1> > >,
void, boost::any const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#51 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized out>,
value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fffa2bd0fa0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup> >
(value=..., name="LinkedObjectGroup", this=<optimized out>)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff9c02e380, message=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fffa2bd1c94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void, StateManagementForCUDA,
const boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> >
(a=<synthetic pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&> (a1=...,
this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#61 0x00007fffa2bd342c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>

::operator() (

Â Â Â this=<error reading variable: value has been optimized out>,
value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fffa643ecc8 in emitOutput (value=..., type=...,
name="value", this=0x7fff9c02a700)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff9c02a700, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fffa643db94 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> > (
Â Â Â a=<synthetic pointer>, f=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=..., this=<optimized out>)
at /opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, KrellInstitute::CBTF::Impl::InputMediator,
boost::any const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any const&>::invoke
(function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#71 0x00007fffa64442a4 in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized out>,
value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fffa2be0f44 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged> >
(value=<synthetic pointer>, name="out", this=0x7fff9c0076f0)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff9c0076f0, in=...)
Â Â Â at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fffa2bd67a4 in operator() (a1=..., p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>, boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> > (a=<synthetic pointer>, f=...,
this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#79 operator()<const boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet>
const&>::invoke (function_obj_ptr=..., a0=...)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#81 0x00007fffa63e357c in operator() (a0=..., this=<error reading
variable: value has been optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>

::operator() (this=<error reading variable: value has been

optimized out>, value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to quit---
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl (this=<optimized
out>, name=..., type=..., value=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fffa63c9494 in
emitOutput<boost::shared_ptr<MRN::Packet> > (value=...,
name="value", this=0x7fff9c0072d0)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff9c0072d0, packet=...)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fffa63dafe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=...,
Â Â Â this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>

(a1=..., u=..., this=<optimized out>)

, boost::arg<1> > >, void, boost::shared_ptr<MRN::Packet>

const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#92 0x00007fffa63d1460 in operator() (a0=...,
this=0x7fff8802c6a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fffa63bab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump
(this=0x7fff9c002a80)
Â Â Â at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fffa63bcea4 in operator() (p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0>
(a=<synthetic pointer>, f=..., this=<optimized out>)
Â Â Â at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void, KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>

::run (this=<optimized out>) at

_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api