[DynInst_API:] Abort while building Dyninst with Intel compiler - tbb related

[Jim Galarowicz <jeg@xxxxxxxxxxxxxxxxx>]

Hi all,

We have a user that wants to build OpenSpeedShop with Intel compilers.ÂÂ The compile/compiler is aborting when building Dyninst where it is referencing TBB includes.

I wonder if anyone can see any issues with the compiler version, tbb version or another thing that might be a problem.

[jeg@login01 include]$ icc -v
icc version 19.0.5.281 (gcc version 4.8.5 compatibility)

Listing of abort area:

 96%] Building C object CMakeFiles/dyninstAPI_RT.dir/src/RTstatic_ctors_dtors-x86.c.o
[100%] Linking C shared library libdyninstAPI_RT.so
[100%] Built target dyninstAPI_RT
[  0%] Built target DyninstRT
Scanning dependencies of target common
[  1%] Building CXX object common/CMakeFiles/common.dir/src/mcs-lock.C.o
[  1%] Building CXX object common/CMakeFiles/common.dir/src/pfq-rwlock.C.o
[  1%] Building CXX object common/CMakeFiles/common.dir/src/race-detector-annotations.C.o
[  1%] Building CXX object common/CMakeFiles/common.dir/src/string-regex.C.o
[  2%] Building CXX object common/CMakeFiles/common.dir/src/Timer.C.o
[  2%] Building CXX object common/CMakeFiles/common.dir/src/Types.C.o
[  2%] Building CXX object common/CMakeFiles/common.dir/src/lprintf.C.o
[  2%] Building CXX object common/CMakeFiles/common.dir/src/pathName.C.o
[  3%] Building CXX object common/CMakeFiles/common.dir/src/Time.C.o
[  3%] Building CXX object common/CMakeFiles/common.dir/src/fraction.C.o
[  3%] Building CXX object common/CMakeFiles/common.dir/src/timing.C.o
[  4%] Building CXX object common/CMakeFiles/common.dir/src/stats.C.o
[  4%] Building CXX object common/CMakeFiles/common.dir/src/Annotatable.C.o
[  4%] Building CXX object common/CMakeFiles/common.dir/src/MappedFile.C.o
[  4%] Building CXX object common/CMakeFiles/common.dir/src/sha1.C.o
[  5%] Building CXX object common/CMakeFiles/common.dir/src/util.C.o
[  5%] Building CXX object common/CMakeFiles/common.dir/src/Node.C.o
[  5%] Building CXX object common/CMakeFiles/common.dir/src/Graph.C.o
[  5%] Building CXX object common/CMakeFiles/common.dir/src/Edge.C.o
[  6%] Building CXX object common/CMakeFiles/common.dir/src/DOT.C.o
[  6%] Building CXX object common/CMakeFiles/common.dir/src/dyn_regs.C.o
[  6%] Building CXX object common/CMakeFiles/common.dir/src/AST.C.o
[  6%] Building CXX object common/CMakeFiles/common.dir/src/addrtranslate.C.o
[  7%] Building CXX object common/CMakeFiles/common.dir/src/arch-x86.C.o
[  7%] Building CXX object common/CMakeFiles/common.dir/src/arch-power.C.o
[  7%] Building CXX object common/CMakeFiles/common.dir/src/arch-aarch64.C.o
[  8%] Building CXX object common/CMakeFiles/common.dir/src/debug_common.C.o
[  8%] Building CXX object common/CMakeFiles/common.dir/src/VariableLocation.C.o
[  8%] Building CXX object common/CMakeFiles/common.dir/src/Buffer.C.o
[  8%] Building CXX object common/CMakeFiles/common.dir/src/MachSyscall.C.o
[  9%] Building CXX object common/CMakeFiles/common.dir/src/linuxKludges.C.o
[  9%] Building CXX object common/CMakeFiles/common.dir/src/timing-linux.C.o
[  9%] Building CXX object common/CMakeFiles/common.dir/src/parseauxv.C.o
[  9%] Building CXX object common/CMakeFiles/common.dir/src/addrtranslate-sysv.C.o
[ 10%] Building CXX object common/CMakeFiles/common.dir/src/addrtranslate-auxv.C.o
[ 10%] Building CXX object common/CMakeFiles/common.dir/src/addrtranslate-linux.C.o
[ 10%] Linking CXX shared library libcommon.so
[ 10%] Built target common
Scanning dependencies of target dynElf
[ 10%] Building CXX object elf/CMakeFiles/dynElf.dir/src/Elf_X.C.o
[ 10%] Linking CXX shared library libdynElf.so
[ 10%] Built target dynElf
Scanning dependencies of target dynDwarf
[ 10%] Building CXX object dwarf/CMakeFiles/dynDwarf.dir/src/dwarfResult.C.o
[ 10%] Building CXX object dwarf/CMakeFiles/dynDwarf.dir/src/dwarfExprParser.C.o
[ 10%] Building CXX object dwarf/CMakeFiles/dynDwarf.dir/src/dwarfFrameParser.C.o
[ 11%] Building CXX object dwarf/CMakeFiles/dynDwarf.dir/src/dwarfHandle.C.o
[ 11%] Linking CXX shared library libdynDwarf.so
[ 11%] Built target dynDwarf
Scanning dependencies of target symLite
[ 11%] Building CXX object symlite/CMakeFiles/symLite.dir/src/SymLite-elf.C.o
[ 11%] Linking CXX shared library libsymLite.so
[ 11%] Built target symLite
Scanning dependencies of target symtabAPI
[ 12%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Object.C.o
[ 12%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Aggregate.C.o
[ 12%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Function.C.o
[ 12%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Variable.C.o
[ 13%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Symbol.C.o
[ 13%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/LineInformation.C.o
[ 13%] Building CXX object symtabAPI/CMakeFiles/symtabAPI.dir/src/Symtab.C.o
In file included from /global/software/centos-7/modules/langs/intel/2019.5.281/compilers_and_libraries_2019.5.281/linux/tbb/include/tbb/tbb_machine.h(245),
                 from /global/software/centos-7/modules/langs/intel/2019.5.281/compilers_and_libraries_2019.5.281/linux/tbb/include/tbb/spin_rw_mutex.h(21),
                 from /global/software/centos-7/modules/langs/intel/2019.5.281/compilers_and_libraries_2019.5.281/linux/tbb/include/tbb/concurrent_hash_map.h(27),
                 from /global/home/users/jeg/openspeedshop-externals/BUILD/login01.hpcadvisorycouncil.com/dyninst-10.1.0/symtabAPI/h/Collections.h(34),
                 from /global/home/users/jeg/openspeedshop-externals/BUILD/login01.hpcadvisorycouncil.com/dyninst-10.1.0/symtabAPI/src/Symtab.C(49):
/global/software/centos-7/modules/langs/intel/2019.5.281/compilers_and_libraries_2019.5.281/linux/tbb/include/tbb/machine/linux_common.h(43): internal error: bad pointer
  #if defined(__linux__) || __TBB_has_include(<linux/futex.h>)
                                                             ^

compilation aborted for /global/home/users/jeg/openspeedshop-externals/BUILD/login01.hpcadvisorycouncil.com/dyninst-10.1.0/symtabAPI/src/Symtab.C (code 4)
make[2]: *** [symtabAPI/CMakeFiles/symtabAPI.dir/src/Symtab.C.o] Error 4
make[1]: *** [symtabAPI/CMakeFiles/symtabAPI.dir/all] Error 2
make: *** [all] Error 2


The Futex include:


[jeg@login01 openspeedshop-externals]$ cd /usr/include
[jeg@login01 include]$ find . -name futex.h
./linux/futex.h
[jeg@login01 include]$ cat linux/futex.h
#ifndef _LINUX_FUTEX_H
#define _LINUX_FUTEX_H


#include <linux/types.h>

/* Second argument to futex syscall */


#define FUTEX_WAIT		0
#define FUTEX_WAKE		1
#define FUTEX_FD		2
#define FUTEX_REQUEUE		3
#define FUTEX_CMP_REQUEUE	4
#define FUTEX_WAKE_OP		5
#define FUTEX_LOCK_PI		6
#define FUTEX_UNLOCK_PI		7
#define FUTEX_TRYLOCK_PI	8
#define FUTEX_WAIT_BITSET	9
#define FUTEX_WAKE_BITSET	10
#define FUTEX_WAIT_REQUEUE_PI	11
#define FUTEX_CMP_REQUEUE_PI	12

#define FUTEX_PRIVATE_FLAG	128
#define FUTEX_CLOCK_REALTIME	256
#define FUTEX_CMD_MASK		~(FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME)

#define FUTEX_WAIT_PRIVATE	(FUTEX_WAIT | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAKE_PRIVATE	(FUTEX_WAKE | FUTEX_PRIVATE_FLAG)
#define FUTEX_REQUEUE_PRIVATE	(FUTEX_REQUEUE | FUTEX_PRIVATE_FLAG)
#define FUTEX_CMP_REQUEUE_PRIVATE (FUTEX_CMP_REQUEUE | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAKE_OP_PRIVATE	(FUTEX_WAKE_OP | FUTEX_PRIVATE_FLAG)
#define FUTEX_LOCK_PI_PRIVATE	(FUTEX_LOCK_PI | FUTEX_PRIVATE_FLAG)
#define FUTEX_UNLOCK_PI_PRIVATE	(FUTEX_UNLOCK_PI | FUTEX_PRIVATE_FLAG)
#define FUTEX_TRYLOCK_PI_PRIVATE (FUTEX_TRYLOCK_PI | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAIT_BITSET_PRIVATE	(FUTEX_WAIT_BITSET | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAKE_BITSET_PRIVATE	(FUTEX_WAKE_BITSET | FUTEX_PRIVATE_FLAG)
#define FUTEX_WAIT_REQUEUE_PI_PRIVATE	(FUTEX_WAIT_REQUEUE_PI | \
					 FUTEX_PRIVATE_FLAG)
#define FUTEX_CMP_REQUEUE_PI_PRIVATE	(FUTEX_CMP_REQUEUE_PI | \
					 FUTEX_PRIVATE_FLAG)

/*
 * Support for robust futexes: the kernel cleans up held futexes at
 * thread exit time.
 */

/*
 * Per-lock list entry - embedded in user-space locks, somewhere close
 * to the futex field. (Note: user-space uses a double-linked list to
 * achieve O(1) list add and remove, but the kernel only needs to know
 * about the forward link)
 *
 * NOTE: this structure is part of the syscall ABI, and must not be
 * changed.
 */
struct robust_list {
	struct robust_list *next;
};

/*
 * Per-thread list head:
 *
 * NOTE: this structure is part of the syscall ABI, and must only be
 * changed if the change is first communicated with the glibc folks.
 * (When an incompatible change is done, we'll increase the structure
 *  size, which glibc will detect)
 */
struct robust_list_head {
	/*
	 * The head of the list. Points back to itself if empty:
	 */
	struct robust_list list;

	/*
	 * This relative offset is set by user-space, it gives the kernel
	 * the relative position of the futex field to examine. This way
	 * we keep userspace flexible, to freely shape its data-structure,
	 * without hardcoding any particular offset into the kernel:
	 */
	long futex_offset;

	/*
	 * The death of the thread may race with userspace setting
	 * up a lock's links. So to handle this race, userspace first
	 * sets this field to the address of the to-be-taken lock,
	 * then does the lock acquire, and then adds itself to the
	 * list, and then clears this field. Hence the kernel will
	 * always have full knowledge of all locks that the thread
	 * _might_ have taken. We check the owner TID in any case,
	 * so only truly owned locks will be handled.
	 */
	struct robust_list *list_op_pending;
};

/*
 * Are there any waiters for this robust futex:
 */
#define FUTEX_WAITERS		0x80000000

/*
 * The kernel signals via this bit that a thread holding a futex
 * has exited without unlocking the futex. The kernel also does
 * a FUTEX_WAKE on such futexes, after setting the bit, to wake
 * up any possible waiters:
 */
#define FUTEX_OWNER_DIED	0x40000000

/*
 * The rest of the robust-futex field is for the TID:
 */
#define FUTEX_TID_MASK		0x3fffffff

/*
 * This limit protects against a deliberately circular list.
 * (Not worth introducing an rlimit for it)
 */
#define ROBUST_LIST_LIMIT	2048

/*
 * bitset with all bits set for the FUTEX_xxx_BITSET OPs to request a
 * match of any bit.
 */
#define FUTEX_BITSET_MATCH_ANY	0xffffffff


#define FUTEX_OP_SET		0	/* *(int *)UADDR2 = OPARG; */
#define FUTEX_OP_ADD		1	/* *(int *)UADDR2 += OPARG; */
#define FUTEX_OP_OR		2	/* *(int *)UADDR2 |= OPARG; */
#define FUTEX_OP_ANDN		3	/* *(int *)UADDR2 &= ~OPARG; */
#define FUTEX_OP_XOR		4	/* *(int *)UADDR2 ^= OPARG; */

#define FUTEX_OP_OPARG_SHIFT	8	/* Use (1 << OPARG) instead of OPARG.  */

#define FUTEX_OP_CMP_EQ		0	/* if (oldval == CMPARG) wake */
#define FUTEX_OP_CMP_NE		1	/* if (oldval != CMPARG) wake */
#define FUTEX_OP_CMP_LT		2	/* if (oldval < CMPARG) wake */
#define FUTEX_OP_CMP_LE		3	/* if (oldval <= CMPARG) wake */
#define FUTEX_OP_CMP_GT		4	/* if (oldval > CMPARG) wake */
#define FUTEX_OP_CMP_GE		5	/* if (oldval >= CMPARG) wake */

/* FUTEX_WAKE_OP will perform atomically
   int oldval = *(int *)UADDR2;
   *(int *)UADDR2 = oldval OP OPARG;
   if (oldval CMP CMPARG)
     wake UADDR2;  */

#define FUTEX_OP(op, oparg, cmp, cmparg) \
  (((op & 0xf) << 28) | ((cmp & 0xf) << 24)		\
   | ((oparg & 0xfff) << 12) | (cmparg & 0xfff))

#endif /* _LINUX_FUTEX_H */
[jeg@login01 include]$ 

Thanks,
Jim G