[DynInst_API:] Status of Dyninst 10

[Xiaozhu Meng]

Hi,

Dyninst 10 is going to be released before July. The current master branch
on the github is close to Dyninst 10 in terms of API.

One of the remaining task before releasing Dyninst 10 is integrating
parallel parsing. The main interface change caused by this task is changing
InstructionAPI::Instruction objects from passing by shared pointers to
passing by values.

Right now, you have
InstructionAPI::Instruction::Ptr insn = decoder.decode();
printf("Instruction %s\n", insn->format().c_str());

In Dyninst 10, you will have

InstructionAPI::Instruction insn = decoder.decode();
printf("Instruction %s\n", insn.format().c_str());


Other than this interface breaking change, there could be new interface
added. For example, there is a pull request of adding instrumentation
snippet for xor. There should also be new interface for specifying the
number of threads to use for parallel parsing.

In summary, Dyninst 10 will be significantly different from its previous
version internally, but the interface will be quite compatible excluding
the Instruction part.

Thanks,

--Xiaozhu

Hi!
I am writing to inquire about the status of the next major release of
Dyninst 10.
We were planning to do a new development on top of Dyninst, but in the
roadmap shown in the GitHub page we read that you were planning to
introduce big changes and break compatibility with several APIs.
If you could give us any update on the estimated release date, this would
be
helpful for us to decide whether it's worth to hold our development
waiting for the latest version, or go ahead with the current one despite
the upcoming changes.
Best regards!
-- G.
http://bsc.es/disclaimer
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[Germán Llort]

Hi!

I am writing to inquire about the status of the next major release of
Dyninst 10.

We were planning to do a new development on top of Dyninst, but in the
roadmap shown in the GitHub page we read that you were planning to
introduce big changes and break compatibility with several APIs.

If you could give us any update on the estimated release date, this would be
helpful for us to decide whether it's worth to hold our development
waiting for the latest version, or go ahead with the current one despite
the upcoming changes.

Best regards!
-- G.


http://bsc.es/disclaimer

[Stan Cox]

Post by Xiaozhu Meng
Dyninst 10 is going to be released before July. The current master
branch on the github is close to Dyninst 10 in terms of API.

Is there a revised schedule for dyninst 10?  We are wondering for purposes of Red Hat Linux planning; e.g.  libdwarf could possibly be deprecated.

Thanks
Stan

[William Cohen]

Hi,
Dyninst 10 is going to be released before July. The current master branch on the github is close to Dyninst 10 in terms of API.
One of the remaining task before releasing Dyninst 10 is integrating parallel parsing. The main interface change caused by this task is changing InstructionAPI::Instruction objects from passing by shared pointers to passing by values.
Right now, you have
InstructionAPI::Instruction::Ptr insn = decoder.decode();
printf("Instruction %s\n", insn->format().c_str());
In Dyninst 10,  you will have
InstructionAPI::Instruction insn = decoder.decode();
printf("Instruction %s\n", insn.format().c_str());
Other than this interface breaking change, there could be new interface added. For example, there is a pull request of adding instrumentation snippet for xor. There should also be new interface for specifying the number of threads to use for parallel parsing. 
In summary, Dyninst 10 will be significantly different from its previous version internally, but the interface will be quite compatible excluding the Instruction part.
Thanks,
--Xiaozhu

Hi,

There is a tool, libabigail, to check ABI compatibility issues:

https://developers.redhat.com/blog/2014/10/23/comparing-abis-for-compatibility-with-libabigail-part-1/
https://developers.redhat.com/blog/2014/10/28/comparing-abis-for-compatibility-libabigail-part-2/
https://fedoraproject.org/wiki/How_to_check_for_ABI_changes_with_abipkgdiff

Have there been checks of the dyninst shared libraries to make sure there are not other changes in the ABI?

So with this change to InstructionAPI::Instruction objects one would need to compile code either for the old or the new version of library? What was the reason for changing to passing by value?

-Will

Hi!
I am writing to inquire about the status of the next major release of
Dyninst 10.
We were planning to do a new development on top of Dyninst, but in the
roadmap shown in the GitHub page we read that you were planning to
introduce big changes and break compatibility with several APIs.
If you could give us any update on the estimated release date, this would be
helpful for us to decide whether it's worth to hold our development
waiting for the latest version, or go ahead with the current one despite
the upcoming changes.
Best regards!
-- G.
http://bsc.es/disclaimer
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[Xiaozhu Meng]

In terms of the timeline for Dyninst 10.0, I will need to double-check with
Bart because he is traveling today, but I think by the end of month we
should be able to release 10.0.

In terms of ABI compatibility, we have not used libabigail yet. We tried to
keep track of API-breaking changes. But for ABI-breaking changes, we do not
have a list. My estimation is that there are other ABI-breaking changes in
10.0 since it is going to be major release.

For the change for InstructionAPI::Instruction, this is an API-breaking
change. So, user of Dyninst will need to either compile with the old
version or the new version. Making this change was before me started to
work with parallel code parsing, but I believe shared instruction objects
have caused too many data races and lock contention between threads.

Post by Xiaozhu Meng

Post by Xiaozhu Meng
Hi,
Dyninst 10 is going to be released before July. The current master

branch on the github is close to Dyninst 10 in terms of API.

Post by Xiaozhu Meng
One of the remaining task before releasing Dyninst 10 is integrating

parallel parsing. The main interface change caused by this task is changing
InstructionAPI::Instruction objects from passing by shared pointers to
passing by values.

Post by Xiaozhu Meng
Right now, you have
InstructionAPI::Instruction::Ptr insn = decoder.decode();
printf("Instruction %s\n", insn->format().c_str());
In Dyninst 10, you will have
InstructionAPI::Instruction insn = decoder.decode();
printf("Instruction %s\n", insn.format().c_str());
Other than this interface breaking change, there could be new interface

added. For example, there is a pull request of adding instrumentation
snippet for xor. There should also be new interface for specifying the
number of threads to use for parallel parsing.

Post by Xiaozhu Meng
In summary, Dyninst 10 will be significantly different from its previous

version internally, but the interface will be quite compatible excluding
the Instruction part.

Post by Xiaozhu Meng
Thanks,
--Xiaozhu

Hi,
https://developers.redhat.com/blog/2014/10/23/comparing-abis-for-compatibility-with-libabigail-part-1/
https://developers.redhat.com/blog/2014/10/28/comparing-abis-for-compatibility-libabigail-part-2/
https://fedoraproject.org/wiki/How_to_check_for_ABI_changes_with_abipkgdiff
Have there been checks of the dyninst shared libraries to make sure there
are not other changes in the ABI?
So with this change to InstructionAPI::Instruction objects one would need
to compile code either for the old or the new version of library? What was
the reason for changing to passing by value?
-Will

Post by Xiaozhu Meng
Hi!
I am writing to inquire about the status of the next major release of
Dyninst 10.
We were planning to do a new development on top of Dyninst, but in

the

Post by Xiaozhu Meng
roadmap shown in the GitHub page we read that you were planning to
introduce big changes and break compatibility with several APIs.
If you could give us any update on the estimated release date, this

would be

Post by Xiaozhu Meng
helpful for us to decide whether it's worth to hold our development
waiting for the latest version, or go ahead with the current one

despite

Post by Xiaozhu Meng
the upcoming changes.
Best regards!
-- G.
http://bsc.es/disclaimer
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api <

https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>

Post by Xiaozhu Meng
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[Stan Cox]

we should be able to release 10.0.

Any thoughts on the release 10.0? We are wondering if we can drop
libdwarf in the next version of rhel. I can start testing upstream
dyninst with that version of rhel.

[Xiaozhu Meng]

Parallel parsing is still the major item to clear for 10.0. But the current
dyninst master already dropped libdwarf and depends on libdw. So, for
libdwarf testing purposes, current master well represents 10.0.

Post by Stan Cox

we should be able to release 10.0.

Any thoughts on the release 10.0? We are wondering if we can drop
libdwarf in the next version of rhel. I can start testing upstream
dyninst with that version of rhel.

[Mark W. Krentel]

Stan,

As Xiaozhu said, one of the major additions to 10.x will be the merge
of the new-parallel-parsing branch.

If you want to get a jump on things, you could try building and
running the new-parallel-parsing branch. Fairly recent master has
been merged into new-par-par, so it should be pretty close to how 10.x
will build and run.

There are a couple new things in the build. Nothing radical, but it
will use Intel TBB and openmp as dependencies. Plus, be able to run
parsing functions with openmp threads.

Of course, you wouldn't want to release new-par-par until it's merged
into master and declared 10.x.

--Mark

Post by Xiaozhu Meng
Parallel parsing is still the major item to clear for 10.0. But the
current dyninst master already dropped libdwarf and depends on libdw.
So, for libdwarf testing purposes, current master well represents 10.0.

we should be able to release 10.0.

Any thoughts on the release 10.0? We are wondering if we can drop
libdwarf in the next version of rhel. I can start testing upstream
dyninst with that version of rhel.

[Stan Cox]

I have not been able to dive into this but mentioning in case there is an obvious change to configuring and building the new-parallel-parsing branch.

With fedora 28
 cmake -DCMAKE_INSTALL_PREFIX=.../dyninst/install -DCMAKE_BUILD_TYPE=debug .../dyninst/src/CMakeLists.txt
In CMakeError.log I see (-lpthread on linux)
Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /work/scox/dyninst/src/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/gmake" "cmTC_3252e/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_3252e.dir/build.make CMakeFiles/cmTC_3252e.dir/build
gmake[1]: Entering directory '.../dyninst/src/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o
/usr/bin/gcc   -DCHECK_FUNCTION_EXISTS=pthread_create   -o CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o   -c /usr/share/cmake/Modules/CheckFunctionExists.c
Linking C executable cmTC_3252e
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_3252e.dir/link.txt --verbose=1
/usr/bin/gcc  -DCHECK_FUNCTION_EXISTS=pthread_create    -rdynamic CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o  -o cmTC_3252e -lpthreads
/usr/bin/ld: cannot find -lpthreads

[Xiaozhu Meng]

Hi Stan,

The CMakeError.log is misleading. The real error should be output to the
terminal.

CMake will always test different flags for pthread, such as -lpthreads,
-pthreads, and so on. Some of them will fail and CMake will write them to
CMakeError.log. Somehow CMake does not write the real error to
CMakeError.log.

Let us know the real error.

Thanks,

--Xiaozhu

Post by Stan Cox
I have not been able to dive into this but mentioning in case there is an
obvious change to configuring and building the new-parallel-parsing branch.
With fedora 28
cmake -DCMAKE_INSTALL_PREFIX=.../dyninst/install
-DCMAKE_BUILD_TYPE=debug .../dyninst/src/CMakeLists.txt
In CMakeError.log I see (-lpthread on linux)
Determining if the function pthread_create exists in the pthreads failed
Change Dir: /work/scox/dyninst/src/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_3252e/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_3252e.dir/build.make
CMakeFiles/cmTC_3252e.dir/build
gmake[1]: Entering directory '.../dyninst/src/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o
/usr/bin/gcc -DCHECK_FUNCTION_EXISTS=pthread_create -o
CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o -c
/usr/share/cmake/Modules/CheckFunctionExists.c
Linking C executable cmTC_3252e
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_3252e.dir/link.txt
--verbose=1
/usr/bin/gcc -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
CMakeFiles/cmTC_3252e.dir/CheckFunctionExists.c.o -o cmTC_3252e -lpthreads
/usr/bin/ld: cannot find -lpthreads
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[Stan Cox]

Post by Xiaozhu Meng
Let us know the real error.

There was a problem with the examples dir; it was unpopulated for some
reason so I repopulated it from master. With that I then got (Fedora 28):

[ 44%] Linking CXX shared library libparseAPI.so
CMakeFiles/parseAPI.dir/src/debug_parse.C.o: In function
`Dyninst::ParseAPI::parsing_printf_int(char const*, ...)':
/work/scox/dyninst/src/parseAPI/src/debug_parse.C:65: undefined
reference to `omp_get_thread_num'
collect2: error: ld returned 1 exit status
make[2]: *** [parseAPI/CMakeFiles/parseAPI.dir/build.make:1160:
parseAPI/libparseAPI.so.9.3.2] Error 1
make[1]: *** [CMakeFiles/Makefile2:1082:
parseAPI/CMakeFiles/parseAPI.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

To get around that I added this to parseAPI/CMakeLists.txt

set(OPENMP_LIBRARIES gomp)
# scox added OPENMP_LIBRARIES
target_link_private_libraries(parseAPI ${Boost_LIBRARIES}
${TBB_LIBRARIES} ${OPENMP_LIBRARIES} tbbmalloc)

and got past that. However I still hit the problem building the
examples (issue #459)

[ 93%] Linking CXX executable codeCoverage
/usr/bin/ld: cannot open output file codeCoverage: Is a directory
collect2: error: ld returned 1 exit status

I pulled in the #459 patch which resolves the above but hit a different
problem building examples that I'll look at.

[ 94%] Linking CXX executable cfg_to_dot
/usr/bin/ld: CMakeFiles/cfg_to_dot.dir/__/parseAPI/doc/example.cc.o:
undefined reference to symbol '_ZN5boost6system16generic_categoryEv'
//usr/lib64/libboost_system.so.1.66.0: error adding symbols: DSO missing
from command line
collect2: error: ld returned 1 exit status

[Stan Cox]

Post by Stan Cox
undefined reference to symbol '_ZN5boost6system16generic_categoryEv'
//usr/lib64/libboost_system.so.1.66.0: error adding symbols: DSO missing
from command line

This patch resolves the above for new-parallel-parsing branch so it
builds on fedora 28. It is essentially a reworking of the #459 patch
for n-p-p. It builds codeCoverage/codeCoverage and unstrip/unstrip in
their subdirectory; otherwise building in the examples directory results
in a name clash between the executable and directory. Next, I'll try
n-p-p with our systemtap dyninst backend.

[Xiaozhu Meng]

Hi Stan,

Can you checkout the current Dyninst master? I merged the
new-parallel-parsing branch back to the master earlier today and Sasha made
a new fix later.

In terms of the problem of building the example, can you re-generate a
patch if this problem still exists for the master branch?

The latest commit adds an option USE_OpenMP to the CMake system. By
default, it is "ON" and will compile Dyninst with openmp for parallel
parsing. Setting USE_OpenMP to other value will not use openmp (and
of course no parallel parsing).

Lastly, my merge should fix the problem of "undefined reference to symbol
'_ZN5boost6system16generic_categoryEv"

Let me know what you find with the master branch.

Thanks,

--Xiaozhu

Post by Stan Cox

Post by Stan Cox
undefined reference to symbol '_ZN5boost6system16generic_categoryEv'
//usr/lib64/libboost_system.so.1.66.0: error adding symbols: DSO missing
from command line

This patch resolves the above for new-parallel-parsing branch so it
builds on fedora 28. It is essentially a reworking of the #459 patch
for n-p-p. It builds codeCoverage/codeCoverage and unstrip/unstrip in
their subdirectory; otherwise building in the examples directory results
in a name clash between the executable and directory. Next, I'll try
n-p-p with our systemtap dyninst backend.
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[Stan Cox]

Post by Xiaozhu Meng
Can you checkout the current Dyninst master?
In terms of the problem of building the example, can you re-generate a
patch if this problem still exists for the master branch?

Updated master and rebuilt using pristine sources and the example build issue is still there on Linux:
[ 95%] Linking CXX executable codeCoverage
/usr/bin/ld: cannot open output file codeCoverage: Is a directory
collect2: error: ld returned 1 exit status

I'll regenerate a patch.

[Xiaozhu Meng]

Thanks!

Do you typically build Dyninst in the Dyninst source directory? I only
build Dyninst out of source, and I guess this may be reason that I never
saw this error before.

Post by Stan Cox

Post by Xiaozhu Meng
Can you checkout the current Dyninst master?
In terms of the problem of building the example, can you re-generate a
patch if this problem still exists for the master branch?

Updated master and rebuilt using pristine sources and the example build
[ 95%] Linking CXX executable codeCoverage
/usr/bin/ld: cannot open output file codeCoverage: Is a directory
collect2: error: ld returned 1 exit status
I'll regenerate a patch.
_______________________________________________
Dyninst-api mailing list
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api