Start usage of extern templates

Summary

This is the 3rd attempt to integrate extern templates into OTB.

Rationale

This work is part of the build performance story (#1649 (closed))

Implementation Details

This 3rd try is based on branches 1649-extern-templates and 1649-extern-templates-take2. Here is the summary:

• As in "take2", use a custom GenerateExportHeaderCustom module to generate export headers.
• Use extern templates with common types for classes :
  • Image
  • VectorImage
  • ImageFileReader
  • ImageFileWriter
• Clean export macros (remove/replace ITK_EXPORT) in modules ImageBase and ImageIO
• other minor fixes to reduce the number of symbols compiled

You can use the following command to check the impact:

find . -name "*.o" -exec nm -g --demangle '{}' \; | grep " W otb::" | sort | uniq -c | sort -n

Additional notes

Two classes have not been exported by default (ITK_EXPORT removed) as they are not used outside their module :

• otb::ExtractROIBase
• otb::ImageSeriesFileReaderBase

Copyright

The copyright owner is CNES and has signed the ORFEO ToolBox Contributor License Agreement.

---

Check before merging:

• All discussions are resolved
• At least 2 votes from core developers, no vote.
• The feature branch is (reasonably) up-to-date with the base branch
• Dashboard is green
• Copyright owner has signed the ORFEO ToolBox Contributor License Agreement

changed milestone to %7.0.0

added Doing + 1 deleted label

added 93 commits

• 96688c77...988ab22c - 92 commits from branch develop
• 09728a32 - MRG: Synchronize with branch develop

Compare with previous version

wow amazing! I am looking forward to dashboards results! I am very worried about the maintainability of all the vodoo dark magic in CMake/GenerateExportHeaderCustom.cmake and CMake/exportheader.cmake.in however. But let's see!

Dashboard is green. Amazing :O

I ran 5 builds to see if the effect on build time. I don't see any significant improvement unfortunately :( At least in my config.

command:  make clean; cmake .; time make -j24
config: no ccache, (GCC) 6.3.0, RH7

develop:
real	7m12.820s
user	29m44.164s
sys	10m23.880s

real	6m41.824s
user	29m40.990s
sys	9m11.434s


1649-extern-templates-take3:
real	7m40.508s
user	29m10.167s
sys	10m28.825s


real	7m47.959s
user	29m8.816s
sys	10m38.250s


real	7m11.230s
user	29m4.193s
sys	9m22.560s

I will try it on another machine when I get the time.

If we think about merging this I think we should:

• Demonstrate it's improving the build time (more benchs needed, more modules?)
• Squash the branch into a single commit so it's easy to revert
• Document the cmake stuff more

Then I think we should investigate the ApplicationEngine module next. It seems to always take a lot of time during a build.

Results from our new CI seems to show some improvements:

• native-build:
  • MR !378 (merged) : 37m 45s
  • this MR : 28m 17s
• shark-build:
  • MR !378 (merged) : 35m 52s
  • this MR : 31m 12s

If your compiler is doing some caching on its own, it may explain the timings you get. Maybe try with less jobs (-j4 perhaps).

I don't really see the point of squashing, if you want to revert you can revert the merge commit.

added 1 commit

• d92282d4 - DOC: some documentation on GenerateExportHeaderCustom

Compare with previous version

Other pipelines have been run and this MR is around 29min per job, while other MRs take between 34min and 37min (only logger_bug is suprisingly fast). I think the improvement on build time is validated.

I pushed some documentation about the CMake scripts, they have been cloned from CMake so at some point we should contribute those patches to CMake.

Need one more vote for merge.

merged

mentioned in commit 0a4ef44a

On the dashboard, the gain on build time goes from 5min to 10min

great! Is it doable for other modules?

Yes, I think modules such as Common and ITK are good candidates. Also, I remember in previous branches an attempt to export the FillIn / FillOut functions of the CastImageFilter (in order to reduce compilation time for OTBApplicationEngine).

added refactoring label

removed Doing label