Skip to content
Snippets Groups Projects

Performance differences between RH6 and RH7

    • Closed Issue created by Julie Brossard @jbrossar

    Description

    Decreased performances in the OTB applications have been observed in the cluster of CNES between RH6 and RH7 : if the type of the output is not a float RH7 takes more time than RH6.

    Results :

    • RH7 :
    Writing output/5_UpdateSynthesis_R1.tif...: 100% [**************************************************] (4e+01 seconds)
Writing output/5_UpdateSynthesis_R2.tif...: 100% [**************************************************] (1e+01 seconds)
 
real	0m51.818s
user	3m9.529s
sys	0m9.358s
    • RH6
    Writing output/5_UpdateSynthesis_R1.tif...: 100% [**************************************************] (3e+01 seconds)
Writing output/5_UpdateSynthesis_R2.tif...: 100% [**************************************************] (1e+01 seconds)

real	0m43.846s
user	2m24.699s
sys	0m6.023s

    Steps to reproduce

    The application used in the example can be found here : UpdateSynthesis

    ./otbcli_UpdateSynthesis -inr1 4_CP_R1.tif -xml SENTINEL2A_20180418-104512-083_L2A_T32ULV_C_V2-2/SENTINEL2A_20180418-104512-083_L2A_T32ULV_C_V2-2_MTD_ALL.xml -cld 4_cld10.tif -wat 4_wat10.tif -snw 4_snw10.tif -weightl2a 4_WeightTotal.tif -outr1 4_UpdateSynthesis_R1.tif -prevproductr1 3_UpdateSynthesis_R1.tif -inr2 4_CP_R2.tif -outr2 4_UpdateSynthesis_R2.tif -prevproductr2 3_UpdateSynthesis_R2.tif -progress 1

    Configuration information

    OTB > 6.2

    Edited

    Designs

      Child items ...

      2. Activity

      • Julie Brossard changed title from Performance difference between RH6 and RH7 to Performance differences between RH6 and RH7

        changed title from Performance difference between RH6 and RH7 to Performance differences between RH6 and RH7

      • Laurențiu Nicola
        Laurențiu Nicola @lnicola ·
        Developer

        Can you record a pair of profiles using perf record otbApplicationLauncherCommandLine . UpdateSynthesis, then look at the output using perf report? It should be available in the perf package.

        And does this only happen when the output image type is not float?

        Edited by Laurențiu Nicola
      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        For information these differences were found on the CNES MAJA processing chain. It extensively uses OTB applications.

        After a lot of search we found out that the issue comes from the anormal use of CastImageFilters during the reading/writing of images when using SWIGPointer within a python application. In Valgrind we can see that the number of instructions on RH7 for the memory allocation is quite bigger than on RH6. Since this issue is not python specific we think that this could happen in any case of application branching ( pipelines ...).

        Also we monitored the execution time for each part of applications ( image reading, writing, processing) and the time of writing was almost twice the time on RH7 than RH6. Even with a patch for the OTB issues ( #2042 #2043 ) the problem is still here as MAJA application works on DoubleVectorImages while native OTB applications works in FloatVectorImage.

        MAJA (DoubleVector) -> Cast -> OTB ( FloatVector) -> Cast -> MAJA (DoubleVector)

        We also works on other image types such as Int16 UInt8 ...

        Here is an example of logging on the Cast Filter ( "Casted Bad" means that there was a mismath type ) :

        '''

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:48 Initializing : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:56 {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B2.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B2.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:86 Parameters for BandMath : {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B2.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B2.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:88 Parameters type {'exp': (<class 'str'>, 'str'), 'il': (<class 'list'>, 'list'), 'out': (<class 'str'>, 'str'), 'ram': (<class 'str'>, 'str')}

        273: InputImageParam Casting Input !!

        273: InputImageParam In: UInt8VectorImageType : TImageType

        273: Casted Bad

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:73 Running : BandMath

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #1 (closed) has 1 components

        273:

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #2 (closed) has 16 components

        273:

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:78 Finished : BandMath ,Time: 00 min 00 seconds ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:81 Running : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 post:120 Write output False

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:85 Finished : BandMath ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:48 Initializing : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:56 {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10: (im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG- Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B3.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B3.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:86 Parameters for BandMath : {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B3.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B3.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:88 Parameters type {'exp': (<class 'str'>, 'str'), 'il': (<class 'list'>, 'list'), 'out': (<class 'str'>, 'str'), 'ram': (<class 'str'>, 'str')}

        273: InputImageParam Casting Input !!

        273: InputImageParam In: UInt8VectorImageType : TImageType

        273: Casted Bad

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:73 Running : BandMath

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #1 (closed) has 1 components

        273:

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #2 (closed) has 16 components

        273:

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:78 Finished : BandMath ,Time: 00 min 00 seconds ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:81 Running : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 post:120 Write output False

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:85 Finished : BandMath ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:48 Initializing : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:56 {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B4.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B4.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:86 Parameters for BandMath : {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B4.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B4.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:88 Parameters type {'exp': (<class 'str'>, 'str'), 'il': (<class 'list'>, 'list'), 'out': (<class 'str'>, 'str'), 'ram': (<class 'str'>, 'str')}

        273: InputImageParam Casting Input !!

        273: InputImageParam In: UInt8VectorImageType : TImageType

        273: Casted Bad

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:73 Running : BandMath

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #1 (closed) has 1 components

        273:

        273: 2020-05-05 12:23:48 (INFO) BandMath: Image #2 (closed) has 16 components

        273:

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:78 Finished : BandMath ,Time: 00 min 00 seconds ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:81 Running : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 post:120 Write output False

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _post:85 Finished : BandMath ,RAM current : 74 MB ,RAM max : 852 MB

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:48 Initializing : BandMath

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 init:56 {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG- Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B5.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B5.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:86 Parameters for BandMath : {'exp': '(im2b1 || (im1b1-0.00001)<0)?-10:(im1b12e-05+-0.1)/cos(27.6494306199999970.017453292519943295)', 'il': ['/MAJA_SHARED_FOLDER/MAJA-NG-Data/TV/LANDSAT8/Input/L8-V-L2INIT-001-F-N-LANDSAT8-GENERAL/LC08_L1TP_199029_20170527_20170615_01_T1/LC08_L1TP_199029_20170527_20170615_01_T1_B5.TIF', <Swig Object of type 'ImageBaseType *' at 0x7fbded548f30>], 'out': '/MAJA_SHARED_FOLDER/beongpu/TU/LANDSAT8/Maja_L1Read_ll1furd5/TOAConverter_B5.tif', 'ram': '7974'}

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 pre:88 Parameters type {'exp': (<class 'str'>, 'str'), 'il': (<class 'list'>, 'list'), 'out': (<class 'str'>, 'str'), 'ram': (<class 'str'>, 'str')}

        273: InputImageParam Casting Input !!

        273: InputImageParam In: UInt8VectorImageType : TImageType

        273: Casted Bad

        273: 2020-05-05 12:23:48 [4921] DEBUG MAJA 4.2 _run:73 Running : BandMath

        '''

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        I started to create forked OTB applications for double on BandMath for example and the performance improvements were huge. We also don't have the "Casted Bad" logs as the types are matching.

        However I think that this issue needs a change on the OTB level as it impact many chains ( see WASP ...).

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        As for why RH7 is taking much more instructions/time for small memory allocation ( per pixel VariableLengthVector allocation) we don't know yet.

        • Esquis Benjamin
          Esquis Benjamin @esquisb ·
          Developer

          One solution that I have already tried on my side is to have various DoExecute method to handle various input image types.

          As the input pointer might have different types (Vector or Not, UInt, Float ...) their treatment might differs and uses other filters more appropriate for these types and more importantly will not do any cast. Not all Apps have to be modified for now, only the most commonly used with other types than Float ( the default for OTB apps).

          For example in the BandMath App there is no need to have an ExtractROI for each band if the input image is a single Image.

          Also please note that adding CastImageFilters in the pipeline has performance impact as it artificially enlarge the size of the pipeline.

        • Cédric Traizet
          Cédric Traizet @ctraizet ·
          Developer

          We should make a list of application that would benefit the most from having different DoExecute method. We could start by refactoring image utility applications like Concatenate, BandMath etc. Do you have the list of the OTB applications used in MAJA where the refactoring would make a difference ?

          If the application class starts supporting various DoExecute methods, e.g. DoExecute templated on input type, we need to think about how this would be integrated in the class. It should not be to complex/hard to maintain and the API should not be broken.

          Also how should the case where the application has several inputs be handled ? The number of input is only known at run-time (DoInit).

        • Please register or sign in to reply
      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        I attach the benchmark on MAJA on the CNES Cluster

        Capture

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Dear all,

        We are above 100% on RH7 because of the way the nodes are allocated on RH7. We only takes 8 threads but the loadaverage is for all cores (12) thus when I do loadavg()/nbcores it is more than 100% since nbcores=8. On RH6 we can allocate the correct number of core. These are benchmarks provided by CNES to ensure MAJA performances are not degraded from one version to another. Moreover they ensure that MAJA is always launched with the same configuration anywhere.

        Also it is not a problem with binaries not compatible RH7 ( we already worked this out).

        Regards

      • Luc Hermitte
        Luc Hermitte @lhermitte ·
        Contributor

        If in the end it comes down to static cast VLV, I found an old patch that did not make it into ITK source code.

        Unfortunately, its syntax is really perfectible.

        TOutpixelPixelType o(/*rightsize*/);
for (loop on outIt and inIt)
{
    CastInto(o, someexpression with *inIt);
    outIt.Set(o);
}

        To fix/improve it, we would need support for mono-parameter expression templates and/or use it to provide CastInto<> that resolves into an expression template

        Modules/Core/Common/include/itkPixelTraits.h

        
 namespace itk
 {
/**\ingroup ITKCommon
 * Generic function to cast a pixel into another pixel type.
 * This function is aimed at writing efficient algorithms that cast pixel
 * values. It is particularly important to not degrade performances on
 * \c itk::VariableLentghVector based pixels.
 *
 * This overload is the default case. For numeric pixels, a simple \c
 * static_cast<> should be enough.
 * @tparam TDest Pixel type of the destination pixel
 * @tparam TOrig Pixel type of the origin pixel
 * @param[out] d TDestination pixel that shal receive the value of \c o
 * @param[in]  o TOrigin pixel
 *
 * @throw Whatever a <tt>static_cast<TDest>(o)</tt> may throw.
 *
 * @post a copy is expected to be made. Hence, use this function to copy a
 * pixel value that could be a proxy into another pixel variable. In other
 * cases, \c itk::MoveInto() is to be prefered.
 * @code
 * RealType & value = m_cachedPixel[threadId];
 * CastInto(value, inputImagePtr->GetPixel(index);
 * @endcode
 *
 * \see <tt>itk::CastInto(VariableLengthVector<TDest>, VariableLengthVector<TOrig>)</tt>
 * \todo Handle the case of \c itk::FixedArray<>
 */
template <typename TDest, typename TOrig>
inline
void CastInto(TDest & d, TOrig const& o)
{ d = static_cast<TDest>(o); }

/**\ingroup ITKCommon
 * Generic function to convert a pixel into another pixel type.
 * This function is aimed at writing efficient algorithms that move pixel
 * values around. It is particularly important to not degrade performances on
 * \c itk::VariableLentghVector based pixels.
 *
 * This overload is the default case. For numeric pixels, a simple \c
 * static_cast<> should be enough.
 * @tparam TDest Pixel type of the destination pixel
 * @tparam TOrig Pixel type of the origin pixel
 * @param[out] d TDestination pixel that shal receive the value of \c o
 * @param[in]  o TOrigin pixel
 *
 * @throw Whatever a <tt>static_cast<TDest>(o)</tt> may throw.
 * \see <tt>itk::MoveInto(VariableLengthVector<TDest>, VariableLengthVector<TOrig>)</tt>
 * \see <tt>itk::MoveInto(VariableLengthVector<T>, VariableLengthVector<T>)</tt>
 */
template <typename TDest, typename TOrig>
inline void
MoveInto(TDest & d, TOrig const& o)
{ d = static_cast<TDest>(o); }

        Modules/Core/Common/include/itkVariableLengthVector.h

        // aim: avoid a static cast that generates a new VLV
template <typename TDest, typename TOrig>
inline
void CastInto(VariableLengthVector<TDest> & d, VariableLengthVector<TOrig> const& o)
{ d = o; }

/**
 * Specialized function to convert a vector pixel into another vector pixel type.
 * This function is aimed at writing efficient algorithms that move pixel
 * values around. It is particularly important to not degrade performances on
 * \c itk::VariableLengthVector based pixels.
 *
 * While this overload does not really move (the internals of) \c o into \d
 * (as C++11 rvalue-references would permit), it will nonetheless avoid
 * allocating memory for a new \c VariableLengthVector, thing that would have
 * been a consequence of a \c static_cast<>.
 *
 * Instead, this overload will use the conversion assignment operator from \c
 * VariableLengthVector to copy (and convert on-the-fly) the value from \c o.
 *
 * \tparam TDest Internal pixel type of the destination pixel
 * \tparam TOrig Internal pixel type of the origin pixel
 * \param[out] d Destination pixel that shal receive the value of \c o
 * \param[in]  o Origin pixel
 * \pre \c d shall not be a \c VariableLengthVector that acts as a proxy.
 *
 * \throw None
 * \see <tt>itk::MoveInto(TDest, TOrig)</tt>
 * \see <tt>itk::MoveInto(VariableLengthVector<T>, VariableLengthVector<T>)</tt>
 * \ingroup DataRepresentation
 * \ingroup ITKCommon
 */
template <typename TDest, typename TOrig>
inline
void MoveInto(VariableLengthVector<TDest> & d, VariableLengthVector<TOrig> const& o)
{ d = o; }

        PS: this is an old code. I'm not sure the distinction between MoveInto and CastInto is correctly implemented.

        Edited by Luc Hermitte
      • Luc Hermitte mentioned in issue s1-tiling/s1tiling#10 (closed)

        mentioned in issue s1-tiling/s1tiling#10 (closed)

      • Luc Hermitte
        Luc Hermitte @lhermitte ·
        Contributor

        On the subject, I've observed a lot of cycles spent to convert pixel types in s1tling project while SARCalibration is executed alone -- I guess the same issue can be found in other scenarios.

        Here is a an excerpt of what callgrind can report. callgrind-calib

        After some analysis, two things can permit to reduce conversion cost

        1. Don't convert InternalPixelType -> VariableLengthVector<double> -> OutputPixelType (actually, the less we use VariableLengthVector, the less we waste in memory allocations and releases ; IMO VLV use shall be restricted to application that do indeed manipulate multi-channels images)
        2. For the cases we need to convert in between vectorimages and complex images or other vector images, a pixel component iterator could be used -- I've written the component iterators for array and scalar images, I'll share it once I'm done with complex (number) images.
      • Cédric Traizet
        Cédric Traizet @ctraizet ·
        Developer

        Another solution (or workaround) to your problem would be to define the type of otb applications at compile time (e.g. with a CMake variable). OTB applications could be compiled to use float or double as their main type (for IO and for defining filter types).

        This would reduce the number of cast in processing chains with custom applications using double for precision.

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Indeed this would be a workaround in our case but it won't correct the whole paradigm issue as nothing prevents such cast.

      • Yannick TANGUY
        Yannick TANGUY @ytanguy ·
        Developer

        @esquisb, @ctraizet and @julienosman : could you precise the solution you have implemented and open two issues ?

        • one with that implementation (specialized applications (BandMath, ConcatenateImages, etc.) for int16, float, double pixel type). This implementation gives satisfying performances and could be merged in OTB 7.2
        • a "feature" issue, to keep in mind that we would like to keep genericity (one single app can handle efficiently all image types) and performance. For that issue, it seems that there could be a lot of work to do : it may be planned in OTB 8.x releases.
      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Dear Yannick,

        Bear in mind that the 'MAJA' OTB modified applications are not simply specialized applications for double ,they are extensively modified to bypass most of the ApplicationEngine input machinery. So I don't think they can be merged in the OTB as is without work.

        Regards

        Edited by Esquis Benjamin
      • Yannick TANGUY
        Yannick TANGUY @ytanguy ·
        Developer

        Bear in mind that the 'MAJA' OTB modified applications are not simply specialized applications for double ,they are extensively modified to bypass most of the ApplicationEngine input machinery. So I don't think they can be merged in the OTB as is without work.

        Ok, could you precise your implementation, and the impacts ? When we discussed with @msavinaud, it seemed that it was quite straigth forward to merge your modifications in OTB. But if it has two many side effects, we should discuss again of the best solution, that improves performance and does not delay OTB 7.2.

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Dear Yannick,

        We use the ImageBasePointer then try to cast it into a type to determine it's type then we use a templated DoExecute to do the job.

        For me it is way more than just specializing it to double, here is an exemple of what we do :

        
#define CLAMP_BANDMATH_IMAGE_IF(Out, In, image_base)       \
  {                                               \
    In* in_image = dynamic_cast<In*>(image_base); \
    if (in_image)                                 \
		outIm = Cast<In, Out>(in_image);        \
  }

#define CLAMP_BANDMATH_IMAGE_VECTOR_BASE(T, image_base)                   \
    T* outIm =nullptr;                                                    \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt8VectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, Int16VectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt16VectorImageType, image_base);        \
    CLAMP_BANDMATH_IMAGE_IF(T, FloatVectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, DoubleVectorImageType, image_base);        \

#define CLAMP_BANDMATH_IMAGE_BASE(T, image_base)                          \
  	T* outIm =nullptr;                                                     \
	CLAMP_BANDMATH_IMAGE_IF(T, UInt8ImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, Int16ImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt16ImageType, image_base);              \
    CLAMP_BANDMATH_IMAGE_IF(T, FloatImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, DoubleImageType, image_base);              \

        Regards

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Dear all , Here is the list of applications modified in the current performance branch of MAJA:

        • BandMath
        • ConcatenateImages
        • ExtractROI modifiée en ExtractChannels pour extraire une ou plusieurs bandes

        We also use :

        • ManageNoData
        • BinaryMorphological

        Regards

      • Peter Kettig
        Peter Kettig @petket-5 ·

        Hi Benjamin,

        Is there any way to get these applications merged into a public OTB development branch? As of now, these changes only affect Maja, but it would be good to share them with the OTB community as a whole. @msavinaud mentioned that this might be possible, but since then I haven't heard of any technical details.

        Kind regards, Peter

      • Esquis Benjamin
        Esquis Benjamin @esquisb ·
        Developer

        Hello Peter,

        I understand that you want Maja to use a standard (or publicly available) version of OTB. However as for now my solution is just a 'crappy' workaround that doesn't meet the OTB expectations.

        We can put it in a branch but it can't be merged into develop as it is bypassing most of the OTBApplication API.

        We can't have applications that does this because it would mean that there is something missing in the OTB :

        #define CLAMP_BANDMATH_IMAGE_IF(Out, In, image_base)       \
  {                                               \
    In* in_image = dynamic_cast<In*>(image_base); \
    if (in_image)                                 \
		outIm = Cast<In, Out>(in_image);        \
  }

#define CLAMP_BANDMATH_IMAGE_VECTOR_BASE(T, image_base)                   \
    T* outIm =nullptr;                                                    \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt8VectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, Int16VectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt16VectorImageType, image_base);        \
    CLAMP_BANDMATH_IMAGE_IF(T, FloatVectorImageType, image_base);         \
    CLAMP_BANDMATH_IMAGE_IF(T, DoubleVectorImageType, image_base);        \

#define CLAMP_BANDMATH_IMAGE_BASE(T, image_base)                          \
  	T* outIm =nullptr;                                                     \
	CLAMP_BANDMATH_IMAGE_IF(T, UInt8ImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, Int16ImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, UInt16ImageType, image_base);              \
    CLAMP_BANDMATH_IMAGE_IF(T, FloatImageType, image_base);               \
    CLAMP_BANDMATH_IMAGE_IF(T, DoubleImageType, image_base);              \

        Such mecanism must be available in the OTBApplicationEngine or else we will need to rewrite it for each application.

        Regards.

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply