[cfe-users] Boost.GIL test failing with clang 5.x while pass with 15 gcc/clang versions

[Mateusz Loskot via cfe-users]

Hi,

While testing Boost.GIL library [1] with gcc and clang, I noticed a
peculiar issue.
One particular test is failing with lang 5.x while passing with total of 15-17
other versions of clang and GCC (total workflow of CircleCI with at [2]).

Below is extracted minimal program equivalent to Boost.GIL
channel_invert algorithm. It includes two variants: plain expression
and the same expression wrapped with a function template:

#include <limits>
#include <iostream>
#include <typeinfo>

template <typename C>
inline C channel_invert1(C x)
{
    return std::numeric_limits<C>::max() - x + std::numeric_limits<C>::min();
}

template <typename C>
inline C channel_invert2(C x)
{
    return (x - std::numeric_limits<C>::max()) * (-1) +
std::numeric_limits<C>::min();
}

int main()
{
    int x = std::numeric_limits<int>::min();
    std::cout << x << std::endl;

    // plain expressions
    int x_invert1 = std::numeric_limits<int>::max() - x +
std::numeric_limits<int>::min();
    int x_invert2 = (x - std::numeric_limits<int>::max()) * (-1) +
std::numeric_limits<int>::min();
    std::cout << x_invert1 << std::endl;
    std::cout << x_invert2 << std::endl;

    // the same expressions wrapped in function template
    std::cout << channel_invert1<int>(x) << std::endl;
    std::cout << channel_invert2<int>(x) << std::endl;
}

If compiled as optimised variant (-O2 or -O3) with clang 5.x outputs
the following:

-2147483648
2147483647
2147483647
-1
-1

The last two negative one is not expected.

If compiled  with clang 3.9, 4.0 or gcc from 5.1 to 7.3 it outputs

-2147483648
2147483647
2147483647
2147483647
2147483647

Could anyone  help me to understand what is going on in the clang 5 case?
Or, what UB is this hitting?

[1] https://github.com/boostorg/gil/issues/89
[2] https://circleci.com/workflow-run/3a14dd64-6c38-46b2-a6da-678c0075ca27

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net