Fixing this problem was painful.. Anyway, it’s done!

Use —debug-trycompile to see why it doesn’t work:  I found out the problem was that the incremental linker under debug mode in Visual Studio 10 (Win 64) was turned on when CMake checking the C compiler. Because the CMakeTmp project can be successfully built after the incremental linker was turned off in the IDE. See detailed operation: http://www.cmake.org/pipermail/cmake/2013-March/053829.html

Disable MSVC incremental linker of debug mode in CMake: I changed the incremental setting in Windows-MSVC.cmake under <cmake path>\share\cmake-2.8\Modules\Platform, then the error got fixed. This page may be helpful for tracking the problem: http://cmake.3232098.n2.nabble.com/How-to-turn-off-incremental-linking-for-MSVC-Debug-and-RelWithDebInfo-targets-td4536618.html

The original file turns on the incremental linker for debug mode: see SETTING 1 below. Then I disabled it: see SETTING 2.

>> SETTING 2

# add /debug and /INCREMENTAL:NO to DEBUG and RELWITHDEBINFO also add pdbtype
# on versions that support it
set( MSVC_INCREMENTAL_NO_FLAG "/INCREMENTAL:NO")

if (CMAKE_COMPILER_SUPPORTS_PDBTYPE)
set (CMAKE_EXE_LINKER_FLAGS_DEBUG_INIT “/debug /pdbtype:sept ${MSVC_INCREMENTAL_NO_FLAG}”)
set (CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO_INIT “/debug /pdbtype:sept ${MSVC_INCREMENTAL_NO_FLAG}”)
else ()
set (CMAKE_EXE_LINKER_FLAGS_DEBUG_INIT “/debug ${MSVC_INCREMENTAL_NO_FLAG}”)
set (CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO_INIT “/debug ${MSVC_INCREMENTAL_NO_FLAG}”)
endif ()

>> SETTING 1

# add /debug and /INCREMENTAL:YES to DEBUG and RELWITHDEBINFO also add pdbtype
# on versions that support it
set( MSVC_INCREMENTAL_YES_FLAG "")
if(NOT MSVC_INCREMENTAL_DEFAULT)
set( MSVC_INCREMENTAL_YES_FLAG "/INCREMENTAL:YES")
else()
set( MSVC_INCREMENTAL_YES_FLAG "/INCREMENTAL" )
endif()

if (CMAKE_COMPILER_SUPPORTS_PDBTYPE)
set (CMAKE_EXE_LINKER_FLAGS_DEBUG_INIT “/debug /pdbtype:sept ${MSVC_INCREMENTAL_YES_FLAG}”)
set (CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO_INIT “/debug /pdbtype:sept ${MSVC_INCREMENTAL_YES_FLAG}”)
else ()
set (CMAKE_EXE_LINKER_FLAGS_DEBUG_INIT “/debug ${MSVC_INCREMENTAL_YES_FLAG}”)
set (CMAKE_EXE_LINKER_FLAGS_RELWITHDEBINFO_INIT “/debug ${MSVC_INCREMENTAL_YES_FLAG}”)
endif ()

The weird thing of this error is that it never appeared before. I checked more code in the original Windows-MSVC.cmake and found one block of code which is a likely cause of the error:

if("${MSVC_VERSION}" GREATER 1599)
set(MSVC_INCREMENTAL_DEFAULT ON)
endif()

Maybe MSVC_INCREMENTAL_DEFAULT was always off before, but since my laptop does automatic update for windows applications… MSVC_VERSION may became greater than 1599 after a recent update…