From the above diagram,
#hat (11pi)/6# is in IV quadrant where cos is positive.
#cos ((11pi)/6) = cos (((11pi)/6) - 2pi) = cos -(pi/6) = cos (pi/6)#
But #cos (pi/6) = cos 60 = x = 1/2# #:. color(red)(cos ((11pi)/6) = 1/2#
#hat (5pi)/4# is in III Quadrant where sin is negative.
#sin ((5pi)/4) = sin (pi + (pi/4)) = - sin (pi/4) = - sin 45#
But #sin (pi/4) = 1/sqrt2#
#:. color(green)(sin ((5pi)/4) = y = -1/sqrt2#
Returning to the given sum,
#6 cos ((11pi)/6) - 2 sin ^2 ((5pi)/4) = 6 * (1/2) - 2 (-(1/sqrt2))^2#
# => (6 * (1/2)) -(2* (1/2)) = 3 - 1 = 2#