First we expand that:
cos(arctan(-2) + arccos(5/13)) =
cos(arctan(-2))cos(arccos(5/13)) - sin(arctan(-2))sin(arccos(5/13))
We know that cos(arccos(theta)) = theta so we can rewrite that to
(5cos(arctan(-2)))/13 - sin(arctan(-2))sin(arccos(5/13))
We know that sin^2(theta) = 1 - cos^2(theta), so
sin^2(arccos(5/13)) = 1 - (5/13)^2 = 1 - 25/169 = 144/169
Taking the root
sin(arccos(5/13)) = 12/13
It's positive because the sine is always positive in the arccosine range. It can let us rewrite the first expansion to:
(5cos(arctan(-2)))/13 - (12sin(arctan(-2)))/13
Now, from sin^2(theta) + cos^2(theta) = 1, if we divide both sides by cos^2(theta) we get tan^2(theta) + 1 = 1/cos^2(theta). So:
tan^2(arctan(-2)) + 1 = 1/cos^2(theta) rarr cos^2(theta) = 1/((-2)^2 + 1)
cos^2(theta) = 1/5 rarr cos(theta) = 1/sqrt(5) = sqrt(5)/5
It's positive because on the range of the arctangent, the cosine is always postive. So we can rewrite it to:
sqrt(5)/13 - (12sin(arctan(-2)))/13
Since the tangent is negative, and the cosine positive, it means the sine is negative. Using sin^2(theta) + cos^2(theta) =1 we have that
sin^2(arctan(-2)) = 1 - 1/5 = 4/5
sin(arctan(-2)) = -2/sqrt(5) = -(2sqrt(5))/5
Which means we can rewrite it all to:
sqrt(5)/13 - (12(-2sqrt(5)/5))/13
From there on it's algebra:
(5sqrt(5))/65 + (24sqrt(5))/65 =
(29sqrt(5))/65