Remember that
cos^3(x) = cos^2(x)*cos(x)
cos^2(x) = 1 - sin^2(x)
cos^3(x) = (1-sin^2(x))cos(x)
so
I = int_(pi/8)^((11pi)/12)cos^3(x) - sin(x)dx
equals
int_(pi/8)^((11pi)/12)(1-sin^2(x))cos(x)dx - int_(pi/8)^((11pi)/12)sin(x)dx
On the left integral if we say u = sin(x), we have
du = cos(x) dx
So rewriting (and
int_(sin(pi/8))^(sin((11pi)/12))(1-u^2)du - int_(pi/8)^((11pi)/12)sin(x)dx
Both integrals are standard so all we gotta do now is evaluate them
(sin(x)-sin^3(x)/3+cos(x))|_(pi/8;(11pi)/12)
(sqrt(3)-1)/(2 sqrt(2))-(sqrt(3)-1)^3/(48 sqrt(2))-(1+sqrt(3))/(2 sqrt(2)) - (sqrt(125/144+161/(144 sqrt(2))))
I ~= 2.001
