I think the easiest way to get started, would be to note that #sin^2(x) = 1-cos^2(x)#
So looking at this integral, we have #intsin^5(x)* cos^2(x) dx = intsin(x) * (sin^2(x))^2 * cos^2(x)dx = intsin(x) * (1-cos^2(x))^2 * cos^2(x)dx#
Why do it like this?
Because you usually want to look ahead at what is going to happen, after you have found the derivative of #u#.
In this case, using the fact that #sin^2(x) = 1-cos^2(x)#, I can rewrite and use #u'# to get rid of either one of trigonometric functions. Had I chosen to get rid of all instances of #cos(x)#, I would have ended up with a square root. Getting rid of #sin(x)# on the other hand, you can already see from now that I won't get any nasty square roots, and it looks like it's just gonne be adding together powers of #u#. That is much simpler to do.
So, let's continue :)
#u = cos(x)#
#(du)/dx = -sin(x) iff dx = (du)/-sin(x)#
#int(sin(x)*(1-u^2)^2*u^2)/-sin(x)du = -intu^2*(1-u^2)^2du#
#=-intu^2*(1+u^4-2u^2)du#
#=-intu^2+u^6-2u^4du#
#= -intu^2du-intu^6du-int-2u^4du#
#=-1/3u^3-1/7u^7+2/5u^5+k#
#=2/5cos^5(x)-1/3cos^3(x)-1/7cos^7(x)+k#
Here's a double check: https://www.wolframalpha.com/input/?i=derivative(2%2F5cos%5E5(x)-1%2F3cos%5E3(x)-1%2F7cos%5E7(x)%2Bk)
