Using the trigonometric identity:
tan^2x = sec^2x-1
we can observe that in general
int tan^nxdx = int (sec^2x-1)tan^(n-2)x dx = int tan^(n-2)xsec^2xdx-int tan^(n-2)x dx
Remembering that:
d/(dx) tan x = sec^2x
the first integral is easy to solve:
int tan^(n-2)x sec^2x dx = int tan^(n-2)x d(tanx) = 1/(n-1)tan^(n-1) +C
So that we get the reduction formula:
int tan^nxdx = 1/(n-1)tan^(n-1)x - int tan^(n-2)x dx
Applying this formula twice we have:
int tan^5xdx =1/4 tan^4x - int tan^3xdx = 1/4 tan^4x -1/2 tan^2x + int tanxdx
We can solve this last integral directly:
int tanx dx = int sinx/cosx dx = int (d(-cosx))/cosx = -ln abs (cosx) + C
So that:
int tan^5xdx = 1/4tan^4x - 1/2 tan^2x - ln abs (cosx) + C