Write the integrand as: sec^5(x) = sec^2(x) sec^3(x) and integrate by parts considering that:
d/dx (tanx) = sec^2(x) ,
so:
int sec^5x dx = int sec^2(x) sec^3(x)dx
int sec^5x dx = int sec^3(x)d(tanx)
int sec^5x dx = tanxsec^3x - int tanx d(sec^3(x))
and as:
d/dx (sec^3(x)) = 3sec^2(x) d/dx sec(x) = 3sec^3(x) tanx
we have:
int sec^5x dx = tanxsec^3x - 3int tan^2x sec^3x dx
use now the trigonometric identity:
tan^2 theta = sin^2 theta/cos^2 theta = (1-cos^2 theta)/cos^2theta = sec^2theta -1
to have:
int sec^5x dx = tanxsec^3x - 3int (sec^2x -1) sec^3x dx
and using the linearity of the integral:
int sec^5x dx = tanxsec^3x + 3int sec^3x dx -3 int sec^5x dx
The integral now appears on both sides of the equation and we can solve for it obtaining a reduction formula:
int sec^5x dx = 1/4(tanxsec^3x + 3int sec^3x dx)
Solve now the resulting integral with the same procedure:
int sec^3x dx = int secx d(tanx)
int sec^3x dx = tanxsecx - int tanx d(secx)
int sec^3x dx = tanxsecx - int tan^2x secx dx
int sec^3x dx = tanxsecx - int (sec^2x-1) secx dx
int sec^3x dx = tanxsecx + int secx dx - int sec^3x dx
int sec^3x dx = 1/2(tanxsecx + int secx dx)
To solve the resulting integral note that:
d/dx (tanx + secx) = sec^2x +secx tanx = secx(tanx+secx)
so dividing and multiplying the integrand by (secx+tanx):
int secx dx = int (secx(secx+tanx))/(secx+tanx) dx
int secx dx = int (d(secx+tanx))/(secx+tanx)
int secx dx = ln abs(secx+tanx) +C
Putting it all together:
int sec^5x dx = (2tanxsec^3x+ 3tanxsecx + 3ln abs(secx+tanx))/8 +C