Rewrite the integrand using tan^2x = sec^2x-1.
Let's give the integral we want the name I
I = int tan^2xsec^3x dx = int (sec^5x-sec^3x)dx
Next we'll integrate sec^5x by parts.
int sec^5x dx = int sec^3 x sec^2x dx
Let u = sec^3 x and dv = sec^2x dx.
Then du = 3tanx sec^3x dx and v = tanx
We get
int sec^5 x dx = sec^3x tanx - 3int tan^2x sec^3x dx
Again, use tan^2x = sec^2 x-1 to get
int sec^5 x dx = sec^3x tanx - 3int (sec^2 x-1) sec^3x dx
int sec^5 x dx = sec^3x tanx - 3int sec^5 dx + 3int sec^3x dx
Which gets us
4int sec^5 x dx = sec^3x tanx + 3int sec^3x dx
and
int sec^5 x dx = 1/4sec^3x tanx + 3/4 int sec^3x dx
Recalling that the other integral we need is int sec^3x dx, let's simplify our lives by writing:
I = intsec^5xdx-intsec^3xdx
= underbrace(1/4sec^3x tanx + 3/4 int sec^3x dx)_(intsec^5 x dx)-intsec^3xdx
= 1/4sec^3x tanx -1/4 int sec^3x dx
So now find int sec^3x dx = int secx sec^2x dx using the same general approach and integration by parts. You should get
int sec^3x dx = 1/2secxtanx - 1/2int secx dx
So at this point we have
I = 1/4sec^3x tanx -1/4underbrace( [1/2secxtanx - 1/2int secx dx])_(intsec^3 x dx)
= 1/4sec^3x tanx -1/8secxtanx + 1/8 int secx dx
Now int secx dx can be evaluated in a couple of ways, the usual trick is to multiply by (secx+tanx)/(secx+tanx) to get int 1/u du = ln absu = ln abs(secx+tanx)
So finally we finish with
I = 1/4sec^3x tanx - 1/8secxtanx + 1/8 ln abs(sec x + tan x ) +C
Alternative form for int secx dx
intsecx dx can also be found by substitution and partial fractions to get 1/2ln abs((sinx+1)/(sinx-1))+C
(Yes, the is equivalent to ln abs(secx+tanx)+C)