I = inttan^2(x)*sec^3(x)dx
I = int(sec^2(x)-1)*sec^3(x)dx
I = intsec^5(x)dx - intsec^3(x)dx
For the second integral, you need integration by parts, say
u = sec(x), so du = sec(x)tan(x) and
dv = sec^2(x) so v = tan(x)
intsec^3(x)dx = sec(x)tan(x) - intsec(x)tan^2(x)dx
intsec^3(x)dx = sec(x)tan(x) - intsec(x)(sec^2(x)-1)dx
intsec^3(x)dx = sec(x)tan(x) - int(sec^3(x)-sec(x))dx
intsec^3(x)dx = sec(x)tan(x) - intsec^3(x)dx -intsec(x)dx
2intsec^3(x)dx = sec(x)tan(x) - intsec(x)dx
The last integral is a tabled one, so
intsec^3(x)dx = (sec(x)tan(x)+ln|sec(x)+tan(x)|)/2 + c
For intsec^5dx, using integration by parts
u = sec^3(x) so du = 3tan(x)sec^3(x)
dv = sec^2(x) so v = tan(x)
intsec^5dx = sec^3(x)tan(x) - inttan(x)*3tan(x)sec^3(x)dx
intsec^5dx = sec^3(x)tan(x) - 3inttan^2(x)sec^3(x)dx
But we originally had that
inttan^2(x)sec^3(x)dx = intsec^5(x)dx - intsec^3(x)dx
So substituting that we have
inttan^2(x)sec^3(x)dx = sec^3(x)tan(x) - 3inttan^2(x)sec^3(x)dx - (sec(x)tan(x)+ln|sec(x)+tan(x)|)/2
4inttan^2(x)sec^3(x)dx = (2sec^3(x)tan(x) - sec(x)tan(x)-ln|sec(x)+tan(x)|)/2
inttan^2(x)sec^3(x)dx = (2sec^3(x)tan(x) - sec(x)tan(x)-ln|sec(x)+tan(x)|)/8