I'm really happy you asked this question because the process of solving it is very interesting.
We have to be clever to start off here. No substitutions seem to fit, nor does integration by parts. If instead we had int tanxsec^2xdx, we could make the simple substitution u=tanx, du=sec^2xdx, which would make the integral intudu. Unfortunately, that is not the case.
If we remember our Pythagorean Identities, though, we can make some progress. Recall that tan^2x+1=sec^2x, or tan^2x=sec^2x-1. Using this, we can rewrite the integral as:
int (sec^2x-1)secxdx
=int sec^3x-secxdx
The sum rule for integrals says we can further simplify it to:
intsec^3xdx-intsecxdx
Second Integral
We start here because 1) we can and 2) intsecxdx is fairly simple: it is equal to lnabs(secx+tanx). It's the other one that poses a fairly interesting challenge.
First Integral
Before going on to this one, let's review our entire problem:
inttan^2xsecxdx=intsec^3xdx-lnabs(secx+tanx)
Now we have to think about how to do intsec^3xdx. Note that we can rewrite it as:
intsec^2xsecxdx
Note also that intsec^2xdx=tanx. If only we could simplify it like that...
But wait, we can! Using integration by parts, with u=secx and dv=sec^2xdx:
color(blue)(u=secx)
(du)/dx=secxtanx->color(blue)(du=secxtanxdx)
dv=sec^2xdx->intdv=intsec^2xdx->color(red)(v=tanx)
The integration by parts formula is:
intudv=color(blue)(u)color(red)(v)-intcolor(red)(v)color(blue)(du)
Plugging in what we just found,
intsec^3dx=color(blue)(secx)color(red)(tanx)-intcolor(red)(tanx)color(blue)(secxtanxdx)
And simplifying,
intsec^3dx=color(blue)(secx)color(red)(tanx)-intcolor(red)(tan^2x)color(blue)(secxdx)
Isn't that strange! We end up with inttan^2xsecxdx, which is the integral in our original problem! Now we have:
inttan^2xsecxdx=secxtanx-inttan^2xsecxdx-lnabs(secx+tanx)
We can add the inttan^2xsecxdx on the right to the left to get:
2inttan^2xsecxdx=secxtanx-lnabs(secx+tanx)
inttan^2xsecxdx=(secxtanx-lnabs(secx+tanx))/2+C (don't forget the integration constant!)
And that is our final answer.
