I am assuming that a > 0
Depending on the domain of integration, we can integrate
int sqrt(x^2(x^2-a)) dx = int absx sqrt(x^2-a) dx by substitution.
Note that the domain of the integrand is (-oo,-sqrta] uu [sqrta,oo)
On (-oo,-sqrta], we have sqrt(x^2(x^2-a)) = -xsqrt(x^2-a)
So
int sqrt(x^2(x^2-a)) = int -xsqrt(x^2-a) = -1/3(x^2-a)^(3/2) +C
On [sqrta,oo), we have sqrt(x^2(x^2-a)) = xsqrt(x^2-a)
So
int sqrt(x^2(x^2-a)) = int xsqrt(x^2-a) = 1/3(x^2-a)^(3/2) +C
Note
Because sqrt(x^2)/x = absx/x = {(1,"if",x < 0),(-1,"if",x < 0):}, we could write the answer as
sqrt(x^2)/(3x)(x^2-a)^(3/2) +C or as
(x^2(x^2-a))^(3/2)/(3x^3)