Use Integration by parts
int x^2 e^(2x) dx = int x^2 (1/2 e^(2x))^prime dx
= 1/2 x^2 e^(2x) - int (x^2)^prime * 1/2 e^(2x) dx
= 1/2 x^2 e^(2x) - int x e^(2x) dx
= 1/2 x^2 e^(2x) - int x (1/2 e^(2x))^prime dx
= 1/2 x^2 e^(2x) - ( 1/2 x e^(2x) - int (x)^prime * 1/2 e^(2x) dx )
= 1/2 x^2 e^(2x) - 1/2 x e^(2x) + int 1/2 e^(2x) dx
= 1/2 x^2 e^(2x) - 1/2 x e^(2x) + 1/4 e^(2x) + C
= 1/2 e^(2x) (x^2 - x + 1/2 ) + C