First, note that \int ln(x) dx can be done by parts by letting u=ln(x), dv=dx so that du=dx/x and v=x, leading to \int ln(x) dx=x ln(x)-\int dx=x ln(x)-x+C.
Now, for \int (ln(x))^2 dx, use integration-by-parts again with u=ln(x) and dv=ln(x) dx so that du = dx/x and v=x ln(x)-x. Then \int(\ln(x))^2 dx=x(ln(x))^2-x ln(x)-\int (ln(x)-1)dx=
x(ln(x))^2-x ln(x)+x-(x ln(x)-x)
=x(ln(x))^2-2xln(x)+2x+C.