Binomial expansion of (1+ax)^n(1+ax)n is
1+^nC_1ax+^nC_2(ax)^2+^nC_3(ax)^3+........+^nC_n(ax)^n,
where color(white)(x)^nC_r=(n!)/(r!(n-r)!) or (n(n-1)(n-2)....(n-r+1))/(1xx2xx3xx...xxr) and a is a constant.
As coefficient of x^3 is six times the coefficient of x^2
color(white)(x)^nC_3xxa^3=6xx^nC_2xxa^2
i.e. (n(n-1)(n-2))/(1xx2xx3)xxa=6xx(n(n-1))/(1xx2)
or (n-2)/3xxa=6 i.e. a(n-2)=18 .................(1)
Further, coefficient of x is 12, we have
color(white)(x)^nC_1xxa=12 i.e. axxn=12 .................(2)
Putting (2) in (1) we get 12-2a=18
Hence, a=(18-12)/(-2)=-3
and n=12/-3=-4
and hence expansion is (1-3x)^(-4)
= 1+(-4)/1(-3x)+((-4)(-5))/2(-3x)^2+((-4)(-5)(-6))/6(-3x)^3+................
= 1+12x+90x^2+540x^3+..........