Total Mass: We can write the total mass M of the rod in terms of the coefficients A and B, of the linear mass density, and its length L.
M = \int_0^L \quad \lambda(x).dx = \int_0^L\quad (A+Bx).dx
M = A(L-0) + B/2(L^2-0) = AL + BL^2/2 ...... (Eq 1)
Centre-of-Mass:
x_{cm} = 1/M\int\quad x.dm
If \lambda(x) is the linear mass density (density per unit length) at a distance x from the less-heavy end,
dm = \lambda(x)dx = (A+Bx).dx
x_{cm} = 1/M\int_0^Lx.(A+Bx).dx
x_{cm} = 1/M[A/2(L^2-0^2) + B/3(L^3-0^3)],
x_{cm}=1/M[A/2L^2+B/3L^3]......(Eq 2)
Substituting for M using Eq 1 in Eq 2,
x_{cm} = \frac{A/2L^2+B/3L^3}{AL+B/2L^2}=(\frac{A/2+B/3L}{A+B/2L})L
x_{cm} = (\frac{A+2/3BL}{2A+BL})L