We'll first want to rewrite the polynomials as single expressions.
#(x^3 + x + 3)(x-1) = x^4 - x^3 +x^2 - x +3x - 3#
#x^4 - x^3 +x^2 - x +3x - 3 = x^4 - x^3 +x^2 + 2x - 3#
#(x-5)^2 = (x - 5)(x - 5) = x^2 - 10x + 25#
Now we need to use long division to find our answer.
Step 1: #x^2# goes into #x^4#, #x^2# times, so we need to multiply our divisor, #x^2 - 10x + 25#, by #x^2#, and subtract that from the dividend, #x^4 - x^3 +x^2 + 2x - 3#.
#x^2 (x^2 - 10x + 25) = x^4 - 10x^3 +25x^2#
#(x^4 - x^3 +x^2 + 2x - 3) - (x^4 - 10x^3 +25x^2) = 9x^3 -24x^2 + 2x - 3#
Step 2: #x^2# goes into #9x^3#, #9x# times. Repeat step 1 with these values.
#9x(x^2 - 10x + 25) = 9x^3 - 90x^2 + 225x#
#(9x^3 -24x^2 + 2x - 3) - (9x^3 - 90x^2 + 225x) = 66x^2 - 223x - 3#
Step 3: #x^2# into #66x^2#, 66 times. Repeat step 1.
#66(x^2 - 10x + 25) = 66x^2 - 660x + 1650#
#(66x^2 - 223x - 3) - (66x^2 - 660x + 1650) = 437x - 1653#
Our three divisors are then added together to find our value, #x^2 + 9x + 66#. However, we have remainder of #437x - 162#, so our answer is #x^2 + 9x + 66 + (437x - 165)/(x^2 - 10x + 25)#