Start by noticing that both x = 1 and x = -1 are solutions, so (x - 1) and (x + 1) are both factors.
Then solve:
(x - 1)(x + 1)(ax^3 + bx^2 + cx + d) = x^5 - 5x^4 - x^3 + x^2 + 4
for a, b, c and d:
Expanding:
(x - 1)(x + 1)(ax^3 + bx^2 + cx + d) = (x^2 - 1)(ax^3 + bx^2 + cx + d)
= ax^5 + bx^4 + (c - a)x^3 + (d - b)x^2 - cx - d
Comparing the coefficients of x^5, x^4, etc., we can quickly see that a = 1, b = -5, c = 0 and d = -4.
So, thus far we have:
x^5 - 5x^4 - x^3 + x^2 + 4 = (x - 1)(x + 1)(x^3 - 5x^2 - 4).
Differentiating the cubic x^3 - 5x^2 - 4 gives 3x^2 - 10x. This is zero for x = 0 and x = 10/3. These are the x coordinates of the turning points of the cubic. Substituting each of these two values of x into the cubic give negative values, so the cubic has only 1 real root - the other two are complex numbers. Notice that x^3 - 5x^2 - 4 is negative for x = 5 and positive for x = 6. So the real root lies somewhere between 5 and 6, but it is not a rational number, let alone an integer. So the factorisation above is 'complete'.