The simplest means of doing this is to graph the function.
graph{x^5lnx [-10, 10, -5, 5]}
The function exists on the interval from (0,oo). On this interval x^5 is always positive, and ln(x) is negative until x=1. Looking at the graph, we know that the function will be concave upwards and increasing after x=1, but via taking the derivatives we can find when exactly the change from decreasing to increasing occurs.
We must use the product rule for this, which states that for f(x)=g(x)h(x), f'(x) = g'(x)h(x)+g(x)h'(x). Thus here...
f(x) = x^5ln(x) -> f'(x) = 5x^4lnx + x^5/x = 5x^4lnx + x^4 = x^4(5lnx +1).
We can find where the change from f'(x)<0 to f'(x)>0 occurs by finding where f'(x)=0. From our equation, this cannot happen at x=0 because the natural log is undefined there. Therefore, setting the equation equal to zero...
f'(x) = x^4(5lnx +1) = 0 -> 5lnx +1 = 0 -> lnx = -1/5 -> x = e^(-1/5) is our critical point. Plugging in values slightly larger and slightly smaller than this for x reveals that for x < e^(-1/5), f'(x)<0, and for x>e^(-1/5), f'(x)>0. Therefore the function is decreasing on (0,e^(-1/5)), and increasing on (e^(-1/5), oo)
To find the concavity, we must instead find f''(x) in the same way as we found the first derivative:
f''(x) = d/dx(5x^4lnx + x^4) = (20x^3lnx + 5x^3 + 4x^3 = 20x^3lnx + 9x^3 = x^3(20lnx + 9).
The concavity can only switch at a point where this second derivative is equal to zero. As above, we know that x=0 is not a zero of this second derivative because the natural log is undefined there...
f''(x) = 0 = x^3(20lnx + 9) = 20lnx + 9 -> 20lnx = -9 -> lnx = -9/20 -> x = e^(-9/20)
This is our zero for the second derivative. As above, plugging in a slightly smaller x (i.e. e^(-1/2)) yields a negative value (lnx = -1/2< -9/20 -> 20lnx+9<0 -> x^3(20lnx+9)<0), and using a larger x yields a positive value, i.e. x=e -> ln e = 1 >-9/20 -> e^3(20lne+9) = e^3(20+9) = e^3(29)>0. Thus, the function is concave down for (0,e^(-9/20)), and concave up for (e^(-9/20),oo)