We can think of #sqrt(108x^5y^8# as the same as #sqrt(108 xx x^5 xx y^8#. Then we can can square root each number individually to get #6x^2y^4sqrt(3x)#
So the square root of 108 is #6sqrt(3)# because #sqrt108# can also be written as #sqrt(36 xx 3)#. We know the square root of 36 to be 6, so we write 6 outside of the square root. Thus we have #6sqrt(3)# so far.
Then we can do the same for #x^5 xx y^8#. However, when square rooting powers, we divide the power by the root. For example #sqrt(y^8)# becomes #y^4#. For #sqrt(x^5)#, we get #x^2sqrt(x)# because we can only square root 4 so we have a remainder.
We can then put all of these answers together to get #6x^2y^4sqrt(3x)#