We should indeed use the polar coordinate system:
#x=r cos phi#
#y=r sin phi#
When you substitute these in #x^2+y^2#, you get
#x^2+y^2 = (r cos phi)^2 + (r sin phi)^2 #
#= r^2 cos^2 phi+ r^2 sin^2 phi#
#= r^2 (cos^2 phi+ sin^2 phi) #
#= r^2#
for #cos^2+sin^2 = 1#
Our area becomes #9 <=r^2<=64# by substituting #x^2+y^2=r^2#.
Therefore, our limits for #r# are: #3<=r<=8#, by taking the square root.
When we use polar coordinates, we should take the Jacobian(#=J(r,phi)#) into account, which is equal to #r# for polar coordinates. Our integral becomes:
#int sin(x^2+y^2)dr = int sin(r^2)*J(r,phi)dr = int_3^8 r sin(r^2)dr#
To solve this integral, we need to use substitution. We substitute #u=r^2#, so #du=2rdr#, in other words #dr = 1/(2r) du#.
#int_3^8 r sin r^2 dr = int_9^64 r sin u 1/(2r) du = int_9^64 1/2 sinu du = [-1/2cos(u)]_9^64 =(cos(9)-cos(64))/2#
