well, let's bear in mind that the hypotenuse is never negative, since it's just a radius unit.
now, π < θ < 3π/2, is another way of saying, the angle θ is in the III Quadrant, where as you already know, the sine and cosine as well as opposite and adjacent sides are both negative.
[tex]\bf sin(\theta )=\cfrac{\stackrel{opposite}{-1}}{\stackrel{hypotenuse}{5}}\qquad \impliedby \textit{let's find the \underline{adjacent side}} \\\\\\ \textit{using the pythagorean theorem} \\\\ c^2=a^2+b^2\implies \pm\sqrt{c^2-b^2}=a \qquad \begin{cases} c=hypotenuse\\ a=adjacent\\ b=opposite\\ \end{cases}[/tex]
[tex]\bf \pm\sqrt{5^2-(-1)^2}=a\implies \pm\sqrt{24}=a\implies \stackrel{III~Quadrant}{-\sqrt{24}=a}\implies -2\sqrt{6}=a \\\\[-0.35em] ~\dotfill\\\\ ~\hfill cos(\theta )=\cfrac{\stackrel{adjacent}{-2\sqrt{6}}}{\stackrel{hypotenuse}{5}}~\hfill[/tex]
