Answer:
Remember, if [tex]\{x_1,x_2,...,x_k\}[/tex] is a basis for a subspace W of [tex]\mathbb{R}^n[/tex] then [tex]\{q_1,q_2,...,q_k\}[/tex] is an orthonormal basis of W, where
[tex]q_i=\frac{1}{||v_i||}[/tex] and [tex]v_i[/tex] is defined as:
[tex]v_1=x_1\\v_2=x_2-\frac{v_1\cdot x_2}{v_1\cdot v_1}v_1\\v_k=x_k-\frac{v_1\cdot x_k}{v_1\cdot v_1}v_1 - \cdots - \frac{v_{k-1}\cdot x_k}{v_{k-1}\cdot v_{k-1}}v_{k-1}[/tex]
Then, to find a orthonormal basis of [tex]\{x_1=(8, -6, 0), x_2=(5, 1, 0), x_3=(0, 0, 2)\}[/tex] we will find first the [tex]v_i[/tex]'s.
[tex]v_1=x_1[/tex]
[tex]v_2=x_2-\frac{v_1\cdot x_2}{v_1\cdot v_1}v_1=(5,1,0)-\frac{(8,-6,0)\cdot (5,1,0)}{(8,-6,0)\cdot (8,-6,0)}(8,-6,0)=\\=(5,1,0)-\frac{17}{50}(8,-6,0)=(\frac{57}{25},\frac{76}{25},0)[/tex]
[tex]v_3=x_3-\frac{v_1\cdot x_3}{v_1\cdot v_1}v_1-\frac{v_2\cdot x_3}{v_2\cdot v_2}v_2=\\=(0,0,2)-\frac{(8-6,0)\cdot (0,0,2)}{(8,-6,0)\cdot (8,-6,0)}(8,-6,0) -\frac{(\frac{57}{25},\frac{76}{25},0)\cdot (0,0,2)}{(\frac{57}{25},\frac{76}{25},0)\cdot (\frac{57}{25},\frac{76}{25},0)}(\frac{57}{25},\frac{76}{25},0)=\\=(0,0,2)-0-0=\\=(0,0,2)[/tex]
Therefore, [tex]\{v_1=(8,-6,0),v_2=(\frac{57}{25},\frac{76}{25},0),v_3=(0,0,2)\}[/tex] is a ortogonal basis for [tex]\mathbb{R}^3[/tex]. But we need a orthonormal basis. Then is enough find the corresponding unit vector of the ortogonal basis found.
[tex]q_1=\frac{1}{||v_1||}v_1=\frac{1}{\sqrt{100}}(8,-6,0)\\q_2=\frac{1}{||v_2||}v_2=\frac{1}{\frac{19}{5}}(\frac{57}{25},\frac{76}{25},0)=\frac{5}{19}(\frac{57}{25},\frac{76}{25},0)\\q_3=\frac{1}{||v_3||}v_3=\frac{1}{2}(0,0,2)[/tex]
Hence
[tex]\{q_1=({\frac{8}{\sqrt{100}},\frac{-6}{\sqrt{100}},0), q_2=(\frac{3}{5},\frac{4}{5},0), q_3=(0,0,1)\}[/tex] is a orthonormal basis for [tex]\mathbb{R}^3[/tex]
