mag(x):=sqrt(x.x)$ normalize(x):=(1/mag(x))*x$

Define a basis B

b1:[1,0,2]$
b2:[2,1,0]$
b3:[0,1,1]$
B:transpose(matrix(b1,b2,b3));

Gram-Schmidt

a1:normalize(b1);
(b2.a1)*a1; b2-%; a2:normalize(b2-(b2.a1)*a1)$
a3:ratsimp(normalize(b3-(b3.a2)*a2-(b3.a1)*a1))$
Q:transpose(matrix(a1,a2,a3));

Check that our Q (=A) is orthogonal

transpose(Q).Q$
ratsimp(%);

Gram-Schmidt, change of basis matrix R (=S)

R:ratsimp(matrix(
[mag(b1),a1.b2,a1.b3],
[0,mag(b2-(b2.a1)*a1),a2.b3],
[0,0,mag(b3-(b3.a2)*a2-(b3.a1)*a1)]));

Check: B=QR

ratsimp(B-Q.R);