The points $A$ and $B$ have position vectors $\mathbf{a}$ and $\mathbf{b}$ respectively. $C$ is the point on line $OB$ such that $AC$ is perpendicular to $OB$.

1. Express the displacement vector $\overrightarrow{AC}$ in terms of a scalar parameter.

Since point $C$ lies on the line $OB$, $\overrightarrow{OC}$ is collinear with $\mathbf{b}$. Let $\overrightarrow{OC} = k\mathbf{b}$ for some scalar $k$.
Using vector subtraction, $\overrightarrow{AC} = \overrightarrow{OC} - \overrightarrow{OA} = k\mathbf{b} - \mathbf{a}$.

2. Set up an equation using the scalar product based on the perpendicularity condition.

We are given that $AC$ is perpendicular to $OB$. Therefore, their scalar product is zero:
$\overrightarrow{AC} \cdot \overrightarrow{OB} = 0$
$(k\mathbf{b} - \mathbf{a}) \cdot \mathbf{b} = 0$

3. Solve for the parameter $k$ and substitute back to find $\overrightarrow{OC}$.

Expanding the scalar product:
$k(\mathbf{b} \cdot \mathbf{b}) - \mathbf{a} \cdot \mathbf{b} = 0$
Using the property that the dot product of a vector with itself is the square of its magnitude ($\mathbf{b} \cdot \mathbf{b} = |\mathbf{b}|^2$):
$k|\mathbf{b}|^2 = \mathbf{a} \cdot \mathbf{b}$
$k = \frac{\mathbf{a} \cdot \mathbf{b}}{|\mathbf{b}|^2}$
Substitute $k$ back into the expression for $\overrightarrow{OC}$:
$\overrightarrow{OC} = \frac{(\mathbf{a} \cdot \mathbf{b})}{|\mathbf{b}|^2}\mathbf{b} \quad \text{(shown)}$

1. Identify the expression as the magnitude of $\overrightarrow{OC}$.

From part (a), we have $\overrightarrow{OC} = \frac{\mathbf{a} \cdot \mathbf{b}}{|\mathbf{b}|^2}\mathbf{b}$.
Taking the magnitude on both sides:
$|\overrightarrow{OC}| = \left| \frac{\mathbf{a} \cdot \mathbf{b}}{|\mathbf{b}|^2}\mathbf{b} \right| = \frac{|\mathbf{a} \cdot \mathbf{b}|}{|\mathbf{b}|^2}|\mathbf{b}| = \frac{|\mathbf{a} \cdot \mathbf{b}|}{|\mathbf{b}|}$
Therefore, the expression represents the length of the line segment $OC$.

1. Find the vector $\overrightarrow{AB}$ and express its squared magnitude in terms of $h$.

$\overrightarrow{AB} = \mathbf{b} - \mathbf{a} = \begin{pmatrix} 1 \\ h \\ 0 \end{pmatrix} - \begin{pmatrix} -3 \\ 3 \\ -1 \end{pmatrix} = \begin{pmatrix} 4 \\ h - 3 \\ 1 \end{pmatrix}$

$|\overrightarrow{AB}|^2 = 4^2 + (h-3)^2 + 1^2$

2. Set up an equation using the given length of $AB$ and solve for $h$.

$16 + (h-3)^2 + 1 = 5^2$
$(h-3)^2 + 17 = 25$
$(h-3)^2 = 8$
$h - 3 = \pm \sqrt{8}$
$h = 3 \pm 2\sqrt{2}$

3. Use the condition that angle $AOB$ is obtuse to establish an inequality for $h$.

For angle $AOB$ to be obtuse, $\cos(\angle AOB) < 0$, which means the scalar product $\mathbf{a} \cdot \mathbf{b} < 0$.

$\mathbf{a} \cdot \mathbf{b} = \begin{pmatrix} -3 \\ 3 \\ -1 \end{pmatrix} \cdot \begin{pmatrix} 1 \\ h \\ 0 \end{pmatrix} = -3(1) + 3(h) + (-1)(0)$
$\mathbf{a} \cdot \mathbf{b} = 3h - 3$

$3h - 3 < 0 \implies h < 1$

4. Determine the correct exact value of $h$ that satisfies the inequality.

Since $3 + 2\sqrt{2} \approx 5.83 > 1$ and $3 - 2\sqrt{2} \approx 0.17 < 1$, the only valid solution is the latter.

$h = 3 - 2\sqrt{2}$