I’m sure there’s some short elegant solution here that uses beautiful vector math. Instead I went for the butt ugly coordinate geometry solution.

Let u = <a, b> and v = <c, d>

See diagram. WLOG, assume u has a smaller angle than v. We can find cos(θ) by constructing a right triangle as shown, and by finding a new vector w, in the same direction as u, which has the correct length to complete our right triangle. Once done, we will have cos(θ) = |w| / |v|.

Let’s consider each vector to be anchored at the origin. So we can say u lies on the line y = (b/a)x, and v lies on the line y = (d/c)x. To find w, let us first find z - the third side of the triangle, which we know must be perpendicular to u, and pass through (c, d).

The line perpendicular to y = (b/a)x and passing through (c, d) is y = (-a/b)(x-c) + d. So this is the line containing the third side of our constructed triangle, z. To find w, then, let’s find its point of intersection with y = (b/a)x, the line containing w.

(b/a)x = (-a/b)(x-c) + d → Setup

(b/a)x = (-a/b)x + (ca/b) + d → Distribute on right

x(b/a + a/b) = (ca/b) + d → Add (a/b)x to both sides, factor out x on left

x(a² + b²) = ca² + dab → Multiply both sides by ab

x = (ca² + dab) / (a² + b²) → Divide both sides by (a² + b²)

x = (ca² + dab) / |u|² → a² + b² is |u|²

y = (b/a)x = (b/a)(ca² + dab) / |u|² → Plug solution for x into (b/a)x, don’t bother simplifying as this form is useful in a moment

So w = <(ca² + dab) / |u|², (b/a)(ca² + dab) / |u|²>. Now we need |w|.

|w| = sqrt((ca² + dab)² / |u|⁴ + (b²/a²)(ca² + dab)² / |u|⁴) → Plugging into normal |w| formula

|w| = sqrt(((ca² + dab)² / |u|⁴) * (1 + b² / a²)) → Factor out (ca² + dab)² / |u|⁴

|w| = (ca² + dab) / |u|² * sqrt(1 + b² / a²) → Pull (ca² + dab)² / |u|⁴ out of the root

|w| = (ca² + dab) / |u|² * sqrt((a² + b²) / a²) → Combine terms in root to one fraction

|w| = (ca² + dab) / |u|² * |u| / a → Evaluate sqrt

|w| = (ca + db) / |u| → Cancel out |u| and a from numerator and denominator

So this is the length of our adjacent side in the constructed right triangle. Finally, to find cos(θ), divide it by the length of the hypotenuse - which is |v|.

cos(θ) = |w| / |v| = (ac + bd) / (|u||v|)

And note that ac + bd is u•v. So we’re done.

cos(θ) = u•v / (|u||v|), or |u||v|cos(θ) = u•v.

Note that since this formula is symmetric with respect to u and v, our assumption that u’s angle was smaller than v’s angle did not matter - so this should hold regardless of which is larger.

By definition of |u| and Pythagoras, we can see that u1 and u2 can represent the lengths the legs of a right triangle with |u| being the length of the hypotenuse. Similarly for v. The triangles can be visualized in the usual way by plotting the vectors in the XY plane centered at the origin, with u1 and v1 constituting the X coordinates and u2 and v2 the Y coordinates of u and v, respectively.

Let α and β be the respective angles that u and v make with the X-axis. Then, θ = β - α, and

u1 = |u| cos α

u2 = |u| sin α

v1 = |v| cos β

v2 = |v| sin β

Now consider the expression:

|u| |v| cos θ

= |u| |v| cos (β - α)

= |u| |v| (cos β cos α + sin β sin α) (cosine angle difference formula)

= (|u| |v| cos β cos α) + (|u| |v| sin β sin α) (distribute)

= (|u| cos α)(|v| cos β) + (|u| sin α)(|v| sin β) (commutativity, associativity)

= (u1)(v1) + (u2)(v2) (substituting)

= u • v (definition)