There are various metrics around for choosing which edges to collapse (and in what order), and usually involve some approximation of the local curvature of the mesh around each edge. Examples range from simple dihedral angle checks, to the more advanced quadrics method. However, these are just a few simple examples of the many papers which have been published out there.

Blender’s decimate modifier for many years only used an edge-collapse technique, using a quadrics-based cost function.

EDIT:

An alternative to choosing edges to collapse is to find vertices to remove (i.e. vertex decimation).