Many contributions to compiler-builtins don't have any need to touch
libm, and could get by with the few minutes of CI for compiler-builtins
rather than the ~30 minutes for libm. We already have some scripts that
handle changed file detection, so expand its use to skip libm CI if it
doesn't need to run.