regex: reject invalid \digit back reference in BRE

in BRE \n matches the nth subexpression, but regcomp did not check if
the nth subexpression was complete or not, only that there were more
subexpressions overall than the largest backref.

fix regcomp to error if the referenced subexpression is incomplete.
the bug could cause an infinite loop in regexec:

 regcomp(&re, "\\(^a*\\1\\)*", 0);
 regexec(&re, "aa", 0, 0, 0);

since BRE has backreferences, any application accepting a BRE from
untrusted sources is already vulnerable to an attacker-controlled
near-infinite (exponential-time) loop, but this particular case where
the loop is actually infinite can and should be avoided.

ERE is not affected since the language an ERE describes is actually
regular.

Reported-by: Simon Resch <simon.resch@code-intelligence.com>