Handle the case of reparenting while updating children

Instead of trying to flush the detached children from the child list, we keep
the set of detached children up to date and query on every read.