fix: transformers v5 compatibility for AUTOMODEL_CAUSALLM VLMs (#3276)

fix: transformers v5 compatibility for AUTOMODEL_CAUSALLM VLMs (#3273)

In transformers v5, AutoProcessor.from_pretrained returns a
TokenizersBackend directly for pure-tokenizer processors (e.g.
Falcon-OCR with AUTOMODEL_CAUSALLM), which has no .tokenizer
attribute. Resolve the tokenizer via a helper that falls back to
the processor itself so both v4 wrapper processors and v5
TokenizersBackend shapes are supported.

Signed-off-by: Georg Heiler <georg.kf.heiler@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>