fix(retrieval): export Sentence Transformers metadata (#3021)

* fix(checkpoint): shard extracted exports by size

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* feat(retrieval): define sentence transformer export contract

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(checkpoint): generate sentence transformer metadata

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(checkpoint): enforce sentence transformer export contracts

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(checkpoint): respect local model subfolders

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* test(retrieval): specify sentence transformer interoperability

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): restore sentence transformer wrapper behavior

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(checkpoint): enforce sentence transformer export semantics

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* Update uv lock

Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>

* fix(retrieval): enforce sentence transformer round trips

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(checkpoint): preserve root legal assets

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): preserve Sentence Transformers semantics

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* chore(retrieval): remove unrelated branch churn

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): derive Sentence Transformers metadata

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* docs(retrieval): describe restored embedding metadata

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* chore(retrieval): remove Allium export specification

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): own Sentence Transformers export metadata

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): preserve Hub legal assets on export

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(checkpoint): consolidate model unwrapping

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): scope model unwrapping to DDP

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): narrow sentence transformer checkpoint integration

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): isolate sentence transformer metadata

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): remove config kwargs allowlist

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* refactor(retrieval): narrow sentence transformer export scope

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* test(retrieval): verify consolidated sentence transformer export

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): keep sentence transformer export additive

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): canonicalize mean pooling alias

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* docs(retrieval): limit export behavior guidance

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* fix(retrieval): preserve exported tokenizer pad token

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

* docs(retrieval): update Ministral3 routing guidance

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

---------

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Co-authored-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com>
Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>