feat(dataset): add `collect_all_keys` option and keep CSV export columns aligned (#2079)
- Fixes CSV export writing values under the wrong column. Rows were written positionally with `csv.writer`, so any item whose key set or key order differed from the first item's had its values shifted or placed under the wrong header. Rows are now written by key with `csv.DictWriter`. - Columns are taken from the first non-empty item. Keys that appear only in later items are dropped instead of being appended out of position, and an item missing a column now gets an empty cell instead of a short row. Exports of datasets with mixed keys therefore change content. - Dropped keys are reported once per export in a warning that names them. Crawlee for JS drops them silently, so this is a Python-only addition. - Adds a `collect_all_keys` parameter to `Dataset.export_to` and `BasicCrawler.export_data`. When True, the columns are the union of all item keys in first-seen order, so nothing is dropped. It reads the whole dataset before writing the header, unlike the default. Named and behaving like `collectAllKeys` in Crawlee for JS. - Adds `restval`, the value written for columns an item has no key for. It was rejected before, because `csv.writer` has no such argument. - Documents `ExportDataCsvKwargs` as mirroring `csv.DictWriter` and `ExportDataJsonKwargs` as mirroring `json.dump`, so Crawlee-specific options stay out of them. - Documents the column behavior in the dataset export example. - Alignment with Crawlee for JS: the by-key row mapping, deriving columns from the first item, and `collectAllKeys` all match `Dataset.exportTo` in `packages/core/src/storages/dataset.ts`. *✍️ Drafted by Claude Code*
A
Anas Khan committed
cd1e1f632cab692a54ba8db63e9f64ce3130a6d6
Parent: 3468eee
Committed by GitHub <noreply@github.com>
on 7/28/2026, 12:16:31 PM