feat(hackernews): Add ability to start from given date (#6560)
This adds the ability to specify a start date to start the sync from. Hacker News data goes back many years, and many use cases will not require syncing it all (which can take a day or two). With this setting, a date like Jan 1st 2022 can be given to sync only the last year of data, which can be done in around two-three hours from a modest laptop on a home internet connection. In terms of implementation, we binary search for the first post after the given date, and then start the cursor from there. This also fixes the timestamp column type, which accidentally got reverted during the transformer transition.
H
Herman Schaaf committed
977fa1693f07040daed9a77ed437bbced4f0b69c
Parent: 75de638
Committed by GitHub <noreply@github.com>
on 1/10/2023, 3:53:49 PM