generate_vocab keeps on running for a few HPLT datasets

[bhavitvyamalik]

The generate_vocab step keeps on running for some datasets. Interestingly, this happens for HPLT datasets mostly as we sub-sample 10M sentences for larger datasets to generate vocab. However for HPLT we don't have that many sentences and we end up using all sentences for generating vocabulary.

This is a dataset related issue but I feel our pipeline should be robust enough to handle such problems.