The raw step of the pipeline includes each corpus twice in the categories.json file

[bhaddow]

We are seeing that the generated script for the raw file includes each dataset twice. For example, see /pfs/lustrep3/scratch/project_465000574/malikbha/experiments/en-hr/simple/v0/s.raw.en-hr/step.command

The dataset is included only once in the original categories.json. However as far as I can see, when the pipeline generates the raw step, it does not see the original categories.json, but generates its own. This records each file twice, because of faulty logic in

OpusPocus/opuspocus/pipeline_steps/raw.py

Line 59 in 6dfd3ac

logger.info(

[bhaddow]

The reason that the categories.json was being ignored is that @bhavitvyamalik temporarily by-passed it. But still, the pipeline should not generate two entries in this case.

[bhavitvyamalik]

I have fixed that in my new commit. I have pushed changes to lumi-tmp branch since @varisd's latest changes weren't working for me. It was more of a hack for time being, but now we can fix this problem i.e., to skip processing empty datasets (#12)

[rggdmonk]

Resolved.