Slow tasks

[guiscaranse]

Please answer these questions before submitting your issue. Thanks!

Gorse version
0.4.5

Describe the bug
My tasks have been taking several days to complete (sometimes never completing), I am wondering if this is normal or I am doing something wrong.

Additional context

• Items: 724173
• Number of User Labels: 2570647
• Number of Item Labels: 2571813
• Master RAM: 6gb
• Master CPU Cores: 2 CPU cores
• Number of tasks on master (config.yml): 8

[zhenghaoz]

There are two problems:

1. Gorse can't run multiple tasks concurrently. A slow task will block following tasks. This problem will be addressed in the next version.
2. It seems that you have too much labels. Are they really needed? And how many labels each user or item has? I think you should review your dataset.

[guiscaranse]

It seems that you have too much labels. Are they really needed? And how many labels each user or item has? I think you should review your dataset.

@zhenghaoz an item has around 40 tags, is that a large amount? What would be a good amount of tags from a performance standpoint?

[zhenghaoz]

It seems that you have too much labels. Are they really needed? And how many labels each user or item has? I think you should review your dataset.

@zhenghaoz an item has around 40 tags, is that a large amount? What would be a good amount of tags from a performance standpoint?

Are these tags shared by multiple items?

[guiscaranse]

They are, I've started sampling the tags and reduced them by 95% and now the training is pretty fast (went from 5 days to 1 hour). Closing this issue :)

[sethmills21]

@guiscaranse how long would it take to load your dataset? i'm still fighting loading a lot of data over here too