I am trying to run LDA in VW using Rosetta. It seems to be working fine for smaller number of topics but as soon as I go to 50 or 100, step 3: read the results with LDAResults fails => I get a "Killed" error. I don't think this is a memory problem because I am running my code on a robust machine with 50GB of RAM. What's going on? Is this a VW or Rosetta issue? How can I solve it? Thanks!
Once I have doc_tokens.vw, this is what I am running in order:
Step 1: `from rosetta.text.text_processors import SFileFilter, VWFormatter
sff = SFileFilter(VWFormatter())
sff.load_sfile('doc_tokens.vw')
df = sff.to_frame()
df.head()
df.describe()
sff.filter_extremes(doc_freq_min=500, doc_fraction_max=0.8)
sff.compactify()
sff.save('sff_file.pkl')`
Step 2: rm -f *cache vw --lda 100 --lda_alpha 0.1 --lda_rho 0.1 --cache_file ddrs.cache --passes 10 -p prediction.dat --readable_model topics.dat --bit_precision 16 doc_tokens_filtered.vw
Step 3: from rosetta.text.vw_helpers import LDAResults num_topics = 5 lda = LDAResults('topics.dat', 'prediction.dat', 'sff_file.pkl', num_topics=num_topics) lda.print_topics()
I am trying to run LDA in VW using Rosetta. It seems to be working fine for smaller number of topics but as soon as I go to 50 or 100, step 3: read the results with LDAResults fails => I get a "Killed" error. I don't think this is a memory problem because I am running my code on a robust machine with 50GB of RAM. What's going on? Is this a VW or Rosetta issue? How can I solve it? Thanks!
Once I have doc_tokens.vw, this is what I am running in order:
Step 1: `from rosetta.text.text_processors import SFileFilter, VWFormatter
sff = SFileFilter(VWFormatter())
sff.load_sfile('doc_tokens.vw')
df = sff.to_frame()
df.head()
df.describe()
sff.filter_extremes(doc_freq_min=500, doc_fraction_max=0.8)
sff.compactify()
sff.save('sff_file.pkl')`
Step 2:
rm -f *cache vw --lda 100 --lda_alpha 0.1 --lda_rho 0.1 --cache_file ddrs.cache --passes 10 -p prediction.dat --readable_model topics.dat --bit_precision 16 doc_tokens_filtered.vwStep 3:
from rosetta.text.vw_helpers import LDAResults num_topics = 5 lda = LDAResults('topics.dat', 'prediction.dat', 'sff_file.pkl', num_topics=num_topics) lda.print_topics()