Very large number of "Feature skipped" error messages about mitlm HOT 6 OPEN

Wasn't sure how to chage the Type/Priority.

Hi perpperbc,

I am having difficulty replicating the problem.  The feature skipped messages
indicate that there is an n-gram in the feature file that do not exist in the 
n-gram
model.  Currently, the only way I can think where this occurs is if the 
.effcounts
are computed for higher order n-grams (ex. 4) than the target n-gram model (ex. 
3). 
In this case, although you will get a lot of feature skipped warning messages, 
the
target LM should still be computed correctly.

Can you please confirm if this is the cause of the problem?  Regardless, I will
modify the warning messages to display more information and summarize the 
warnings
beyond 3 or so messages.

P.S. You can change the type and priority by clicking on the label itself.

That seems like it would have been the problem, as I had forgotten to to use 
the the
-o 6 flag, and all the component models were 6-gram. Unfortunately, I got a 
segfault
when I added the flag.

Interpolating component LMs...
Tying parameters across n-gram order...
Segmentation fault

After removing, which became 5 ~1GB), I still got a very large number of 
"Feature
skipped."  error messages. There were also some "Feature value
pow2:log1p:sumhist:literal-all.effcounts exceed 20.0" after each effcounts file 
was
said to be loaded. It then segfaulted again, after having only read 3 of 5 
feature
count files.

I am running it one more time with only 4 models, and it has been streaming 
"Feature
skipped" messages for about half an hour ( though it hasn't crashed). I will 
update
if anything interesting happens with it.

I also ran a CM build with only two models "interpolate-ngram -lm "1.lm, 2.lm"
-interpolation CM -op dev.txt -wl CM.lm  -interpolation-features
"log:sumhist:%s.corp.txt" -o 6", 
got 22 Feature skipped messages, and it completed.

Are there any key variables I could have printed at the same time as the 
"Feature
skipped." message in order to get more insight into what is happening?


On a side note, the "Feature skipped." messages seem to be printed directly to 
the
screen and not to stderr or stdout, as I cannot redirect them.

I noticed my count files I had also failed to make 6-gram, so I rebuilt these, 
but
the error messages persisted. I removed the error message from the code, 
rebuilt, and
restarted the 4-model build, and it completed with decent success after about 17
minutes (it seemed the error message was slowing things down a lot). I removed 
the
log^2 from the features however, because of the error messages, and it had not
completed after about 80 minutes. Still unsure as to why the error messages 
occur,
however.

One more comment. Even though I made it go through ignoring the errors, it 
seems they
still represent some problem, as my perplexities are significantly higher than 
LI for
CM (CM-52 vs LI-46, CM-37 vs LI-33), and marginally lower to marginally higher 
than
LI for GLI (GLI-48.5 vs LI-46.,3, GLI-32.3 vs LI 33.3) with log:sumhist.