Compiling mitlm 0.4 on Ubuntu 9.10 about mitlm HOT 25 CLOSED

Also, when you get errors about missing string functions, add "#include 
<cstring>"
and for missing limits, add "#include <limits>"

Is it available as a patch file ?

The patch file is attached to this post. Just put it into your mitlm.0.4 
directory
and apply the patch:

patch -p1 -i mitlm.0.4.ubuntu9.10.patch

make clean
make

All these issues should be fixed in revision 45.
In revision 45 autotools are used, so you need to install autoconf, automake 
and libtool to build mitlm.
You can build mitlm using the following commands:
./autogen.sh --prefix=$(pwd)/usr
make
make install

On ubuntu you should also be able to create a debian package of mitlm with the 
following command line (you need to run autogen.sh before using it):
dpkg-buildpackage -rfakeroot

Can you please provide feedback on these commands?

Hi, I downloaded the latest svn revision 45. There is still the problem with 
f77. Compilation stops with this message:

libtool: link: g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  
./.libs/libmitlm.so
./.libs/libmitlm.so: undefined reference to `etime_'
./.libs/libmitlm.so: undefined reference to `s_wsfe'
./.libs/libmitlm.so: undefined reference to `s_cmp'
./.libs/libmitlm.so: undefined reference to `e_wsfe'
./.libs/libmitlm.so: undefined reference to `do_lio'
./.libs/libmitlm.so: undefined reference to `s_wsle'
./.libs/libmitlm.so: undefined reference to `s_copy'
./.libs/libmitlm.so: undefined reference to `do_fio'
./.libs/libmitlm.so: undefined reference to `f_open'
./.libs/libmitlm.so: undefined reference to `e_wsle'
collect2: ld returned 1 exit status
make[1]: *** [evaluate-ngram] Error 1
make[1]: Leaving directory `/home/miso/install/mitlm-svn'
make: *** [all-recursive] Error 1

...which complains about a missing library 'f2c'. When I run this:
g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  ./.libs/libmitlm.so 
-lf2c

another error occurs:
/usr/lib/gcc/i486-linux-gnu/4.4.3/../../../../lib/libf2c.so: undefined 
reference to `MAIN__'
collect2: ld returned 1 exit status

...so I found a workaround here: 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337460
...then this command works fine:
g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  ./.libs/libmitlm.so 
-lf2c -u MAIN__

For the autotools, I put these 3 lines into Makefile.am:
evaluate_ngram_LDFLAGS = -lf2c -u MAIN__
estimate_ngram_LDFLAGS = -lf2c -u MAIN__
interpolate_ngram_LDFLAGS = -lf2c -u MAIN__

then ./autogen.sh and make work fine. I am not sure, whether this is the 
correct solution for all systems, but it works on Ubuntu 10.04.

Michal

Hi Michal,
  I'm trying to replicate the problem on my pc, without success. (I've tried gcc 4.3.5 and 4.4.5, using gfortran)

Obviously the line "libtool: link: g++ -g -O2 -o .libs/evaluate-ngram 
src/evaluate-ngram.o  ./.libs/libmitlm.so" is wrong, because there should be 
some additional token there.

Which fortran compiler are you using? What's the output of the configure script 
in the failing case?

Giulio.

Hi Giulio,

sorry for the incompleteness of the previous message. The output of 
./autogen.sh is attached as configure.out, then there is config.log with config 
variables and make.out with the full output of make.

I use Ubuntu 10.04 with gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3

As for the f77 compiler, it points to fort77, which is in version 1.15-8

Version of f2c is 20090411-1 and libf2c2 is 20090411-2

You can see in the config.log, that FLIBS variable is empty. Ideally there 
should be the '-lf2c'. The MAIN__ function can be integrated into the fortran 
code directly.

Michal

Thank you for all the details.
I was able to replicate the problem and investigate its causes.
There were two problems:
1) I did not include a test to detect the need of a MAIN__ function.
2) The current autoconf macro to detect F77 compilers is unable to properly 
detect fort77. I already contacted autoconf developers about this issue. I hope 
they will fix it in future autoconf versions.

With the latest subversion revision (47) you should be able to build mitlm 
using the following commands:
1) If f77 is fort77 (the f2c wrapper) and you want to use it
FLIBS="-lf2c -lm" ./autogen.sh --prefix=$(pwd)/usr
make
make install
2) If you want to specify another compiler that is not the default one (e.g., 
on Debian when you have both fort77 and gfortran, and you want to use gfortran) 
you can also use these commands:
F77=gfortran ./autogen.sh --prefix=$(pwd)/usr
make
make install

The issue cannot be solved in mitlm.
It can be fixed by manually patching the configure script before releasing next 
mitlm version.

You're right, Giulio. Both methods you described, work with the revision 47 on 
my system. Thanks for your work.
Michal

Thank you for your confirmation.

I think the issue can be closed now, because it cannot be solved in mitlm.
We can add a note about this issue in the README file (suggesting the above 
command lines and pointing to the INSTALL file for further information about 
configure script usage).

Yes, you can close this issue as SOLVED. That should be enough to put a note 
about this issue in the README file.
Michal

Hello Michal,

After applying the patch, I still have Fortran link error:

g++ src/estimate-ngram.o -o estimate-ngram -L. -lgfortran -lmitlm  -O3 
-funroll-loops
./libmitlm.a(lbfgsb.o): In function `prn3lb_':
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2788: undefined 
reference to `_gfortran_compare_string'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined 
reference to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined 
reference to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'

It looks like the gfortran library, located in

  /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so

didn't get linked in. The Makefile was patched successfully:

INC      = -Isrc
CXXFLAGS = -g -Wall -fPIC -fmessage-length=0 $(INC)
#LDFLAGS  = -L. -lg2c -lmitlm
LDFLAGS  = -L. -lgfortran -lmitlm
FFLAGS   = -g -fPIC -fmessage-length=0
FC       = gfortran

Somehow g++ linker wasn't able to located the lib ?

Chan

Hi Chan,

I assume you compile it with gcc version 4.6, right? You could try modifying 
LDFLAGS in the Makefile to:

LDFLAGS  = -L. -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -lgfortran -lmitlm

or try directly this:

LDFLAGS  = -L. -l/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so -lmitlm

if that does not work, this command should tell you whether the correct symbols 
are included in the libgfortran.so:

nm -D /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so | grep 
_gfortran_compare_string

...it should give you something like:

000d2b90 T _gfortran_compare_string
000d3320 T _gfortran_compare_string_char4

Let me know if it still doesn't work,
Michal

Hi Chan,
did you also tried the solutions proposed at Comment 8? In your case the second 
solution should work.

Anyway on my system (Debian wheezy amd64) I need "-lgfortran -lm -lquadmath" to 
link against gfortran.

Bests,
Giulio.

Hi Michal & Giulio,

I've tried all of your linking options as suggested, but no luck.
It's quite certain that the linker has its access to the lib of
libgfortran.so, and the underscroed symbol ids are matching each
other (one from the linker complaint and the other from the library).
But then the linker still cannot fins the symbols:

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ nm -D 
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so | grep _gfortran_compare_string
00000000000e8000 T _gfortran_compare_string
00000000000e87e0 T _gfortran_compare_string_char4

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ g++ src/estimate-ngram.o -o 
estimate-ngram -v -L. -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -lgfortran -lmitlm 
-lquadmath |& more
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubu
ntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages
=c,c++,fortran,objc,obj-c++,go --prefix=/usr --program-suffix=-4.6 --enable-shar
ed --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-i
ncluded-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4
.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable
-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --
disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release
 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu
/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr
/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu/
4.6.1/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../..
/lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/..
/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-o' 'estimate-ngram' '-v' '-L.' '-L/usr/lib/gcc/x86_64-linu
x-gnu/4.6' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.6.1/collect2 --build-id --no-add-needed --as-ne
eded --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-lin
ux-x86-64.so.2 -z relro -o estimate-ngram /usr/lib/gcc/x86_64-linux-gnu/4.6.1/..
/../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_
64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtbegin.o -L. -L/usr/li
b/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1 -L/usr/lib/gcc/
x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu
/4.6.1/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-l
inux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../.. src/es
timate-ngram.o -lgfortran -lmitlm -lquadmath -lstdc++ -lm -lgcc_s -lgcc -lc -lgc
c_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtend.o /usr/lib/gcc/x86_64-linux
-gnu/4.6.1/../../../x86_64-linux-gnu/crtn.o
./libmitlm.a(lbfgsb.o): In function `prn3lb_':
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2788: undefined referen
ce to `_gfortran_compare_string'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined referen
ce to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined referen
ce to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined referen
ce to `_gfortran_st_write'

These 'write' fortran function calls (from lbfgsb.f) were converted to different
write library functions (depending on the args types) with underscored names,
which are matched each other.

Trying to figure out why the linker cannot resolve these symbols.

Thanks for your help,

Chan

This is quite strange. My system is very similar to your (I have gcc-4.6.2, 
compiled with very similar flags, for the same architecture), but I am not able 
to replicate the problem on my system.

Just to be sure:
1) Can you verify that you have also the libgfortran.a file? (find /usr/lib/ 
-name "*gfortran*") On my system it is in the gfortran-4.6 package.
2) Have you tried using the mitlm version on the SVN repository, using the 
autoconf build system?

Hi Giulio,

Yes, taking MITLM from the svn repository did the trick.
SVN by:

  svn checkout http://mitlm.googlecode.com/svn/trunk/ mitlm

Then I did 'autogen.sh' to auto configure:

  ./autogen.sh

Then a make built all of the LM tools:

  make

Thanks for your help !

Chan

Anyway I still do not know why it did not work with the Michal's patch...

Apart from using the SVN source, did you change anything else on your system?

(e.g., what was the answer to my first question at comment 16?)

Nothing changed other than downloading new source from the svn repository.
All of the gfortran libs were existing:

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ find /usr/lib -name "*gfortran*" -print
/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0
/usr/lib/x86_64-linux-gnu/libgfortran.so.3
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.spec
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.a
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortranbegin.a

Ok, thank you for the feedback. If you will ever know why it failed with the 
simple Makefile method, please let me know.

Bests,
Giulio.

I followed the steps outlined in comment 17, but  when I run './autogen.sh'
I get the following error

http://pastebin.com/94tds5a2

The last line is the error line:
config.status: error: cannot find input file: `Makefile.in'

I am running ubuntu 11.10

[deleted comment]

 I had the same error message. Installed libtool and it worked.

Yes, libtool automake autoconf are all required to create the files for 
building.
I would also add that autoconf >= 2.69 is also required for proper building 
(fortran detection has been fixed there).