MinGW and libxlsxwriter produce invalid xlsx files on Windows Vista and above about libxlsxwriter HOT 15 CLOSED

Hi Evgeny,

Nice to hear from you again. Have you given up on Python. ;-)

The tests are all passing on Linux with gcc and clang (https://travis-ci.org/jmcnamara/libxlsxwriter) and also on MacOS.

Q1. I haven't used MinGW. Perhaps you can detail some steps to set it up and how to reproduce the issue?

The functional test are all failing with variants of "The system cannot find the path specified." so maybe the test runner isn't working in a MinGW/Windows environment.

Q2. Do the unit tests pass (make test_unit)?

Q3. Do the examples build (make examples)? If so do you get an xlsx file when your execute one of them: ./examples/hello?

John

from libxlsxwriter.

Hi John,

I didn't expect that you remember me. I feel flattered. Thanks.
Python - forever! ;-). But now I am trying to take up the Go language and have plans to make simple bindings for generating xlsx files from Go.

Q1:

I just got MSYS2 from http://sourceforge.net/projects/msys2/, updated your Makefiles to disable building of.so files, run make and that's it.

Q2:

Almost all the tests failed because of empty value in the got variable. For example, output of test_xmlwriter:

...
TEST 13/13 xmlwriter:xml_data_element_with_escapes ←[0;31m[FAIL]←[0m
←[0;31m  ERR: test_xmlwriter.c:226
        expected '<foo span="8">&amp;&lt;&gt;"</foo>',
        got      ''←[0m
←[0;31m
RESULTS: 13 tests (0 ok, 13 failed, 0 skipped) ran in 93 ms←[0m

It looks like that the issue caused by tmpfile(), as when I open a file manually and pass it to _xml_declaration() and others - all work fine. I suspect that the system cannot open a file for reading while it is opened for writing or perhaps there is needed fflush() call at the end.... In progress.

Q3:

All the files produced by the example applications are broken.The generated xlsx files contain empty xml files inside.

from libxlsxwriter.

It looks like that the issue caused by tmpfile(), as when I open a file manually and pass it to _xml_declaration() and others - all work fine. I suspect that the system cannot open a file for reading while it is opened for writing.... In progress.

That sounds like the issue. The code relies heavily on tmpfile().

I don't think I'll be able to help you on this since I've never used Msys2. If you find a fix let me know.

from libxlsxwriter.

#define RUN_XLSX_STREQ(exp, got)                                    \
    fflush(testfile);                                               \
    int file_size = ftell(testfile);    
    ....                            \

file_size is always equal to -1.. moreover tmpfile() returns NULL! I should have checked it at first, sorry.

I don't think I'll be able to help you on this since I've never used Msys2. If you find a fix let me know.

Yes, I understand. I will try to resolve it by myself. Thank you for support.

from libxlsxwriter.

Some news:
According to https://msdn.microsoft.com/en-us/library/x8x7sakw.aspx tmpfile() on Windows (al least its Microsoft's implementation) creates temporary files in the root directory and therefore needs admin privileges...

The tmpfile function creates a temporary file and returns a pointer to that stream. The temporary file is created in the root directory. To create a temporary file in a directory other than the root, use tmpnam or tempnam in conjunction with fopen.

All the tests passed when I run them as administrator...

from libxlsxwriter.

on Windows (al least its Microsoft's implementation) creates temporary files in the root directory and therefore needs admin privileges...

Aargh! I should have thought of that first thing when you mentioned temp files and Windows.

That is the reason the Python and Perl modules have a tmpdir option to avoid that very problem:

from libxlsxwriter.

Can you try settin the TMPDIR, TEMP, or TMP environmental variable to control where the tmp dir is?

from libxlsxwriter.

Can you try settin the TMPDIR, TEMP, or TMP environmental variable to control where the tmp dir is?

I've tried this:

set TMPDIR=c:\temp
set TEMP=c:\temp
set TMP=c:\temp
test_xmlwriter.exe >test.log 2>&1

..and for c:/temp as well and have got the same errors.

Now I'm looking into the sources of tmpfile() and _tmpnam_r(ptr, s) and don't see that they use any environment variables
https://github.com/msysgit/msys/blob/master/newlib/libc/stdio/tmpfile.c (not checked yet, but hope, that msys2 contains similar implementation)
https://github.com/msysgit/msys/blob/master/newlib/libc/stdio/tmpnam.c

The only function which reads $TMPDIR is _tempnam_r (ptr, dir, pfx), but the function is not called by tmpfile().

Btw, the constant _P_tmpdir used as a fallback has the following value:
stdio.h:

#ifdef _POSIX_
#define _P_tmpdir "/"
#define _wP_tmpdir L"/"
#else
#define _P_tmpdir "\\"
#define _wP_tmpdir L"\\"
#endif

from libxlsxwriter.

I'm only guessing because I don't know how the msys2 "shell" works but maybe the TMPDIR needs to be set in the Windows environment and not in the shell. What does the following print with and without the set TMPDOR:

perl -MFile::Spec -le "print File::Spec->tmpdir()"

Or perhaps it doesn't matter and, as you say, it doesn't have an effect on tmpfile() either way.

It seems odd that there isn't some way to workaround this in msys2. It seems like this would affect any application that calls tmpfile().

from libxlsxwriter.

maybe the TMPDIR needs to be set in the Windows environment and not in the shell.

I've tried it from the both places

What does the following print with and without the set TMPDOR:

Perl from msys prints /tmp

It seems odd that there isn't some way to workaround this in msys2. It seems like this would affect any application that calls tmpfile().

Anyway, microsoft documentation (link above) says that it needs adminstrator rights:

Note

This example requires administrative privileges to run on Windows Vista

So, at least visual c++ compiler is out of the party despite pure ansi c.

Some libraries bypass it with their own implementation of tmpfile:
http://cgit.freedesktop.org/cairo/commit/?id=4fa46e3caaffb54f4419887418d8d0ea39816092
(not sure if it's still there though)
I've got there via this discussion: http://sourceforge.net/p/gnuwin32/discussion/74807/thread/903411c4/

from libxlsxwriter.

Perl from msys prints /tmp

That is interesting. If the a C program sees the same directory (and it probably does) then maybe you can map it to a writeable directory using the msys2 fstab file. But maybe not based on your links.

Anyway, microsoft documentation (link above) says that it needs adminstrator rights:

I don't disagree. And if that is sufficient for your needs then that is okay.

Some libraries bypass it with their own implementation of tmpfile:

That's interesting. Thanks for the update.

If there is no further updates in a few days I will close this issue (or you can close it) as I don't want to have to change the tmpfile() implementation. I'll suggest the Administrator workaround for anyone who needs it.

from libxlsxwriter.

And if that is sufficient for your needs then that is okay.

No, that's not sufficient for me at all. That was just a temporary workaround to make sure that tmpfile() is the culprit. On Linux nobody wants to run ordinary tools as root, don't they? I'll try to find a more descent solution...

from libxlsxwriter.

Try something like the following:

diff --git a/src/Makefile b/src/Makefile
index 5be9364..18fd45a 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -19,7 +19,7 @@ INC_DIR     = ../include
 MINIZIP_DIR = ../third_party/minizip

 # Flags passed to compiler.
-CFLAGS   += -g -O3 -Wall -Wextra -pedantic -ansi
+CFLAGS   += -g -O3 -Wall -Wextra -pedantic

 # Library names.
 LIBXLSXWRITER_A  = libxlsxwriter.a
diff --git a/src/utility.c b/src/utility.c
index 2fd629b..29a244a 100644
--- a/src/utility.c
+++ b/src/utility.c
@@ -12,6 +12,7 @@
 #include <string.h>
 #include <stdint.h>
 #include <stdlib.h>
+#include <unistd.h>
 #include "xlsxwriter/utility.h"

 /*
@@ -411,5 +412,15 @@ lxw_quote_sheetname(char *str)
 FILE *
 lxw_tmpfile(void)
 {
-    return tmpfile();
+    char template[] = "/tmp/libxlsxwriter.XXXXXX";
+    FILE *tempfile;
+    int fd;
+
+    fd = mkstemp(template);
+    tempfile = fdopen(fd, "w+");
+    unlink(template);
+
+    return tempfile;
+
+    /* return tmpfile(); */
 }

It is available on the mkstemp branch. Change the template to suit your system.

It isn't ansi compatible however.

from libxlsxwriter.

Interesting! At first I was skeptical as it seemed (and actually "is" on Windows) useless to unlink/remove open files on Windows - the system doesn't let to do it. But mkstemp works well by itself - the temporary files get deleted just after they are closed. Thank you!

from libxlsxwriter.

Closing. Use workaround if required.

from libxlsxwriter.