Comments (6)
So it wasn't quite as trivial as hoped, because mmap wants a file descriptor
(int) instead of a FILE*. Using fopen() and fclose() is the portable way to do
things, but under the covers they do a bunch of buffering and such that we
don't really need for our mmap call. On the bright side it seems like using
fileno() from stdio.h allows us to get the file descriptor, and I can't imagine
that we will have any problems with mmap'ing the region read-only so long as we
are also fopen()'ing in "r" (read-only) mode, even if it mmaps the same region
under the covers. So hopefully all is well.
All unit tests pass in both dynamic and non-dynamic modes with the attached
patch, so I think everything should be alright.
Original comment by [email protected]
on 13 Aug 2014 at 10:02
Attachments:
from cld2.
I had a pro/con discussion with another engineer on this, and here is the
summary: the patch will fix the problem, but is a little sketchy because of the
mixing of fopen() with mmap(). It would be better to #include unistd.h, and
continue using open() instead of fopen(); but this is specific to Unix and
wouldn't work on Windows. That said, the code is *already* broken on Windows
because of the use of sys/mman.h (for mmap).
So: If we use unistd.h, we deepen the problems for Windows. If we use fopen(),
we do something questionable for non-Windows (it works, but it isn't really the
right thing to do).
I think the answer here is to use unistd.h for linux and use a workaround like
this to fix win32:
// Based on definitions from
http://sourceforge.net/p/predef/wiki/OperatingSystems/:
#ifdef _WIN32
#include <io.h>
#define OPEN _open
#define CLOSE _close
#else
#include "unistd.h"
#define OPEN open
#define CLOSE close
#endif
We can extend this solution to fix the mmap problem as well, in the near future.
Original comment by [email protected]
on 13 Aug 2014 at 11:12
from cld2.
I've created issue 20 to track the overall Windows compatibility problem for
the dynamic data loader.
Original comment by [email protected]
on 13 Aug 2014 at 11:40
from cld2.
Here is a new patch based on #2 above. As described in issue 20, it disables
support for win32 in the "file"-based dynamic data apis; only raw pointer mode
will work on win32 until we have proper mmap support there.
So to be clear, we didn't end up changing the open() to fopen(), because open()
makes sense for this use case. Updating the bug description accordingly.
Original comment by [email protected]
on 13 Aug 2014 at 12:31
- Changed title: Windows build fails: undeclared identifier 'close'
Attachments:
from cld2.
Should be fixed in r166.
Original comment by [email protected]
on 13 Aug 2014 at 12:35
- Changed state: Fixed
from cld2.
Issue 23 has been merged into this issue.
Original comment by [email protected]
on 29 Aug 2014 at 11:33
from cld2.
Related Issues (20)
- SIGBUS on ARM32 in utf8statetable.cc:517 HOT 13
- compact_lang_det.h: loadDataFromRawAddress should use types from stdint.h instead of "int" HOT 3
- Can't link "dynamic" and "full" HOT 3
- CLD2DynamicDataLoader calls delete instead of delete[] on array types HOT 4
- The ISO 639 code for Hebrew is outdated HOT 1
- Consider declaring dynamic data methods unconditionally HOT 3
- CLD2 result chunk vector omits portions of input file HOT 6
- Dynamic data loading should not use iostream HOT 5
- Support mmap-ing dynamic data on win32 HOT 5
- Build warning on Windows with clang HOT 2
- Eliminate redundancy and/or simplify default case for compiling unittest_data.h HOT 4
- Missing include in cld2_dynamic_data_loader.cc HOT 1
- cld2_dynamic_data.cc and cld2_dynamic_data_loader.cc problems on Win32 HOT 10
- Enable dynamic data for 20141015 release HOT 1
- New GCC 5.0 hits problem with narrowing in list-initializers
- CLD should check result of "new" in all use cases
- please use CFLAGS CXXFLAGS CPPFLAGS and LDFLAGS HOT 3
- please provide a SONAME HOT 8
- cld2 testsuite failures HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cld2.