32bit version is it possible? about zfsin HOT 49 OPEN

Binary release has been added

from zfsin.

Side note:
I have written the article about ZFSin at the popular Russian IT-portal in order to attract more attention to your project!
https://habr.com/post/374317/

from zfsin.

ReactOS can boot from BTRFS. https://reactos.org/blogs/gsoc-2018-booting-btrfs-works

Next turn -ZFS? $-)

from zfsin.

Hello, guys. I think that MSDN article could help us with that issue: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/platform-toolset
I think that we should try to compile the driver using the old DDK platform toolset (instead of Windows 10).
I'll check it some time later.

from zfsin.

@leha-bot:

I think that MSDN article could help us with that issue: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/platform-toolset

I tried reading up that a bit but fail to see any reference to an xp-era drivers (generation), only to Windows8.1:

PlatformToolset (WDK 8.1)	Use
WindowsKernelModeDriver8.1	For kernel-mode drivers and components.
WindowsUserModeDriver8.1	For user-mode drivers and components.
WindowsApplicationForDrivers8.1	For any type of application. This platform toolset provides compatibility with the build options used in the Windows Driver Kit (WDK) for Windows 7 (WDK 7.1), and also uses the default settings that are common for development of user-mode applications that interact with drivers. You might use this setting if you are migrating or converting a project that was built using WDK 7.
Visual Studio 2013 (v120)	Use for any type of Windows application (default).

Alex, did you find anything yet?

OTOH, what @binarymaster mentioned does actually lead to a doc about xp-era targets for VS2017:
https://docs.microsoft.com/en-us/cpp/build/configuring-programs-for-windows-xp?view=vs-2017

But it contains no driver (kernel mode) related topics, only a user-mode related ones.

Googling for v141_xp and DDK|WDK didn't reveal any useful points to me. So I decided to look up for the latest WDK version supporting NT5.x drivers. It's Windows Driver Kit 7.1.0 | 7600.16385.1.

Googling for it + some VS keywords revealed a pair of hints (albeit for VS2013).

From Build XP driver from Visual Studio 2013 / WDK 8.1:

Martin Dráb | 18 Jan 16 10:04

Hello, well, it is possible to use the WDK 8.1 toolset to build a driver that runs on Windows XP. However, there isno target for that (AFAIK), so you must be careful not to use any functionality introduced in later versions of Windows.

You probably need to do some changes in project settings (such as disabling security checks) to make the driver run on XP. Just build the driver and compare its imports with routines exported on XP. Using the Filesystem Minifilter project template proved to be a good option for me.

From Building “old” Windows drivers (XP, Vista, 2003 and 2008) with Visual Studio 2013:

3. Motivation

• Maintain single code base for variety of Windows OSes
• Build XP and Vista drivers with Visual Studio 2013
• Use C++11 features of Visual Studio 2013 compiler for the development of kernel drivers

4. What are the basic steps?

• Use VS2013 compiler
• Compile XP and Vista drivers using legacy DDK for include files and libraries
• Change appropriate preprocessor definitions
• Adjust linker options
• Adjust packaging for KMDF

5. Create configuration

• Create configuration for desired OS and platform by copying existing configuration (for example Windows 7)

6. Adjust includes and libraries paths

• Include directories (when using Win7 DDK):

C:\WinDDK\7600.16385.1\inc\ddk;
C:\WinDDK\7600.16385.1\inc\crt;
C:\WinDDK\7600.16385.1\inc\api

7. Adjust libraries path

• Change your libraries path according to the following table:

XP 32bit	C:\WinDDK\7600.16385.1\lib\wxp\i386
Windows Server 32 bit	C:\WinDDK\7600.16385.1\lib\wnet\i386
Windows Server 64 bit	C:\WinDDK\7600.16385.1\lib\wnet\amd64
Vista 32bit	C:\WinDDK\7600.16385.1\lib\wlh\i386
Vista 64bit	C:\WinDDK\7600.16385.1\lib\wlh\amd64

8. Adjust preprocessor definitions

• Use the following link to set correct versions according to the target OS for:

  • WINVER
  • NTDDI_VERSION
  • _WIN32_WINNT

• Example for Windows Vista:

  • _WIN32_WINNT=0x0600;WINVER=0x0600;NTDDI_VERSION=0x06000000

9. Adjust linker settings (1)

• Adjust “Additional Library Directories” to point to legacy DDK directory according to your platform (see also “Adjust libraries path” slide)

10. Adjust linker settings (2)

• Change “Additional dependencies” to reflect needed libraries
• The screenshot is an example of NDIS miniport driver

 NetKVM Property Pages:

  Configuration: Active(Vista Release)
  Platform: Active(Win32)

  (Page tree)
   ...
    Linker
     Input
      (Properties list)
      Additional Dependencies: ndis.lib;ntoskrnl.lib;BufferOverflowK.lib;hal.lib

• For WDM driver remove ndis.lib and add wdm.lib

11. Adjust linker settings (3)

• Change “version” and “subsystem” parameter to reflect correct Windows version.
• For the versions numbers check http://msdn.microsoft.com/en-us/library/windows/desktop/aa383745(v=vs...

12. Package project

• Adjust OS list for Inf2Cat
• Possible options:

  XP_X86,
  Server2003_X86,
  XP_X64,
  Server2003_X64,
  Server2003_IA64,
  Vista_X86,
  Server2008_X86,
  Vista_X64,
  Server2008_X64,
  Server2008_IA64

13. Check command lines of compiler and linker

• Check compiler (C\C++) and linker command line for references to new WDK (Windows Kits\8.1 or Windows Kits\8.0)
• Check that all the version numbers reflect correct Windows version
• Fix if needed according to previous slides

14. BUILD THE DRIVER

(I omitted the KMDF part since ReactOS seems to have no support for it)

Resume of my morning's quest:

• In VS2017 (I think) WDK 8.1 PlatformToolset gives a reference to Visual Studio 2013 (v120).
• And VS2013 can use WDK 7.1.0 (unofficially).

So maybe it's worth a try to run some experiment by installing WDK 8.1 + WDK 7.1.0 along VS2017 and then adjusting some configuration options as described above.

from zfsin.

10142ef
Enables me to at least produce the binaries in 32bit, but still fighting VS to deploy it to remote machine.

from zfsin.

Hmm so both 32bit and 64bit are set as Unicode character set, but I noticed that the 32bit zpool.exe will not open the files when I call CreateFile (like 64bit does). It will open it if I call CreateFileA(). Anyone know how this black magic works?

from zfsin.

@jeditobe : You open the ZFSin utility under Dependency Walker in ReactOS and you see at once all the apis that it uses and that are not in ReactOS (they show up in red).

from zfsin.

ReactOS has actually a static library for some of these ntoskrnl vista+ functions:
https://git.reactos.org/?p=reactos.git;a=tree;f=sdk/lib/drivers/ntoskrnl_vista;hb=bd2f3d39a92483679986107466636fe84e0420af
So far it covers: RtlUnicodeToUTF8N and RtlUTF8ToUnicodeN, KeQueryActiveProcessorCount, and two FsRtl functions. This lib is linked statically when we recompile imported 3rd-party drivers that need these.
For whatever reason Win2k3 (and thus, ReactOS) kernel doesn't export RtlDuplicateUnicodeString, however this function really exists (it's eg. exported by ntdll.dll) so we have an implementation for it.

Concerning the extra Cc and FsRtl functions one would have to keep in touch with ReactOS's filesystem guy.

from zfsin.

PS. Found some guy hacking MSBuild version of VS2017 to work with WDK8.1:
ProcessHacker can't build driver

No idea if it's worth mentioning.

from zfsin.

@jeditobe @lundman If I am reading this correctly, the original issue appears to be fixed. Maybe this should be closed in favor of a new issue for ReactOS (Windows XP?) compatibility.

from zfsin.

It could be possible, there was some 32bit code removed during porting, not much but some, but it could be restored. I have never tried it though

from zfsin.

@lundman we need 32-bit version to test it in ReactOS. 64bit version of ReactOS is still not ready.

from zfsin.

I'll see if I can setup a 32bit windows and make sure it runs.

from zfsin.

looking forward, to test it against a fresh build of ReactOS

from zfsin.

Ok just needed some deployment settings:

Windows 10 Kernel Version 17134 MP (2 procs) Free x86 compatible
Built by: 17134.1.x86fre.rs4_release.180410-1804
System Uptime: 0 days 0:16:21.864
SPL: start
SPL: total ncpu 2
spl_query_memsize: '.Translated' type 0x8 len 0x64
spl_query_memsize: memsize is 2146394112
SPL: memsize 2146394112 (before adjustment)
SPL: Total memory 1073197056
SPL: vmem_init: real_total_memory 2146394112, large spans 1048576, small spans 262144
SPL: vmem_init adding fixed allocation of 134217728 to the bucket_heap
SPL: starting vmem_update() thread
SPL: Warning: High-overhead kmem debugging features enabled (kmem_flags = 0x6).  Performance degradation and large memory overhead possible. See the Solaris Tunable Parameters Reference Manual.
SPL: spl_zio_no_grow_init done.
SPL: starting KMEM
SPL: beginning spl_free_thread() loop, spl_free == 18382848
SPL: Loaded module v1.6.0-7-gc1b4a00 (DEBUG mode), (ncpu 2, memsize 1073197056, pages 262011)
zvol_init
zfsdev_state: DEE62F80
SPL: Notice: failed to create fm/misc kstat
ZFS: created kernel device node: 8EB5D4B8: name DiskFileSystemDevice: 0x0 \ZFS created
ZFS: created userland device symlink
ZFS: Loaded module v1.6.0-1, ZFS pool version 5000, ZFS filesystem version 5
ZFS filesystem version: 5

from zfsin.

oh, cool news!

Could you upload 32-bit binary packages as well as you did with 64-bit binary packages?

from zfsin.

does it expect some VC++ runtime to be installed? (like, msvcr70.dll or msvcp70.dll or higher versions?)

from zfsin.

Not as far as I know, but I had to copy vcruntime140.dll with it.

from zfsin.

Strange.
Because in ReactOS we got this error

(dll/ntdll/ldr/ldrpe.c:1035) Failed to snap _except_handler4_common
(sdk/lib/rtl/heap.c:2199) HEAP: Trying to free an invalid address 7A07A540!

https://jira.reactos.org/secure/attachment/46555/46555_Capture.PNG

from zfsin.

Don't know how that works in Windows sorry, but I tested loading the 32bit release on my Win10 VM I setup. I fixed the INF file so the devcon.exe shouldn't be required now, but right-click Install should work.

from zfsin.

OK, but all other EXE files from the driver directory give different error

(dll/ntdll/ldr/ldrpe.c:1035) Failed to snap GetFileInformationByHandleEx
GetFileInformationByHandleEx is vista+ API, while ReactOS provides curently only NT 5.2 APIs

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364953(v=vs.85).aspx

https://jira.reactos.org/secure/attachment/46575/Capture2.PNG

Is it possible to replace or avoid this function somehow?

from zfsin.

Yep, it is used only once:

int win_isatty(HANDLE h)
{
        if (!GetFileInformationByHandleEx(h, FileNameInfo, buf, bufSize)) {
                return 0;
        }

So if you were willing to live without a working test for tty it would be trivial.

from zfsin.

It looks to me that win_isatty supposes the handle 'h' points only to some cygwin/msys handle. So basically, if it's just a handle to a regular Windows console, it wouldn't work?

from zfsin.

It was the test that worked in both cygwin and Windows console, all the other examples would only work in one, not the other. But you can certainly go without it, it wont be able to tell if you are doing something silly like "zfs send" to console instead of file.

from zfsin.

Hmm... so something like:

return ((GetFileType(h) & ~FILE_TYPE_REMOTE) == FILE_TYPE_CHAR);

wouldn't work in cygwin? (it works for a regular Windows console.)

from zfsin.

ok Windows console say:

C:\src\ZFSin>zfs.exe send tank@send
win_isatty: return 1
Error: Stream can not be written to a terminal.
You must redirect standard output.

C:\src\ZFSin>zfs.exe send tank@send > po
win_isatty: return 0

But git's bash shell say

$ ./zfs.exe send tank@send > po
win_isatty: return 0

$ ./zfs.exe send tank@send | cat > po
win_isatty: return 0

$ ./zfs.exe send tank@send
win_isatty: return 0
▒˺▒<▒

^-- should be 1

from zfsin.

Strange that mintty/cygwin doesn't report its console handle as being a TTY.

from zfsin.

Don't forget it is not the "console handle". It is a handle I have opened, which may be file, pipe or STDOUT.

from zfsin.

with OpenZFSonWindows32-20180601.zip now we got the different error

it installs OK, but after zpool.exe status command here goes this problem:

(dll/ntdll/ldr/ldrpe.c:1035) Failed to snap ReleaseSRWLockExclusive

ReleaseSRWLockExclusive seems to be Vista+ function too

https://jira.reactos.org/secure/attachment/46590/Capture3.PNG

from zfsin.

I don't use it, but the pthread wrapper library does:

https://github.com/openzfsonwindows/ZFSin/blob/master/ZFSin/zfs/lib/libpthread/pthread.h#L428

from zfsin.

i have found this library https://github.com/neosmart/RWLock
may be it could help?

Pthread's rwlock objects provide a high-performance means of allowing
either multiple readers to access a resource simultaneously or a
single writer. Windows did not have an equivalent thread
synchronization primitive until Windows Vista was released. However,
most software developers still target Windows XP and above, so this
library should come in handy to fill in the gap.

Non-IPC RWLock() implementations will automatically use Microsoft's
SRW Locks behind the scene if run on Windows Vista or above.

Unlike pthread_rwlock_t and SRW Locks, RWLock can also be used for
interprocess read-write synchronization via placement of a single
intptr_t object memory-mapped or stored in a shared data segment.

from zfsin.

Sure could work, but I am concerned you are finding just one problem at a time, and eventually we'll find a show stopper. Is there a way to list all problems in one?

from zfsin.

If we had a full list of external functions used by ZFSin we then could check how many of them are vista+ and/or missing in ReactOS. But it may take as much time as finding APIs one by one in test ^-)

from zfsin.

@HBelusca the problem is that Dependency Walker 2.2.6000 can not open any file for review. Probably it is a regression. File dialog does not open up.

from zfsin.

Then test that on Win2k3!

from zfsin.

@lundman @HBelusca

ReactOS usually mostly fits with Windows Server 2003 in terms of API

Missing functions in WS 2003r2 EE for Zpool.exe: https://jira.reactos.org/secure/attachment/46594/Capture4.PNG

Not so many. 5 related to SRWlock, 4 related to condition variable.

from zfsin.

ConditionVariables implementation can be taken from here https://github.com/adobe/chromium/blob/master/base/synchronization/condition_variable_win.cc#L127

from zfsin.

Also IESHIMS.DLL and WER.DLL are marked missing, but ZFSin may not really use them.

from zfsin.

But zfsin.sys may miss a lot more functions than exe files

from zfsin.

Yep, the driver itself needs a bit more functionality
https://hsto.org/webt/8t/h9/ly/8th9lyz6jw-calrcl8-q7ccybnq.png
https://hsto.org/webt/a_/ve/y_/a_vey_x5wn0qusieioqh23iojmw.png
17 functions are not present in Win2k3!

from zfsin.

As for zpool - those are all in pthread.h, and if you didn't run zdb, could just be commented out. At least as a quick test.

But without ZFSin.sys there would be little point, which would need more work. The Wdf* things can be removed, I didn't know the difference between drivers and followed Wdf guide until I found better examples.

But for example RtlUnicodeToUTF8N() I use everywhere when going from Windows to ZFS, and back. But it isn't too bad, maybe a couple of days of making SPL wrappers for those functions when they are missing.

from zfsin.

The FS ones are not so concerning, the fastfat example where I got them from had #ifdef that I did not copy. Although, might mean it has to be compiled for each version separately.

from zfsin.

Now for the Rtl* functions, do you want me to lift those into ZFSin, and #ifdef them, or are you happy with doing your linking magic as needed?

from zfsin.

@HeisSpiter @ThFabba @HBelusca what do you think?

from zfsin.

Anyone here better than I on VisualStudio projects? I tried to make a new "platform", copied from win32, but set to compile for an "older version" - so that NTDDI_VERSION = 0x0502 - and produce a binary for older windows. But I am starting to think VS2017 can't really compile for older? What is the official way? (fastfat has the same #ifdef NTDDI so clearly Microsoft says this is the way to test for API versions in the source code).

from zfsin.

Try using v141_xp target toolset, but I'm not sure whether VS2017 still supports it.

from zfsin.

@lundman don't duplicate code unless you want your driver to work under W2K3.

ReactOS has its own library for ntoskrnl Vista functions that FSDs are using, and we statically link to it when importing drivers. But your driver is not yet planned for inclusion into ReactOS.

from zfsin.

A lot was said about XP, Server 2003, 32-bits… they all seem niche choices, whereas Windows 7 x64 is popular on par with Windows 10 according to Netmarketshare. Not to mention Windows 10 is and will be unacceptable choice for prudent privacy-concerned users.

from zfsin.