fhs / npz.jl Goto Github PK

A Julia package that provides support for reading and writing Numpy .npy and .npz files

License: Other

Julia 100.00%

npz.jl's Introduction

Overview

The NPZ package provides support for reading and writing Numpy .npy and .npz files in Julia.

An .npy file contains a single numpy array, stored in a binary format along with its shape, data type, etc. An .npz file contains a collection numpy arrays each encoded in the .npy format and stored in a ZIP file. For more information, see the numpy.save, numpy.savez, numpy.savez_compressed, and numpy.load functions in the Numpy documentation.

Installation

Install via the Julia package manager, Pkg.add("NPZ").

Usage

We try to provide an interface similar to the MAT.jl package. Some usage examples are shown below.

Write and then read back an array:

julia> using NPZ

julia> x = [i-j for i in 1:3, j in 1:4];

julia> npzwrite("x.npy", x)

julia> y = npzread("x.npy")
3x4 Int64 Array:
 0  -1  -2  -3
 1   0  -1  -2
 2   1   0  -1

Write and then read back a collection of arrays or numbers:

julia> npzwrite("data.npz", Dict("x" => x, "a" => collect(1:9), "n" => 42))

julia> vars = npzread("data.npz")
Dict{String,Any} with 3 entries:
  "x" => [0 -1 -2 -3; 1 0 -1 -2; 2 1 0 -1]
  "a" => [1, 2, 3, 4, 5, 6, 7, 8, 9]
  "n" => 42

npz.jl's People

Contributors

Stargazers

Watchers

Forkers

mweastwood akuendig simondanisch tkelman joshchristie galenlynch bhalonen mredies jishnub standardgalactic playfloor zsz00 bkamins shamazmazum berquist musoke

npz.jl's Issues

Deprecation messages in julia v0.7

julia> using NPZ
[ Info: Precompiling NPZ [15e1cf62-19b3-5cfa-8e77-841668bca605]
┌ Warning: Deprecated syntax `type` at /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:140.
│ Use `mutable struct` instead.
└ @ ~/.julia/packages/NPZ/onMHH/src/NPZ.jl:140
WARNING: importing deprecated binding Base.Complex64 into NPZ.
WARNING: Base.Complex64 is deprecated, use ComplexF32 instead.
  likely near /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:17
WARNING: importing deprecated binding Base.Complex128 into NPZ.
WARNING: Base.Complex128 is deprecated, use ComplexF64 instead.
  likely near /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:17
┌ Warning: Deprecated syntax `parametric method syntax npzwritearray{T}(f::IO, x::Array{T})` around /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:263.
│ Use `npzwritearray(f::IO, x::Array{T}) where T` instead.
└ @ ~/.julia/packages/NPZ/onMHH/src/NPZ.jl:263
┌ Warning: Deprecated syntax `parametric method syntax npzwritearray{T <: Number}(f::IO, x::T)` around /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:267.
│ Use `npzwritearray(f::IO, x::T) where T <: Number` instead.
└ @ ~/.julia/packages/NPZ/onMHH/src/NPZ.jl:267
┌ Warning: Deprecated syntax `parametric method syntax npzwrite{S <: AbstractString}(filename::AbstractString, vars::Dict{S, Any})` around /home/jishnu/.julia/packages/NPZ/onMHH/src/NPZ.jl:277.
│ Use `npzwrite(filename::AbstractString, vars::Dict{S, Any}) where S <: AbstractString` instead.
└ @ ~/.julia/packages/NPZ/onMHH/src/NPZ.jl:277

Support reading version 2 npys

If I want to open an *.npy file saved in version 2 format (https://docs.scipy.org/doc/numpy-1.14.0/neps/npy-format.html) I get:

julia> den = npzread("bla.npy")
ERROR: unsupported NPZ version
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] npzreadarray(::IOStream) at ~/.julia/packages/NPZ/TnTz7/src/NPZ.jl:209
 [3] npzread(::String) at ~/.julia/packages/NPZ/TnTz7/src/NPZ.jl:245
 [4] top-level scope at none:0

I belive adding support for v2 is quite easy:
Line 211

hdrlen = readle(f, UInt16)

has to be replaced with an if-statement, that executes

hdrlen = readle(f, UInt32)

in case of v2.

Support pre-allocated array and configurable permutedims?

This package is pretty neat and helpful!

Can we add some keyword arguments to the following function to reuse preallocated array and control whether to permute dims or not?

NPZ.jl/src/NPZ.jl

Lines 229 to 240 in 8834d3c

 function _npzreadarray(f, hdr::Header{T}) where {T} 

 toh = hdr.descr 

 if hdr.fortran_order 

 @compat x = map(toh, read!(f, Array{T}(undef, hdr.shape))) 

 else 

 @compat x = map(toh, read!(f, Array{T}(undef, reverse(hdr.shape)))) 

 if ndims(x) > 1 

 x = permutedims(x, collect(ndims(x):-1:1)) 

 end 

 end 

 ndims(x) == 0 ? x[1] : x 

 end

hope register a new version

hope register a new version for compat v4.x

Installation error on Julia v1.9.2

(TauFinder) pkg> add NPZ
   Resolving package versions...
  No Changes to `~/Code/TauFinder/Project.toml`
  No Changes to `~/Code/TauFinder/Manifest.toml`
Precompiling project...
  Progress [>                                        ]  0/1
  ◑ TauFinder

  1 dependency had warnings during precompilation:
┌ TauFinder [0263dbb0-507e-4701-8e31-5f44b497c465]
│  WARNING: --output requested, but no modules defined during run
└  ┌ Error: Pkg.precompile error
│   exception =
│    failed process: Process(setenv(`/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/libexec/julia/lld -flavor gnu '' -shared -o /home/amin/.julia/compiled/v1.9/TauFinder/jl_MPGosx --whole-archive /home/amin/.julia/compiled/v1.9/TauFinder/jl_yqlZkj --no-whole-archive -L/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/lib -L/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/lib/julia -L/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/lib -ljulia -ljulia-internal`,["_CE_M=", "PATH=/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/libexec/julia:/home/amin/.vscode-server/bin/74f6148eb9ea00507ec113ec51c489d6ffb4b771/bin/remote-cli:/usr/local/cuda-11.6/bin:/home/amin/mambaforge/envs/dev/bin:/home/amin/mambaforge/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "CONDA_PYTHON_EXE=/home/amin/mambaforge/bin/python", "CONDA_PREFIX_1=/home/amin/mambaforge", "LD_LIBRARY_PATH=/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/lib/julia:/home/amin/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/lib", "GIT_ASKPASS=/home/amin/.vscode-server/bin/74f6148eb9ea00507ec113ec51c489d6ffb4b771/extensions/git/dist/askpass.sh", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus", "GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=", "BROWSER=/home/amin/.vscode-server/bin/74f6148eb9ea00507ec113ec51c489d6ffb4b771/bin/helpers/browser.sh", "LANG=en_US.UTF-8"  …  "SSH_CLIENT=10.40.105.174 58072 22", "_CE_CONDA=", "CONDA_SHLVL=2", "CONDA_EXE=/home/amin/mambaforge/bin/conda", "HOME=/home/amin", "TERM=xterm-256color", "LS_COLORS=", "S_COLORS=auto", "COLORTERM=truecolor", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]
│    
│    Stacktrace:
│      [1] pipeline_error
│        @ ./process.jl:565 [inlined]
│      [2] run(::Cmd, ::Base.DevNull, ::Vararg{Any}; wait::Bool)
│        @ Base ./process.jl:480
│      [3] run
│        @ ./process.jl:477 [inlined]
│      [4] link_image (repeats 2 times)
│        @ ./linking.jl:166 [inlined]
│      [5] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2206
│      [6] #249
│        @ ~/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1438 [inlined]
│      [7] withenv(f::Pkg.API.var"#249#280"{IOBuffer, String, Base.PkgId}, keyvals::Pair{String, String})
│        @ Base ./env.jl:197
│      [8] (::Pkg.API.var"#248#279"{IOBuffer, String, Base.PkgId})()
│        @ Pkg.API ~/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1436
│      [9] with_logstate(f::Function, logstate::Any)
│        @ Base.CoreLogging ./logging.jl:514
│     [10] with_logger
│        @ ./logging.jl:626 [inlined]
│     [11] macro expansion
│        @ ./timing.jl:393 [inlined]
│     [12] macro expansion
│        @ ~/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1434 [inlined]
│     [13] (::Pkg.API.var"#245#276"{Bool, Bool, Pkg.Types.Context, Vector{Task}, Dict{Base.PkgId, String}, Dict{Base.PkgId, IOBuffer}, Pkg.API.var"#color_string#266", Base.Event, Base.Event, ReentrantLock, Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, String}, Vector{Base.PkgId}, Vector{Base.PkgId}, Dict{Base.PkgId, Bool}, Dict{Base.PkgId, Base.Event}, Dict{Base.PkgId, Bool}, Vector{Pkg.Types.PackageSpec}, Dict{Base.PkgId, String}, Dict{Tuple{Base.PkgId, UInt128, String, String}, Bool}, Vector{Base.PkgId}, Bool, Base.TTY, Base.Semaphore, String, Vector{String}, Vector{Base.PkgId}, Base.PkgId})()
│        @ Pkg.API ./task.jl:514
└ @ Pkg.API ~/.julia/juliaup/julia-1.9.2+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1302

Tag a new release?

NPZ
A Julia package that provides support for reading and writing Numpy .npy and .npz files

Current version: 0.0.1 (2 years ago) / MIT license / Owner: fhs

Tagged version is probably a bit out of date

Error reading python generated .npy with float64 numbers

Hi, I am getting this error when trying to load a matrix written by python in npy format. The matrix contains float64 numbers. It only happens with dtype=float64 numbers, when I try the same with dtype=int I don't get any errors and the matrix is loaded correctly.

python> matrix.dtype

dtype('float64')

python> np.save('data/matrix.npy', matrix)

julia> npzread("data/matrix.npy")
Illegal inttoptr
	  %33 = ptrtoint double addrspace(13)* %32 to i64
Illegal inttoptr
	  %52 = inttoptr i64 %51 to i8 addrspace(13)*

signal (6): Aborted
in expression starting at REPL[4]:1
gsignal at /usr/bin/../lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /usr/bin/../lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f6165e4bd04)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /usr/bin/../lib/x86_64-linux-gnu/libLLVM-8.so.1 (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /usr/bin/../lib/x86_64-linux-gnu/libLLVM-8.so.1 (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /usr/bin/../lib/x86_64-linux-gnu/libLLVM-8.so.1 (unknown line)
unknown function (ip: 0x7f6165f35ac1)
unknown function (ip: 0x7f6165f382d8)
unknown function (ip: 0x7f6165f388cd)
unknown function (ip: 0x7f6165e73b4a)
unknown function (ip: 0x7f6165ea4ed3)
unknown function (ip: 0x7f6165eca84b)
jl_apply_generic at /usr/bin/../lib/x86_64-linux-gnu/libjulia.so.1 (unknown line)
npzreadarray at /home/javier/.julia/packages/NPZ/OBINJ/src/NPZ.jl:227
npzread at /home/javier/.julia/packages/NPZ/OBINJ/src/NPZ.jl:251
unknown function (ip: 0x7f6165edd75b)
unknown function (ip: 0x7f6165edd389)
unknown function (ip: 0x7f6165edd8f0)
unknown function (ip: 0x7f6165ede9c8)
unknown function (ip: 0x7f6165edf616)
unknown function (ip: 0x7f6165ef7e08)
unknown function (ip: 0x7f6165ef83c8)
jl_toplevel_eval_in at /usr/bin/../lib/x86_64-linux-gnu/libjulia.so.1 (unknown line)
eval at ./boot.jl:331
eval_user_input at /build/julia-98cBbp/julia-1.4.1+dfsg/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
macro expansion at /build/julia-98cBbp/julia-1.4.1+dfsg/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:118 [inlined]
#26 at ./task.jl:358
unknown function (ip: 0x7f6165ee2cbb)
unknown function (ip: (nil))
Allocations: 5381468 (Pool: 5380292; Big: 1176); GC: 5
Aborted (core dumped)

UndefVarError in line 116 of NPZ.jl

Hello,

Here is the line 115-117 of NPZ.jl.

        if s[tail_idx] == 'L' # output of firstindex should be a valid code point

            tail_idx = nextind(s, i)

        end

i in nextind(s, i) is not defined in this function, and results in an UndefVarError. I encountered this while running npzread with julia-1.3.0 on Windows 10.

Thanks!

Option to load and save in row major order

Could you add an option to load and save in row major order since interoperability between Python and Julia can be a pain sometimes.

Support reading from an IOStream

Currently npzread is "hardwired" to read from a file:

NPZ.jl/src/NPZ.jl

Line 271 in 838df34

f = open(filename)

It would be more flexible to expose a version of it that allows reading from an IOStream, e.g., for data loaded from a URL. It would be a simple refactoring of npzread. Would you be open to a PR for that?

Issue in writing dict type

Hi,

Thank you for the NPZ.jl package. I am using the package as follows

function SaveNPZ(loc,ROMParam, U)
    if ispath(loc) == false
        println("Current working directory \n", pwd() )
        # Make the directory....
        mkdir(loc)
    end
    tmp_file  =  joinpath(loc,"mydata.npz") #string( loc,"Zone_",string(j) )loc 
    println(tmp_file)
    npzwrite(tmp_file, Dict("ROMParam" => ROMParam, "U" => U) )
end

I am getting following error at the runtime not sure what I am doing wrong..novice to julia...!!

ERROR: LoadError: MethodError: no method matching npzwritearray(::ZipFile.WritableFile, ::Dict{Any,Any}) Closest candidates are: npzwritearray(::IO, !Matched::AbstractArray{UInt8,N} where N, !Matched::DataType, !Matched::Array{Int64,1}) at /Users/shar306/.julia/packages/NPZ/OBINJ/src/NPZ.jl:273 npzwritearray(::IO, !Matched::AbstractArray{T,N} where N) where T at /Users/shar306/.julia/packages/NPZ/OBINJ/src/NPZ.jl:298 npzwritearray(::IO, !Matched::T) where T<:Number at /Users/shar306/.julia/packages/NPZ/OBINJ/src/NPZ.jl:302

Error while downloading

julia> Pkg.add("NPZ")
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
   Cloning [15e1cf62-19b3-5cfa-8e77-841668bca605] NPZ from https://github.com/fhs/NPZ.jl.git
[ Info: Installed NPZ ───────────────── v0.4.0
   Cloning [731186ca-8d62-57ce-b412-fbd966d074cd] RecursiveArrayTools from https://github.com/JuliaDiffEq/RecursiveArrayTools.jl.git
[ Info: Installed RecursiveArrayTools ─ v2.1.0
   Cloning [79e6a3ab-5dfb-504d-930d-738a2a938a0e] Adapt from https://github.com/JuliaGPU/Adapt.jl.git
[ Info: Installed Adapt ─────────────── v1.0.1
   Cloning [8f4d0f93-b110-5947-807f-2305c1781a2d] Conda from https://github.com/JuliaPy/Conda.jl.git
[ Info: Installed Conda ─────────────── v1.4.1
   Cloning [a5390f91-8eb1-5f08-bee0-b1d1ffed6cea] ZipFile from https://github.com/fhs/ZipFile.jl.git
[ Info: Installed ZipFile ───────────── v0.8.4
   Cloning [2b5f629d-d688-5b77-993f-72d75c75574e] DiffEqBase from https://github.com/JuliaDiffEq/DiffEqBase.jl.git
[ Info: Installed DiffEqBase ────────── v6.14.1
   Cloning [bcd4f6db-9728-5f36-b5f7-82caef46ccdb] DelayDiffEq from https://github.com/JuliaDiffEq/DelayDiffEq.jl.git
[ Info: Installed DelayDiffEq ───────── v5.21.0
  Updating `~/.julia/environments/v1.0/Project.toml`
  [15e1cf62] + NPZ v0.4.0
  Updating `~/.julia/environments/v1.0/Manifest.toml`
  [79e6a3ab] ↑ Adapt v1.0.0 ⇒ v1.0.1
  [8f4d0f93] ↑ Conda v1.3.0 ⇒ v1.4.1
  [bcd4f6db] ↑ DelayDiffEq v5.20.0 ⇒ v5.21.0
  [2b5f629d] ↑ DiffEqBase v6.13.3 ⇒ v6.14.1
  [15e1cf62] + NPZ v0.4.0
  [731186ca] ↑ RecursiveArrayTools v2.0.5 ⇒ v2.1.0
  [a5390f91] + ZipFile v0.8.4
  Building Conda ──→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
  Building ZipFile → `~/.julia/packages/ZipFile/DW0Qr/deps/build.log`
┌ Error: Error building `ZipFile`: 
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz to /home/developer/.julia/packages/ZipFile/DW0Qr/deps/usr/downloads/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz...
│ ERROR: LoadError: Could not download https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz to /home/developer/.julia/packages/ZipFile/DW0Qr/deps/usr/downloads/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz:
│ ErrorException("")
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #download#89(::Bool, ::Function, ::String, ::String) at /home/developer/.julia/packages/BinaryProvider/kcGxO/src/PlatformEngines.jl:502
│  [3] #download at ./none:0 [inlined]
│  [4] #download_verify#90(::Bool, ::Bool, ::Bool, ::Function, ::String, ::String, ::String) at /home/developer/.julia/packages/BinaryProvider/kcGxO/src/PlatformEngines.jl:571
│  [5] #download_verify at ./none:0 [inlined]
│  [6] #install#129(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Function, ::String, ::String) at /home/developer/.julia/packages/BinaryProvider/kcGxO/src/Prefix.jl:314
│  [7] (::getfield(BinaryProvider, Symbol("#kw##install")))(::NamedTuple{(:prefix, :force, :verbose),Tuple{Prefix,Bool,Bool}}, ::typeof(install), ::String, ::String) at ./none:0
│  [8] top-level scope at /home/developer/.julia/packages/ZipFile/DW0Qr/deps/build.jl:44
│  [9] include at ./boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at ./loading.jl:1044
│  [11] include(::Module, ::String) at ./sysimg.jl:29
│  [12] include(::String) at ./client.jl:392
│  [13] top-level scope at none:0
│ in expression starting at /home/developer/.julia/packages/ZipFile/DW0Qr/deps/build.jl:42
│ [10:46:14] curl: /home/developer/julia-1.0.5/bin/../lib/julia/libcurl.so.4: no version information available (required by curl)
│ [10:46:14] 
│ [10:46:14] curl: (77) Error reading ca cert file /etc/pki/tls/certs/ca-bundle.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1096

julia> using NPZ
[ Info: Precompiling NPZ [15e1cf62-19b3-5cfa-8e77-841668bca605]
ERROR: LoadError: LoadError: could not open file /home/developer/.julia/packages/ZipFile/DW0Qr/deps/deps.jl
Stacktrace:
 [1] include at ./boot.jl:317 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1044
 [3] include at ./sysimg.jl:29 [inlined]
 [4] include(::String) at /home/developer/.julia/packages/ZipFile/DW0Qr/src/Zlib.jl:26
 [5] top-level scope at none:0
 [6] include at ./boot.jl:317 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1044
 [8] include at ./sysimg.jl:29 [inlined]
 [9] include(::String) at /home/developer/.julia/packages/ZipFile/DW0Qr/src/ZipFile.jl:36
 [10] top-level scope at none:0
 [11] include at ./boot.jl:317 [inlined]
 [12] include_relative(::Module, ::String) at ./loading.jl:1044
 [13] include(::Module, ::String) at ./sysimg.jl:29
 [14] top-level scope at none:2
 [15] eval at ./boot.jl:319 [inlined]
 [16] eval(::Expr) at ./client.jl:393
 [17] top-level scope at ./none:3
in expression starting at /home/developer/.julia/packages/ZipFile/DW0Qr/src/Zlib.jl:50
in expression starting at /home/developer/.julia/packages/ZipFile/DW0Qr/src/ZipFile.jl:43
ERROR: LoadError: Failed to precompile ZipFile [a5390f91-8eb1-5f08-bee0-b1d1ffed6cea] to /home/developer/.julia/compiled/v1.0/ZipFile/cOum2.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
 [6] include at ./boot.jl:317 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1044
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:319 [inlined]
 [11] eval(::Expr) at ./client.jl:393
 [12] top-level scope at ./none:3
in expression starting at /home/developer/.julia/packages/NPZ/OBINJ/src/NPZ.jl:7
ERROR: Failed to precompile NPZ [15e1cf62-19b3-5cfa-8e77-841668bca605] to /home/developer/.julia/compiled/v1.0/NPZ/TV7Kn.ji.
Stacktrace:
 [1] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [2] _require(::Base.PkgId) at ./loading.jl:960
 [3] require(::Base.PkgId) at ./loading.jl:858
 [4] require(::Module, ::Symbol) at ./loading.jl:853

My JUlia version is

julia> VERSION
v"1.0.5"

[PkgEval] NPZ may have a testing issue on Julia 0.4 (2014-09-29)

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.4

On 2014-09-28 the testing status was Tests pass.
On 2014-09-29 the testing status changed to Tests fail, but package loads.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Tests fail, but package loads. means that PackageEvaluator found the tests for your package, executed them, and they didn't pass. However, trying to load your package with using worked.

This error on Julia 0.4 is possibly due to recently merged pull request JuliaLang/julia#8420.
This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("NPZ")' log
INFO: Installing NPZ v0.0.1
INFO: Installing ZipFile v0.2.2
INFO: Installing Zlib v0.1.7
INFO: Package database updated

>>> 'using NPZ' log
Julia Version 0.4.0-dev+842
Commit e5d8c1a (2014-09-29 06:50 UTC)
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

>>> test log

ERROR: InexactError()
 in read at io.jl:108
 in npzreadarray at /home/idunning/pkgtest/.julia/v0.4/NPZ/src/NPZ.jl:169
 in npzread at /home/idunning/pkgtest/.julia/v0.4/NPZ/src/NPZ.jl:206
 in npzread at /home/idunning/pkgtest/.julia/v0.4/NPZ/src/NPZ.jl:186
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
 in _start_3B_3625 at /home/idunning/julia04/usr/bin/../lib/julia/sys.so
while loading /home/idunning/pkgtest/.julia/v0.4/NPZ/test/test.jl, in expression starting on line 51

>>> end of log

Suggest to add string support

In Numpy, string array to npz file is legal. "np.savez_compressed" in Python works fine.
ex: something like ["name1" "name2" "name3"]

So far, NPZ.jl can not read/write such kind of data. Would you like to add the support?

Feature request: Sparse matrices

Are there any plans to support sparse matrices?

Cannot write LinSpace

In Julia v0.5 the output type of the linspace() function has been changed to LinSpace, which is not a subtype of Array, but of AbstractArray. Because of this, it is not seamlessly possible to write the output of linspace() and similar functions.

> NPZ.npzwritearray("test.npz", linspace(1,2))
ERROR: MethodError: no method matching npzwritearray(::String, ::LinSpace{Float64})

It works when you explicitly convert it to Array, but wouldn't it be better to change the signature of npzwritearray() to accept AbstractArray?

npzwrite cannot write Dict in julia 0.6

It seems that the function overload at line 276 in NPZ.jl has not been recognized because the error indicates a call to line 272 (... src/NPZ.jl:272).

My idea to fix it is simply adding a new function called "npzwrite_dict" ... or changing the name of that function. But I am completely new to GitHub and don't know how to update it ...

Any helps ?

yunlong@THEO-7101:~$ julia
_
_ _ ()_ | A fresh approach to technical computing
() | () () | Documentation: https://docs.julialang.org
_ _ | | __ _ | Type "?help" for help.
| | | | | | |/ ` | |
| | || | | | (| | | Version 0.6.0 (2017-06-19 13:05 UTC)
/ |_'|||_'_| | Official http://julialang.org/ release
|__/ | x86_64-pc-linux-gnu

julia> using NPZ

julia> dict = Dict{String,Array{Float64,1}}("en"=>Float64[1.0,-1.0,2.0,-2.0])
Dict{String,Array{Float64,1}} with 1 entry:
"en" => [1.0, -1.0, 2.0, -2.0]

julia> npzwrite("test.npz", dict)
ERROR: MethodError: no method matching npzwritearray(::IOStream, ::Dict{String,Array{Float64,1}})
Closest candidates are:
npzwritearray(::IO, ::Array{UInt8,N} where N, ::DataType, ::Array{Int64,1}) at /home/yunlong/.julia/v0.6/NPZ/src/NPZ.jl:238
npzwritearray(::IO, ::Array{T,N} where N) where T at /home/yunlong/.julia/v0.6/NPZ/src/NPZ.jl:263
npzwritearray(::IO, ::T<:Number) where T<:Number at /home/yunlong/.julia/v0.6/NPZ/src/NPZ.jl:267
Stacktrace:
[1] npzwrite(::String, ::Dict{String,Array{Float64,1}}) at /home/yunlong/.julia/v0.6/NPZ/src/NPZ.jl:272

julia>

Version in official Julia package repo is out of date, gives deprecation warnings in v0.5

The master version here in github works fine in Julia v0.5, but the version pulled from the official package repo via Pkg.add("NPZ") isn't up-to-date with what's here, and gives deprecation warnings.

Error handling empty array

Although it's weird to store an empty array in the npz file (I forgot to delete that field 😢), NPZ.jl will throw an error when it trys to unpack an npz file with empty array.

Steps to reproduce

create an npz file with empty field

In [4]: a=np.load("aba.npz")

In [5]: a["a"]
Out[5]: array([], dtype=float64)

in julia

julia> npzread("aba.npz")
ERROR: EOFError: read end of file
Stacktrace:
  [1] _read(f::ZipFile.ReadableFile, a::Vector{UInt8})
    @ ZipFile ~/.julia/packages/ZipFile/evaHP/src/ZipFile.jl:519
  [2] read!
    @ ~/.julia/packages/ZipFile/evaHP/src/ZipFile.jl:483 [inlined]
  [3] readheader(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:210
  [4] npzreadarray(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:243
  [5] (::NPZ.var"#4#7")(f::ZipFile.ReadableFile)
    @ NPZ ./none:0
  [6] iterate
    @ ./generator.jl:47 [inlined]
  [7] _all(f::Base.var"#318#320", itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"}, #unused#::Colon)
    @ Base ./reduce.jl:1156
  [8] all
    @ ./reduce.jl:1152 [inlined]
  [9] Dict(kv::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:131
 [10] npzread (repeats 2 times)
    @ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:311 [inlined]
 [11] npzread(::String)
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:295
 [12] top-level scope
    @ REPL[1]:1

caused by: EOFError: read end of file
Stacktrace:
  [1] _read(f::ZipFile.ReadableFile, a::Vector{Float64})
    @ ZipFile ~/.julia/packages/ZipFile/evaHP/src/ZipFile.jl:519
  [2] _npzreadarray(f::ZipFile.ReadableFile, hdr::NPZ.Header{Float64, 1, typeof(ltoh)})
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:0
  [3] npzreadarray(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:244
  [4] (::NPZ.var"#4#7")(f::ZipFile.ReadableFile)
    @ NPZ ./none:0
  [5] iterate
    @ ./generator.jl:47 [inlined]
  [6] grow_to!(dest::Dict{Any, Any}, itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:140
  [7] dict_with_eltype
    @ ./abstractdict.jl:550 [inlined]
  [8] Dict(kv::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:129
  [9] npzread (repeats 2 times)
    @ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:311 [inlined]
 [10] npzread(::String)
    @ NPZ ~/.julia/packages/NPZ/uRQRY/src/NPZ.jl:295
 [11] top-level scope
    @ REPL[1]:1

"EOFError: read end of file" reading a file

Here a reproducible example

shell> wget https://github.com/shchur/gnn-benchmark/raw/master/data/npz/cora.npz 

julia> using NPZ

julia> npzread("cora.npz")
ERROR: EOFError: read end of file
Stacktrace:
  [1] _read(f::ZipFile.ReadableFile, a::Vector{UInt8})
    @ ZipFile ~/.julia/packages/ZipFile/fdYkP/src/ZipFile.jl:519
  [2] read!
    @ ~/.julia/packages/ZipFile/fdYkP/src/ZipFile.jl:483 [inlined]
  [3] readheader(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:210
  [4] npzreadarray(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:243
  [5] (::NPZ.var"#4#7")(f::ZipFile.ReadableFile)
    @ NPZ ./none:0
  [6] iterate
    @ ./generator.jl:47 [inlined]
  [7] _all(f::Base.var"#282#284", itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"}, #unused#::Colon)
    @ Base ./reduce.jl:922
  [8] all
    @ ./reduce.jl:918 [inlined]
  [9] Dict(kv::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:131
 [10] npzread (repeats 2 times)
    @ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:311 [inlined]
 [11] npzread(::String)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:295
 [12] top-level scope
    @ REPL[3]:1

caused by: parsing header failed: unsupported type U7
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] parsedtype
    @ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:157 [inlined]
  [3] parseheader(s::SubString{String})
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:185
  [4] readheader(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:226
  [5] npzreadarray(f::ZipFile.ReadableFile)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:243
  [6] (::NPZ.var"#4#7")(f::ZipFile.ReadableFile)
    @ NPZ ./none:0
  [7] iterate
    @ ./generator.jl:47 [inlined]
  [8] grow_to!(dest::Dict{String, Vector{T} where T}, itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"}, st::Int64)
    @ Base ./dict.jl:162
  [9] grow_to!(dest::Dict{String, Vector{Float32}}, itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"}, st::Int64)
    @ Base ./dict.jl:160
 [10] grow_to!(dest::Dict{Any, Any}, itr::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:145
 [11] dict_with_eltype
    @ ./abstractdict.jl:545 [inlined]
 [12] Dict(kv::Base.Generator{Base.Iterators.Filter{NPZ.var"#5#8"{Vector{String}}, Vector{ZipFile.ReadableFile}}, NPZ.var"#4#7"})
    @ Base ./dict.jl:129
 [13] npzread (repeats 2 times)
    @ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:311 [inlined]
 [14] npzread(::String)
    @ NPZ ~/.julia/packages/NPZ/UizJj/src/NPZ.jl:295
 [15] top-level scope
    @ REPL[3]:1

The downloaded file is correctly read from python

Missing support for numpy.record dtype

I have a npy file that I cannot read. I keep getting an error saying parsing header failed: expected character ''', found '['.

The file can be created as follows...

df = pd.DataFrame({"i": [1, 2, 3],"j": [4, 5, 6],"k":[3.14, 2.72, 1.62]})
arr = np.ascontiguousarray(df.to_records(index=False)).view()
np.save("/tmp/temp.npy", arr)

or alternatively just writing the bytes of the file from julia

bytes = UInt8[0x93, 0x4e, 0x55, 0x4d, 0x50, 0x59, 0x01, 0x00, 0x66, 0x00, 0x7b, 0x27, 0x64, 0x65, 0x73, 0x63, 0x72, 0x27, 0x3a, 0x20, 0x5b, 0x28, 0x27, 0x69, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x69, 0x38, 0x27, 0x29, 0x2c, 0x20, 0x28, 0x27, 0x6a, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x69, 0x38, 0x27, 0x29, 0x2c, 0x20, 0x28, 0x27, 0x6b, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x66, 0x38, 0x27, 0x29, 0x5d, 0x2c, 0x20, 0x27, 0x66, 0x6f, 0x72, 0x74, 0x72, 0x61, 0x6e, 0x5f, 0x6f, 0x72, 0x64, 0x65, 0x72, 0x27, 0x3a, 0x20, 0x46, 0x61, 0x6c, 0x73, 0x65, 0x2c, 0x20, 0x27, 0x73, 0x68, 0x61, 0x70, 0x65, 0x27, 0x3a, 0x20, 0x28, 0x33, 0x2c, 0x29, 0x2c, 0x20, 0x7d, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x0a, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f, 0x85, 0xeb, 0x51, 0xb8, 0x1e, 0x09, 0x40, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc3, 0xf5, 0x28, 0x5c, 0x8f, 0xc2, 0x05, 0x40, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xec, 0x51, 0xb8, 0x1e, 0x85, 0xeb, 0xf9, 0x3f]
open("/tmp/temp.npy", "w") do f
    write(f, bytes)
end

Then i read the file the usual way...

npzread("/tmp/temp.npy")

and this is my stacktrace

parsing header failed: expected character ''', found '['

Stacktrace:
 [1] parsechar(::String, ::Char) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:70
 [2] parsestring(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:76
 [3] parsedtype(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:123
 [4] parseheader(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:157
 [5] npzreadarray(::IOStream) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:191
 [6] npzread(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:218

versioninfo():

Julia Version 0.6.1-pre.0
Commit dcf39a1 (2017-06-19 13:06 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libmkl_rt
  LAPACK: libmkl_rt
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

And my NPZ package has head at

* 60c4bc9 2017-02-01 (HEAD, tag: v0.2.0, origin/master, cache/heads/master, master) REQUIRE: downgrade Compat because we don't use Compat.view [Fazlul Shahriar]

unsupported type O8

I tried loading a data file, which should have arrays of floats and bools, and got the following error (with additional debug output): Is this something easy to fix?

!julia> a=npzread("for_deniz.npz")
(:parseheader,s) = (:parseheader,"'descr': '|O8', 'fortran_order': False, 'shape': (3250, 2), }")
(:parseheader,key,s) = (:parseheader,"descr",": '|O8', 'fortran_order': False, 'shape': (3250, 2), }")
(:parsedtype,s) = (:parsedtype,"'|O8', 'fortran_order': False, 'shape': (3250, 2), }")
(:parsedtype,dtype,s) = (:parsedtype,"|O8",", 'fortran_order': False, 'shape': (3250, 2), }")
(:parsedtype,c,t) = (:parsedtype,'|',"O8")
ERROR: parsing header failed: unsupported type O8
in parsedtype at /state/partition1/dyuret/.julia/v0.4/NPZ/src/NPZ.jl:122
in parseheader at /state/partition1/dyuret/.julia/v0.4/NPZ/src/NPZ.jl:146
in npzreadarray at /state/partition1/dyuret/.julia/v0.4/NPZ/src/NPZ.jl:180
in npzread at /state/partition1/dyuret/.julia/v0.4/NPZ/src/NPZ.jl:221
in npzread at /state/partition1/dyuret/.julia/v0.4/NPZ/src/NPZ.jl:201

Suggestion for mmap_mode support

Support for mmap_mode optional argument for NPZ.npzread would be nice, to be able to operate on big .npy files without needing to load the whole array to the memory.

Failed to precompile NPZ

Installed using Pkg.add("NPZ") on a fresh install of Julia 1.0.0. Got the following error when I tried to use import NPZ/using NPZ.

julia> import NPZ
[ Info: Precompiling NPZ [15e1cf62-19b3-5cfa-8e77-841668bca605]
ERROR: LoadError: syntax: extra token "Header" after end of expression
Stacktrace:
 [1] include at ./boot.jl:317 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] include(::Module, ::String) at ./sysimg.jl:29
 [4] top-level scope at none:2
 [5] eval at ./boot.jl:319 [inlined]
 [6] eval(::Expr) at ./client.jl:389
 [7] top-level scope at ./none:3
in expression starting at /home/pal/.julia/packages/NPZ/onMHH/src/NPZ.jl:140
ERROR: Failed to precompile NPZ [15e1cf62-19b3-5cfa-8e77-841668bca605] to /home/pal/.julia/compiled/v1.0/NPZ/TV7Kn.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834

Output of versioninfo(), if it's helpful:

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

Actually writing Dicts with npzwrite

using NPZ
my_dict=Dict("a"=>rand(3,3))
npzwrite("temp.npz",my_dict)

results in

ERROR: MethodError: no method matching npzwritearray(::IOStream, ::Dict{String,Array{Float64,2}})
Closest candidates are:
  npzwritearray(::IO, ::AbstractArray{UInt8,N} where N, ::DataType, ::Array{Int64,1}) at /mnt/ssd/projects/julia-projects/FidelityMetrics/dev/NPZ/src/NPZ.jl:267
  npzwritearray(::IO, ::AbstractArray{T,N} where N) where T at /mnt/ssd/projects/julia-projects/FidelityMetrics/dev/NPZ/src/NPZ.jl:292
  npzwritearray(::IO, ::T<:Number) where T<:Number at /mnt/ssd/projects/julia-projects/FidelityMetrics/dev/NPZ/src/NPZ.jl:296
Stacktrace:
 [1] npzwrite(::String, ::Dict{String,Array{Float64,2}}) at /mnt/ssd/projects/julia-projects/FidelityMetrics/dev/NPZ/src/NPZ.jl:300
 [2] top-level scope at none:0

Errors in julia 0.5

I got the following warnings and an error in Julia 0.5

julia> using NPZ

WARNING: deprecated syntax "[a=>b for (a,b) in c]".
Use "Dict(a=>b for (a,b) in c)" instead.

WARNING: deprecated syntax "[a=>b for (a,b) in c]".
Use "Dict(a=>b for (a,b) in c)" instead.
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:55
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:58
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:65
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:74
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:83
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:89
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:109
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:133

julia> x = [i-j for i in 1:3, j in 1:4];

julia> npzwrite("x.npy", x)

julia> y = npzread("x.npy")
WARNING: Base.UTF8String is deprecated, use String instead.
  likely near no file:0
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near no file:0
WARNING: Base.UTF8String is deprecated, use String instead.
  likely near no file:0
WARNING: Base.ASCIIString is deprecated, use String instead.
  likely near no file:0
WARNING: Base.UTF8String is deprecated, use String instead.
  likely near no file:0
WARNING: ascii(v::Vector{UInt8}) is deprecated, use ascii(String(v)) instead.
 in ascii(::Array{UInt8,1}) at ./deprecated.jl:50
 in npzreadarray(::IOStream) at /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:177
 in npzread(::String) at /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:205
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
 in macro expansion at ./REPL.jl:95 [inlined]
 in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
while loading no file, in expression starting on line 0
3×4 Array{Int64,2}:
 0  -1  -2  -3
 1   0  -1  -2
 2   1   0  -1

julia> npzwrite("data.npz", ["x" => x, "a" => [1:9], "n" => 42])
ERROR: argument is an abstract type; size is indeterminate
 in reinterpret at ./array.jl:73 [inlined]
 in npzwritearray(::IOStream, ::Array{Pair{String,B},1}) at /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:250
 in npzwrite(::String, ::Array{Pair{String,B},1}) at /home/phuoc/.julia/v0.5/NPZ/src/NPZ.jl:259
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
 in macro expansion at ./REPL.jl:95 [inlined]
 in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68

julia>

Enhancement: Load as column-major array

Often, I write a NxF numpy array with N observations and F features in row-major order in order to have the observations in contiguous memory. When loading to Julia, I prefer to load it to a FxN Matrix, such that observations are still contiguous. Currently, this is achievable only by loading in the data (which internally transposes it), then re-transposing it back. This is inefficient.
I propose having some kind of keyword "keep_contiguity" or something less terribly-named, which loads the array in as if it was "fortran-contiguous", even if it actually isn't. I.e, given an NxF numpy array in C-contiguous order, return a FxN Matrix.
I can pitch a PR if you like.

New release with Compat 4 support?

Master has supported Compat 4 for some time, but the latest release is limited to Compat 3.

Error while reading .npy file: parsing header failed: unsupported type O

I'm using julia 0.6.1, installed NPZ by Pkg.add("NPZ")

I have a .npy file that created by numpy 1.14.3 on Ubuntu 16.04, Conda 4.4.10.

I get the following error trying to read it in

parsing header failed: unsupported type O

Stacktrace:
 [1] parsedtype(::String) at /opt/anaconda3/share/julia/site/v0.6/NPZ/src/NPZ.jl:135
 [2] parseheader(::String) at /opt/anaconda3/share/julia/site/v0.6/NPZ/src/NPZ.jl:157
 [3] npzreadarray(::IOStream) at /opt/anaconda3/share/julia/site/v0.6/NPZ/src/NPZ.jl:191
 [4] npzread(::String) at /opt/anaconda3/share/julia/site/v0.6/NPZ/src/NPZ.jl:218

Here is my file if you have time to take a look at. It's just a regular 1D, np.float64 array.

TagBot is not creating GithubReleases

v0.42 and v0.43 hare available in the JuliaRegistries but TagBot didn't create the corresponding GitHub release.
It is the first time I see this behavior and I don't know what is causing it.
According to the TagBot readme no action besides creating the TagBot.yml should be required.

Via Stackoverflow: npzread(filename.npz) very slow

Timings degrade critically when .npz contains several variables, see https://stackoverflow.com/questions/46116346/julia-npzreadfilename-npz-very-slow . In

Possibly because of https://github.com/fhs/NPZ.jl/blob/master/src/NPZ.jl#L197

Deal with "long integers" like "1024L" in shape

Thanks for making NPZ! Nice to be able to use my numpy data so easily :)

Unfortunately, I noticed a small problem:

On some systems, the shape of an array looks like (1024L, 1024L) rather than (1024, 1024), indicating the use of long integers. This is the case on my Anaconda installation on Windows, for example. See also this.

This is also true in the header of generated npy files. For example:
{'descr': '<c16', 'fortran_order': False, 'shape': (3L, 32L, 32L), }

It would be nice if NPZ would handle this case too!

Incorrect handling of Unicode keys when creating npz files

Hi, I am running into issues when using NPZ to create an npz file that uses unicode strings as keys.

Just to be clear, everything works fine when creating the file using Numpy and reading it using NPZ, i.e. this works fine in Python

>>> import numpy as np

>>> np.savez("file.npz", α=1)

>>> D = np.load("file.npz")

>>> print(D["α"])
1

and reading the file in Julia using NPZ also works as expected

julia> using NPZ

julia> D = npzread("file.npz")
Dict{String, Int64} with 1 entry:
  "α" => 1

julia> D["α"]
1

However, if I try creating this file from NPZ, while NPZ can read it as expected, it cannot be properly read by Numpy.
Indeed, from the NPZ side:

julia> npzwrite("file.npz", Dict("α" => 1))

julia> D = npzread("file.npz")
Dict{String, Int64} with 1 entry:
  "α" => 1

julia> D["α"]
1

everything works fine. However, when I try opening the file with Numpy, while it does load it, the keys are not what I would expect:

>>> D = np.load("file.npz")

>>> print(D["α"])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-17-7d756a0b03cf> in <module>
----> 1 print(D["α"])

/usr/lib/python3.9/site-packages/numpy/lib/npyio.py in __getitem__(self, key)
    258                 return self.zip.read(key)
    259         else:
--> 260             raise KeyError("%s is not a file in the archive" % key)
    261 
    262 

KeyError: 'α is not a file in the archive'

Indeed if I print the keys of the loaded file I get some different unicode string:

>>> list(D.keys())
['╬▒']

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

	function _npzreadarray(f, hdr::Header{T}) where {T}
	toh = hdr.descr
	if hdr.fortran_order
	@compat x = map(toh, read!(f, Array{T}(undef, hdr.shape)))
	else
	@compat x = map(toh, read!(f, Array{T}(undef, reverse(hdr.shape))))
	if ndims(x) > 1
	x = permutedims(x, collect(ndims(x):-1:1))
	end
	end
	ndims(x) == 0 ? x[1] : x
	end