Giter VIP home page Giter VIP logo

Comments (10)

WilliamWalker avatar WilliamWalker commented on July 20, 2024

Has (can) anything been done with this issue? I received an OverflowError: Python int too large to convert to C long error (base.py, line 948) when trying to add a WKT VLR to a large (3GB+) LAS file. It works fine with smaller tiled data, but fails on the large full swath data.

from laspy.

grantbrown avatar grantbrown commented on July 20, 2024

I'd need a bit more about what exactly you're trying to accomplish, do you have a code snippet you can post?

from laspy.

WilliamWalker avatar WilliamWalker commented on July 20, 2024
import os, struct, sys, glob
import laspy

if __name__ == "__main__":
    wkt = 'PROJCS["NAD_1983_UTM_Zone_15N",GEOGCS["GCS_NAD_1983_2011",DATUM["NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433],AUTHORITY["EPSG","4269"]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-93.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Meter",1.0]],AUTHORITY["EPSG","26915"]]'

    # Read las files
    for las_file in sorted(glob.glob(sys.argv[1])):
        print("Attempting %s..." % os.path.basename(las_file))

        f = laspy.file.File(las_file, mode='rw')

        # WARNING: Removing all VLRs
        inVLRs = []

        # OGC Coordinate System WKT
        new_vlr = laspy.header.VLR(user_id = "LASF_Projection",
                                   record_id = 2112,
                                   VLR_body = wkt,
                                   description = "OGC Coordinate System WKT")

        inVLRs.append(new_vlr)
        f.header.vlrs = inVLRs

        # Set WKT Global Encoding Bit
        f.header.wkt = 1

        f.close()

When I run this on a 5.8GB LAS file, I receive the following error:

Traceback (most recent call last):
File "D:\path\to\above\code.py" line...
f.header.vlrs = inVLRs
File "C:\path\to\laspy\header.py", line 1275, in set_vlrs
    self.reader.set_vlrs(value)
File "C:\path\to\laspy\base.py", line 948, in set_vlrs
    dat_part_2 = self.data_provider.fileref.read(current_size - old_offset)
OverflowError: Python int too large to convert to C long

If it helps, my file has 198257969 points, version 1.4, scale = (0.001, 0.001, 0.001), offset = (230247.29990000001, 3550963.7000000002, 0)

from laspy.

grantbrown avatar grantbrown commented on July 20, 2024

Hmm, I don't know why it would fail at that step. I'll try to see if I can reproduce the issue on a test file and get back to you. Just to be sure: what is your platform and python version? Are you running 64bit python?

from laspy.

WilliamWalker avatar WilliamWalker commented on July 20, 2024

Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32

from laspy.

hobu avatar hobu commented on July 20, 2024

When I run this on a 5.8GB LAS file

Don't do that! 😄

from laspy.

WilliamWalker avatar WilliamWalker commented on July 20, 2024

Well....I would love to.....but, we've been tasked to deliver full swath data, i.e. uncut flight lines.

from laspy.

hobu avatar hobu commented on July 20, 2024

Just teasing. It makes me shake my head when people ask for such large files.

from laspy.

WilliamWalker avatar WilliamWalker commented on July 20, 2024

Just an update. I've replaced the read/write mode with a read and write mode (2 separate files) and everything works as it should.

for las_file in sorted(glob.glob(sys.argv[1])):
    print("Attempting %s..." % os.path.basename(las_file))

    f = laspy.file.File(las_file, mode='r')
    h = f.header
    inVLRs = []

    # OGC Coordinate System WKT
    new_vlr = laspy.header.VLR(user_id = "LASF_Projection",
                               record_id = 2112,
                               VLR_body = wkt,
                               description = "OGC Coordinate System WKT")

    # Initialize output file for writing data
    out_file = os.path.join(os.path.dirname(las_file), os.path.splitext(las_file)[0] + "_HEADER.las")
    out_f = laspy.file.File(out_file, mode='w', header = h)

    inVLRs.append(new_vlr)
    out_f.header.vlrs = inVLRs

    # Set WKT Global Encoding Bit
    out_f.header.wkt = 1

    # Set GPS Global Encoding Bit
    out_f.header.gps_time_type = 1

    out_f.points = f.points
    f.close()
    out_f.close()

The only downside of this, however, is that this method is extremely slow on my larger files.

from laspy.

grantbrown avatar grantbrown commented on July 20, 2024

I still haven't had a chance to really dig into this, sorry for the delay.

Since you're not doing anything to the point values, the line:

out_f.points = f.points

is going to be doing a lot of work you don't care about. You should be able to speed this up considerably by reaching into some of the guts of the DataProvider and Writer objects instead of using the helper properties on the File object... but I'd need to spend some time figuring out exactly what that would look like (it's not how the library was originally intended to be used).

Long term, Laspy still isn't going to compete in speed with something like PDAL, because:

  1. PDAL is written in C++
  2. You'll only "pay for what you use" in terms of features

If you're going to be processing lots and lots of these, that might be a good investment of time.

from laspy.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.