Comments (10)
Has (can) anything been done with this issue? I received an OverflowError: Python int too large to convert to C long
error (base.py, line 948) when trying to add a WKT VLR to a large (3GB+) LAS file. It works fine with smaller tiled data, but fails on the large full swath data.
from laspy.
I'd need a bit more about what exactly you're trying to accomplish, do you have a code snippet you can post?
from laspy.
import os, struct, sys, glob
import laspy
if __name__ == "__main__":
wkt = 'PROJCS["NAD_1983_UTM_Zone_15N",GEOGCS["GCS_NAD_1983_2011",DATUM["NAD_1983_2011",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433],AUTHORITY["EPSG","4269"]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-93.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Meter",1.0]],AUTHORITY["EPSG","26915"]]'
# Read las files
for las_file in sorted(glob.glob(sys.argv[1])):
print("Attempting %s..." % os.path.basename(las_file))
f = laspy.file.File(las_file, mode='rw')
# WARNING: Removing all VLRs
inVLRs = []
# OGC Coordinate System WKT
new_vlr = laspy.header.VLR(user_id = "LASF_Projection",
record_id = 2112,
VLR_body = wkt,
description = "OGC Coordinate System WKT")
inVLRs.append(new_vlr)
f.header.vlrs = inVLRs
# Set WKT Global Encoding Bit
f.header.wkt = 1
f.close()
When I run this on a 5.8GB LAS file, I receive the following error:
Traceback (most recent call last):
File "D:\path\to\above\code.py" line...
f.header.vlrs = inVLRs
File "C:\path\to\laspy\header.py", line 1275, in set_vlrs
self.reader.set_vlrs(value)
File "C:\path\to\laspy\base.py", line 948, in set_vlrs
dat_part_2 = self.data_provider.fileref.read(current_size - old_offset)
OverflowError: Python int too large to convert to C long
If it helps, my file has 198257969 points, version 1.4, scale = (0.001, 0.001, 0.001), offset = (230247.29990000001, 3550963.7000000002, 0)
from laspy.
Hmm, I don't know why it would fail at that step. I'll try to see if I can reproduce the issue on a test file and get back to you. Just to be sure: what is your platform and python version? Are you running 64bit python?
from laspy.
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
from laspy.
When I run this on a 5.8GB LAS file
Don't do that! 😄
from laspy.
Well....I would love to.....but, we've been tasked to deliver full swath data, i.e. uncut flight lines.
from laspy.
Just teasing. It makes me shake my head when people ask for such large files.
from laspy.
Just an update. I've replaced the read/write mode with a read and write mode (2 separate files) and everything works as it should.
for las_file in sorted(glob.glob(sys.argv[1])):
print("Attempting %s..." % os.path.basename(las_file))
f = laspy.file.File(las_file, mode='r')
h = f.header
inVLRs = []
# OGC Coordinate System WKT
new_vlr = laspy.header.VLR(user_id = "LASF_Projection",
record_id = 2112,
VLR_body = wkt,
description = "OGC Coordinate System WKT")
# Initialize output file for writing data
out_file = os.path.join(os.path.dirname(las_file), os.path.splitext(las_file)[0] + "_HEADER.las")
out_f = laspy.file.File(out_file, mode='w', header = h)
inVLRs.append(new_vlr)
out_f.header.vlrs = inVLRs
# Set WKT Global Encoding Bit
out_f.header.wkt = 1
# Set GPS Global Encoding Bit
out_f.header.gps_time_type = 1
out_f.points = f.points
f.close()
out_f.close()
The only downside of this, however, is that this method is extremely slow on my larger files.
from laspy.
I still haven't had a chance to really dig into this, sorry for the delay.
Since you're not doing anything to the point values, the line:
out_f.points = f.points
is going to be doing a lot of work you don't care about. You should be able to speed this up considerably by reaching into some of the guts of the DataProvider and Writer objects instead of using the helper properties on the File object... but I'd need to spend some time figuring out exactly what that would look like (it's not how the library was originally intended to be used).
Long term, Laspy still isn't going to compete in speed with something like PDAL, because:
- PDAL is written in C++
- You'll only "pay for what you use" in terms of features
If you're going to be processing lots and lots of these, that might be a good investment of time.
from laspy.
Related Issues (20)
- migrate to urllib3 2.0 HOT 2
- Classification in laspy doesn't match with lasinfo HOT 6
- Reclassification of a COPC file HOT 5
- Allow for writing a custom creation date HOT 1
- append_points() function has size limit at approximately 15.1GB? HOT 3
- Ability to read COPC from fsspec sources HOT 2
- LAS/LAZ Header from Remote Source HOT 2
- laszip installation in ubuntu HOT 2
- Append data points to existing LAS file with header that contains extra dimensions. HOT 5
- List of 0 issue HOT 3
- Tests fail: ModuleNotFoundError: No module named 'laspy.cli.core' HOT 11
- laspy installs the 'tests' module globally that would conflict with other packages that accidentally do the same
- The 'rich' dependency is not listed in setup.py HOT 1
- Header update API? HOT 1
- AttributeError: module 'laspy' has no attribute 'read' HOT 7
- Provide pre-built wheels HOT 4
- Conda Forge test failures with Numpy 1.22 + Python 3.9 HOT 6
- Multiple point clouds structure in one LAS file HOT 2
- LAS / LAZ Shuffle in custom attributes in QGIS HOT 7
- Empty string input parse_crs() should result in a None return, but instead a CRSError is thrown in pyproj HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from laspy.