vcityteam / py3dtilers Goto Github PK
View Code? Open in Web Editor NEWTilers accepting various input formats (OBJ, 3DCity databases, GeoJson, IFC) and producing 3DTiles tilesets.
License: Other
Tilers accepting various input formats (OBJ, 3DCity databases, GeoJson, IFC) and producing 3DTiles tilesets.
License: Other
When creating tiles with batch table hierarchy with the CityTiler (in its current state on Master), the resulting tileset can't be loaded in UD-Viz due to this error:
I can't tell if the Batch Table Hierarchy is broken on the creation side (CityTiler) or on the loading side (UD-Viz). Did this feature even work one day ? or is it unfinished job ?
To create a tileset with BTH:
citygml-tiler --with_BTH
When using the temporal tiler the argparse has two entries for db_config_path:
usage: citygml-tiler-temporal [-h] [--obj [OBJ]] [--loa [LOA]] [--lod1] [--offset [OFFSET [OFFSET ...]]] [--scale [SCALE]] [--crs_in [CRS_IN]]
[--crs_out [CRS_OUT]] [--with_texture] [--with_BTH] [--split_surfaces] [--db_config_path DB_CONFIG_PATH [DB_CONFIG_PATH ...]]
[--time_stamps TIME_STAMPS [TIME_STAMPS ...]] [--temporal_graph TEMPORAL_GRAPH [TEMPORAL_GRAPH ...]]
[db_config_path] [{building,relief,water,bridge}]
A small utility that build a 3DTiles tileset out of data
positional arguments:
db_config_path path to the database configuration file
{building,relief,water,bridge}
identify the object type to seek in the database
optional arguments:
-h, --help show this help message and exit
--obj [OBJ] When defined, also create an .obj model of the features. The flag must be followed by the name of the obj that will be created.
--loa [LOA] Creates a LOA when defined. The LOA is a 3D extrusion of polygons. Objects in the same polygon are merged together. Must be followed by
the path to directory containing the polygons .geojson
--lod1 Creates a LOD1 when defined. The LOD1 is a 3D extrusion of the footprint of each object.
--offset [OFFSET [OFFSET ...]]
Substract an offset to all the vertices.
--scale [SCALE] Scale geometries by the input factor.
--crs_in [CRS_IN] Input projection.
--crs_out [CRS_OUT] Output projection.
--with_texture Adds texture to 3DTiles when defined
--with_BTH Adds a Batch Table Hierarchy when defined
--split_surfaces Keeps the surfaces of the cityObjects split when defined
--db_config_path DB_CONFIG_PATH [DB_CONFIG_PATH ...]
Path(es) to the database configuration file(s)
--time_stamps TIME_STAMPS [TIME_STAMPS ...]
Time stamps (corresponding to each database)
--temporal_graph TEMPORAL_GRAPH [TEMPORAL_GRAPH ...]
GraphML-Json temporal data filename(s)
There is a positional db_config_path
argument from CityTiler and an optional --db_config_path
argument from the temporal tiler.
Because of this I am having some issues declaring the database configurations for the temporal tiler as shown in the documentation because everything declared with the --db_config_path
flag is overwitten by the defaut path from db_config_path
For example:
(venv) diego@diego-IdeaPad-L340-17IRH-Gaming:~/Desktop/py3dtilers$ citygml-tiler-temporal \
--db_config_path ~/Desktop/UD-Reproducibility/Computations/3DTiles/GratteCielTemporal2009-2018/CityTilerDBConfig2009.yml \
~/Desktop/UD-Reproducibility/Computations/3DTiles/GratteCielTemporal2009-2018/CityTilerDBConfig2012.yml \
~/Desktop/UD-Reproducibility/Computations/3DTiles/GratteCielTemporal2009-2018/CityTilerDBConfig2015.yml \
~/Desktop/UD-Reproducibility/Computations/3DTiles/GratteCielTemporal2009-2018/CityTilerDBConfig2018.yml \
--time_stamps 2009 2012 2015 2018 \
--temporal_graph ../Datasets/Villeurbanne_GratteCiel_2009-2018/3-diff/2009-2012/DifferencesAsGraph.json \
../Datasets/Villeurbanne_GratteCiel_2009-2018/3-diff/2012-2015/DifferencesAsGraph.json \
../Datasets/Villeurbanne_GratteCiel_2009-2018/3-diff/2015-2018/DifferencesAsGraph.json \
Mismatching number of databases vs time-stamps:
- databases (configurations): py3dtilers/CityTiler/CityTilerDBConfig.yml
- timestamps: ['2009', '2012', '2015', '2018']
Exiting.
Could we change the CityTemporalTiler flag to something unique such as --db_config_paths
for example ? This should fix the conflict
The name (of the type) "ObjectToTile" doesn't carry the right semantics (which is confusing when reading the code).
Change that type name.
Maybe a clearer name would be TilableObject ?
In fact ObjectToTile is an interface (abstract class) to which some derived class must comply in order to be integrated into a Tile. Because "object" is a very broad term that doesn't add much semantics when used in TilableObject maybe we could rename "ObjectToTole" to "TilableInterface" ?
The geometric error defines when a tile should be displayed by the client (depending on the distance to the camera). This should be configurable (and editable with the TilesetReader
). It would also allow to define when to change between LOD1 or LOA and the most detailed tiles for instance.
The 3D Tiles created with the CityTiler have weird vertices: it seems that all buildings are connected with a vertex, while they are not in the original CityGML:
I reproduced it on different input CityGML and the visualisation is the same in iTowns and Cesium. It happens whether the transformation is ran with or without textures. I reproduce this issue before and and after #71 so it does not seem related to this PR and neither to issue #77. Note that it has no visual impact when not in wireframe mode.
Import alternate example files (license free) from various datasets and data types (GeoJson, Ifc,...)
and integrate them in the test suite.
When the data diversity is concerned, and in order to keep the repository history/footprint as light as possible, consider manually reducing the imported data file.
Yet, when specific tests really require extensive data to be meaningful (e.g when testing memory footprint, algorithm complexity) you might consider commiting heavy files to the repository.
Possible sources of data (check the license and cite the source):
The CityTiler and CityTemporalTiler are already be CI-tested (on Travis). But doing so on the user/developer desktop require a manual pre-installation and set up (with psql) of a testing database.
Limitations:
As mentioned below by @LorenzoMarnat , tests have been provide for the following tilers
Add a command line flag to set the maximum number of features in each FeatureList created by the kd_tree
Currently, material textures specified in a .mtl
file next to the .obj
are lost when convertinf between .obj
(Wavefront) files and 3DTiles.
It would be nice to embed the specified texture coordinates and the textures files in the generated 3DTiles.
GeometryNode
, GeometryTree
and their derived classes LOD1, LOA respectively LodTree, TileHierarchy as well as they py3dTiles counterparts Tile and TileSetThe user should be able to choose the ouput directory of a produced 3DTiles tileset with a flag in command line (like --output tileset_directory
)
If you read the documentation, it explains that how a bounding box works.
Currently, when we try and look at bounding boxes in UD-Viz, it shows that west and south extent at 0 for any created tileset.
Maybe I'm wrong but I think it is because of the notion of half-length presented in the documentation : for example, if i look at the tileset for Limonest, I have the following bounding box :
918977,
2590516,
161,
918977,
0.0,
0.0,
0.0,
2590516,
0.0,
0.0,
0.0,
161
The three first elements should be the bounding box center. The next three shows the x-axis direction and half-length.
If I understand the half-length notion correctly, it means that the bounding box does the following math to obtain the min and max for the x axis, using bounding-box[0] and bounding-box[3] : min = bounding-box[0] - bounding-box[3] and max = bounding-box[0] + bounding-box[3]. It gives us the following bounding box for x : [0,1837954].
If I take the extent for the x-axis for the background, it looks like this [1833742,1842872]. As our tileset is included in the extent's background, I think that the min and max x should be included in this range.
Some attentive reading of usages of GeometryTree
(one must look at derived classes) shows that this is tree of GeometryNodes
(in fact derived classes of GeometryNodes).
Provide some hints for a developer to easily get that understanding, just by reading geometry_tree.py
.
Maybe the comments could be stating thits ?
Or maybe the usage of type hinting would be even clearer ?
The constructor could then be written something like
def __init__(self, root_nodes: list[GeometryNode]):
In the current state, the GeojsonTiler only takes the exterior ring of polygon features to create the 3D extrusion. The tiler should be able to also read the interior rings and create the holes in the polygon.
See GeoJSON spec
The 3DTiles of relief/bridges computed with --with_texture
flag have wrong UVs. The SQL requests to add textures to CityGML relief/bridges may need to be fixed.
When reprojecting geometries with --crs_out
flag while using the LODs flags (--lod1
or --loa
), the tiles corresponding to the LODs have wrong positions and bounding boxes.
Running the city tiler with the --with-texture
option results in a tileset with a lot more tiles. From my tests, from 32x to 64x more tiles are generated with the --with-texture
option for the same input dataset. Do you know where this come from and if it is an intended behavior ?
I think this has an impact on the tileset size and on the visualization performances.
The following concerns the geometry_tree.py
file.
If we consider the graph theory terminology, a (tree) leaf is a node (vertex) without sub-nodes. Now, when given a root (node) of a tree one can navigate to the leafs of the tree.
Hence storing them again in leaf_nodes seems only justified on optimization purposes.
But then leaf_nodes must be kept updated when the tree gets edited, which places some constraint on the code.
Are leaf_nodes and leafs (of the root_nodes) (where leafs is a "function" of a node/root_node) the same thing ?
Document/clarify what lead_nodes are for...
In addition, there is no setter for lead_nodes. Having a setter would be a good place to locate/document the type of leaf_nodes. What are leaf_nodes types ? ObjectsToTile ? GeometryNodes ? Derived classes of GeometryNodes ?
Currently, It seems that there is no way to add a specific material to an object in the B3DM produced. Looking at the specification, it should be possible to do so.
It would be interesting to add such feature.
For example, we could then add a color for each building, using their texture to be a little more realist than a white building.
Maybe pg2b3dm have this feature
Adding textures to the 3DTiles with CityTiler multiply the computation by x50.
Make the break down in sub-task e.g.
The GeoJson Tiler can already produce TileSets than converts/encode some input attribute as a rendering color.
On the client side Ud-Viz can be configured with a rendering color.
Systematize/generalize this separation between data and presentation and, when necessary (that is when some coloring data gets extracted/produced), have the tilers produce separate presentation/styling data. By separate we here mean that the styling data should not be included in the tileset files but in a separate file.
To balance our tiles, we put an arbitrary number of features in each tiles (with a kd-tree). Since some features are way bigger than others (more triangles, heavier texture images, ...), the size of the different tiles and their atlas varies a lot. Some of the textures are way to big to be rendered correctly.
To have more balanced tiles, we should be able to limit the size of a tile depending on different parameters:
crs_in
and crs_out
should handle IDs (crs_in 3946
) and proj4 (crs_in "+proj=latlon"
) with pyproj
We could also define some useful CRS (3946, 4978, etc) as constants
It's maybe a little bit secondary but py3dtilers is not referenced by google: searching py3dtilers in google does not yield any result for this repository on none of the two pages. It yields results for py3dtiles, or for py3dtilers on githubmemory, but not the github repository. This may have a strong impact on its visibility.
We should be able to create compressed 3DTiles (geometries and textures).
Some tests to compress 3DTiles have been made with JS tools (see this doc)
About Pillow image compression. compress_level
could become a parameter of py3dtilers
When running the CityTiler with too many / too heavy textures, the process gets killed by my OS (Ubuntu 20.04). According to the kernel logs, it seems to be due to memory usage:
[267574.260974] Out of memory: Killed process 1606130 (citygml-tiler) total-vm:7268524kB, anon-rss:6205768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12756kB oom_score_adj:0
It happens with the 1st borough of Lyon 2018 vintage for instance.
Do you have the same issue ?
May be some memory is not correctly released or all textures are loaded at once ?
When we create split surfaces (with --split_surfaces
flag or textures), we loose the link between a surface and its parent. Knowing which surfaces are part of the same city object would be useful to create LODs, BTH, measuring the size of a city object, etc..
Merge old documentation with up to date pipeline readme
In particular, please document the junk*
output folder for new users
It would be interesting, when computing a tileset
, to display some data about the tiling process itself as well as the created tileset
characteristics. We distinguish two sub-sets:
info
command (see this issue):
If you look for occurrences of the 100
integer in the code located in the IfcTiler module
(e.g. in ifcObjectGeom.py) chances are they concern conversions from centimeters to
meters (used in 3dTiles). Such conversions:
I've observed that in most cases, the 3D Tiles created is bigger than the input CityGML by a factor of 2 of 3 (only happens for conversions with textures). It seems odd since the CityGML is based on XML and not optimized while 3D Tiles is mainly binary and a bit of json and is meant to be optimized. Sometimes though, the 3D Tiles produced is smaller than the input CityGML but I have not been able to find out why yet. I think this issue may be related to #65 ? but it needs to be investigated more thoroughly.
Currently, there is no documentation if one want to work on the py3dtiles core directly. How do we build py3dtilers with a local py3dtiles repository, if it is possible ?
This is a library and hence should be placed in the requirements section of setup.py
(or library will fail on import). Once the setup.py is fixed, remove the install section
from this Readme.md
The current license was imported from Oslandia
s py3dtiles and is just not correct for the Tilers.
Fix this license
Within IfcTiler/ifcObjectGeom.py
code like
transformer = Transformer.from_crs("EPSG:27562", "EPSG:3946")
hard-wires
Such a rewriting/conversion should be within the Tiler but outside of it. Let the Tiler produce a 3dTiles using the projection system of it's input. Then write a 3dTiles converter utility (CLI) that converts the projection system. This would separate the concerns.
Add a --extract-simplified-graph
option to the temporal tiler to output the simplified graph as JSON before tiling.
it is difficult to use now.
I get the following error when running the CityGMLTiler:
Traceback (most recent call last):
File "/home/vincent/Documents/dev/py3dtilers/venv/bin/citygml-tiler", line 33, in <module>
sys.exit(load_entry_point('py3dtilers', 'console_scripts', 'citygml-tiler')())
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/CityTiler.py", line 181, in main
tileset = city_tiler.from_3dcitydb(cursor, objects_type, split_surfaces)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/CityTiler.py", line 129, in from_3dcitydb
cityobjects = CityMCityObjects.retrieve_objects(cursor, objects_type)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/citym_cityobject.py", line 97, in retrieve_objects
result_objects = objects_type()
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/citym_building.py", line 45, in __init__
super().__init__(objects)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/citym_cityobject.py", line 55, in __init__
super().__init__(cityMCityObjects)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/object_to_tile.py", line 99, in __init__
ObjectsToTile.default_mat = ColorConfig().get_default_color()
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Color/color_config.py", line 19, in __init__
with open(config_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: './py3dtilers/Color/default_config.json'
Wrong filepath in last PR or missing default config ?
When using textures together with LOD1 I hit the following error:
Traceback (most recent call last):
File "/home/vincent/Documents/dev/py3dtilers/venv/bin/citygml-tiler", line 33, in <module>
sys.exit(load_entry_point('py3dtilers', 'console_scripts', 'citygml-tiler')())
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/CityTiler.py", line 183, in main
tileset = city_tiler.from_3dcitydb(cursor, objects_type, split_surfaces)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/CityTiler.py", line 148, in from_3dcitydb
return self.create_tileset_from_geometries(objects_to_tile, extension_name=extension_name)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/tiler.py", line 100, in create_tileset_from_geometries
return create_tileset(objects_to_tile, self.args.lod1, create_loa, self.args.loa, extension_name, self.args.with_texture)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/tileset_creation.py", line 13, in create_tileset
lod_tree = LodTree(objects_to_tile, create_lod1, create_loa, polygons_path, with_texture)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/lod_tree.py", line 24, in __init__
lod1_node = Lod1Node(group.objects_to_tile, 5)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/lod_node.py", line 40, in __init__
extruded_polygon = ExtrudedPolygon(object_to_tile)
File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Common/polygon_extrusion.py", line 38, in __init__
points = hull.exterior.coords[:-1]
AttributeError: 'LineString' object has no attribute 'exterior'
Oslandia last updates integrates the usage of flake8.
Provide the ability the apply a color on the city objects. The color depends on the citygml objectclass
of the object. The cityobjects can be buildings, relief, surfaces (roof, wall, etc), ...
This issue lists the possible improvements of the CityTiler/GeojsonTiler rework.
The TilesetReader should be able to add LODs (i.e add a tile as child of an existing tile) to 3DTiles.
Implement multithreading to speed up the transformation process, e.g. like in the point cloud converter of py3dtiles
Hi
I'm having problems with TilesetMerger. When I launch it, the result is:
Traceback (most recent call last):
File "/home/user/py3dtilers/venv/bin/tileset-merger", line 11, in
load_entry_point('py3dtilers', 'console_scripts', 'tileset-merger')()
File "/home/user/py3dtilers/py3dtilers/TilesetReader/TilesetMerger.py", line 104, in main
tilesets = merger.reader.read_tilesets(paths)
AttributeError: 'TilesetReader' object has no attribute 'read_tilesets'
My Debian specification is:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Linux vm-lxosgeo-test 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux
Python 3.9.2
Could anyone help me?
Thank you
Some comments e.g. in Tilers/IfcTiler/IfcTiler.py are in french. Translate them to english.
I think we could reduce the memory footprint of the CityTiler by working tile by tile instead than with the entire tileset at once and therefore scale up to converting big datasets.
From what I understand, the current main steps of the CityTiler
are:
I propose the following workflow:
What do you think ? Could this proposal not be possible due to other tilers or other processes that I missed ?
The geometries of the 3D Tiles produced with the CityTiler move alone when moving the camera:
This happens for conversions with and without textures and since PR #71 . I reproduce this on iTowns and Cesium and I don't reproduce it with tilesets produced before #71 with the same iTowns/Cesium apps.
It could be useful to add progress information of the transformation process, at least the different steps and in the best case a progress bar (e.g. like for the point cloud transformation tiler).
It could be interesting to allow the generation of multiple LODs, each of them with geometry and texture simplification. The number of LOD could be configured and a default mode could be to generate one LOD per OSM zoom level (or per mapbox ones).
This would increase the rendering performance of the tileset produced.
One example of a model with such LODs: https://3d.strasbourg.eu/CESIUM_OPENDATA/
When considering a TileSet
having a batch table nothing is said about the semantics of such data (what the data represents). This fails to respect the linked open data principle (how so?).
In order to integrate a semantic description of data (some would dubiously call it metadata) present in the batch table, we could use a new specific extension as placeholder.
The work involved by the creation of a new extension is similar to the one realized for thee temporal extension, that is
Then on the client side
Eventually, disseminate that extension :
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.