BUG: Unable to append to a layer when using fiona backend about geopandas HOT 3 CLOSED

The error is a bit cryptic, but the problem is that in one of the rows in the second gdf, the "fid" value is the same as one of the rows in the first gdf. Because in GPKG the "fid" is used as primary key, there is an error if a duplicate "fid" is inserted.

If you e.g. rename the "fid" column to "fid_orig", the error disappears.

import geopandas as gpd
from owslib.wfs import WebFeatureService

wfs = WebFeatureService("https://data.3dbag.nl/api/BAG3D/wfs", version="2.0.0")

response1 = wfs.getfeature(
    typename="BAG3D:lod12",
    bbox=(204000.00016165, 486000.00045928, 207000.0001644, 489000.00046109),
    method="Post",
    outputFormat="GML2",
    startindex=0,
    maxfeatures=500,
)

response2 = wfs.getfeature(
    typename="BAG3D:lod12",
    bbox=(204000.00016165, 486000.00045928, 207000.0001644, 489000.00046109),
    method="Post",
    outputFormat="GML2",
    startindex=500,
    maxfeatures=500,
)

gdf1 = gpd.read_file(response1)
gdf2 = gpd.read_file(response2)

print(all(gdf1.columns == gdf2.columns))  # True: Attributes are identical
print(all(gdf1.dtypes == gdf2.dtypes))  # True: Data types are identical

# In the second gdf, there is a row with an fid that already occurs in the first gdf.
# As the fid column is reused as a primary key in GPKG, error.
gdf1 = gdf1.rename(columns={"fid": "fid_orig"})
gdf2 = gdf2.rename(columns={"fid": "fid_orig"})

gdf1.to_file("/tmp/foo.gpkg", driver="GPKG", layer="Building", mode="w", engine="fiona")
gdf2.to_file(
    "/tmp/foo.gpkg", driver="GPKG", layer="Building", mode="a", engine="fiona"
)  # Record does not match collection schema

from geopandas.

For completeness sake, regarding the gdf.set_index(['fid']), two things to note:

• probably just an oversight, but gdf.set_index returns a new dataframe with the index set, so either use gdf.set_index(['fid'], inplace=True) or gdf = gdf.set_index(['fid'])
• when you actually set the index to the "fid" column, I noticed while testing this that the index will still be used for the "fid" column/primary key as long as the index name is "fid". So you need to additionally set the index name to e.g. None, with gdf.index.name = None.

from geopandas.

The error is a bit cryptic, but the problem is that in one of the rows in the second gdf, the "fid" value is the same as one of the rows in the first gdf. Because in GPKG the "fid" is used as primary key, there is an error if a duplicate "fid" is inserted.

If you e.g. rename the "fid" column to "fid_orig", the error disappears.

import geopandas as gpd
from owslib.wfs import WebFeatureService

wfs = WebFeatureService("https://data.3dbag.nl/api/BAG3D/wfs", version="2.0.0")

response1 = wfs.getfeature(
    typename="BAG3D:lod12",
    bbox=(204000.00016165, 486000.00045928, 207000.0001644, 489000.00046109),
    method="Post",
    outputFormat="GML2",
    startindex=0,
    maxfeatures=500,
)

response2 = wfs.getfeature(
    typename="BAG3D:lod12",
    bbox=(204000.00016165, 486000.00045928, 207000.0001644, 489000.00046109),
    method="Post",
    outputFormat="GML2",
    startindex=500,
    maxfeatures=500,
)

gdf1 = gpd.read_file(response1)
gdf2 = gpd.read_file(response2)

print(all(gdf1.columns == gdf2.columns))  # True: Attributes are identical
print(all(gdf1.dtypes == gdf2.dtypes))  # True: Data types are identical

# In the second gdf, there is a row with an fid that already occurs in the first gdf.
# As the fid column is reused as a primary key in GPKG, error.
gdf1 = gdf1.rename(columns={"fid": "fid_orig"})
gdf2 = gdf2.rename(columns={"fid": "fid_orig"})

gdf1.to_file("/tmp/foo.gpkg", driver="GPKG", layer="Building", mode="w", engine="fiona")
gdf2.to_file(
    "/tmp/foo.gpkg", driver="GPKG", layer="Building", mode="a", engine="fiona"
)  # Record does not match collection schema

Thanks, that is indeed the actual problem. I knew fid is treated as primary key of the sqlite database, that is the reason I set it as index. But given the error message it did not occur to me that duplicate primary key is the problem.

from geopandas.