singer-io / tap-bigcommerce Goto Github PK

I have confirmed this in two different accounts the Singer integration with Stitch Data will always result in a the error message ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). It looks like it might be a header issue in the request. The tap runs for a short period of time and then is aborted with the above error.

In orders.json, products.external_id (line 444) is defined as an integer. Our source data includes hash strings in this field, causing inserts to fail.

Example row:

products.0.external_id: 0ce0319c-6157-41b3-85a6-47799368664c does not match {'inclusion': 'available', 'type': ['integer', 'null']}

Pull request issued: #13

The products.json schema includes nested records of: variants, images, custom_fields, bulk_pricing_rules and videos. Currently, none of these will be extracted. Per the BigCommerce API docs for Get a product, subresources are not included by default.

The API request needs to change from

GET https://api.bigcommerce.com/stores/:storeNumber/v3/catalog/products

to

GET https://api.bigcommerce.com/stores/:storeNumber/v3/catalog/products?include=:subResources

where :subResources is a comma separated list of the above nested records in the schema.

Each of these products have been replicated to (and exists in) the Products table in the target database, however the last digit of the SKU has been truncated. The truncation has no relationship to the length of the number.

These messages were found in the Stitch logs for an unrelated reason:

products.0: {'id': 2272556, 'order_id': 838319, 'product_id': 13291, 'variant_id': 24994, 'order_address_id': 837151, 'name': '*****', 'name_customer': '', 'name_merchant': '', 'sku': '1042762', ...

products.1: {'id': 2272557, 'order_id': 838319, 'product_id': 20175, 'variant_id': 93230, 'order_address_id': 837151, 'name': '', 'name_customer': '', 'name_merchant': '****', 'sku': '1066132', ...

products.2: {'id': 2272558, 'order_id': 838319, 'product_id': 23155, 'variant_id': 109572, 'order_address_id': 837151, 'name': '', 'name_customer': '', 'name_merchant': '****', 'sku': '54674907382',

products.3: {'id': 2272559, 'order_id': 838319, 'product_id': 14048, 'variant_id': 43536, 'order_address_id': 837151, 'name': '', 'name_customer': ', 'name_merchant': '****', 'sku': '1048692',

products.4: {'id': 2272560, 'order_id': 838319, 'product_id': 19459, 'variant_id': 89933, 'order_address_id': 837151, 'name': '', 'name_customer': '', 'name_merchant': '****', 'sku': '10420905642'

Screenshot of these truncated SKUs in the target database:
https://share.getcloudapp.com/z8u1Gx4n

Certain reports, such as net sales, are hard to create without more detailed returns information. BigCommerce provides a Get All Refunds API.

Certain reports, such as net sales, are hard to generate without shipment details. BigCommerce provides a Get Shipments for Order API.

If BigCommerce provided a Get All Shipments API we could put it in a separate table.

In orders.json, product_options.value(line 430) is defined as an number. Our source data includes strings in this field, causing inserts to fail.

Example row:

products.1.product_options.1.value: Simba (Dalby) does not match {'inclusion': 'available', 'type': ['number', 'null']}

Pull Issued- #18