Giter VIP home page Giter VIP logo

Comments (5)

farzeni avatar farzeni commented on June 3, 2024 2

Hi @mike-luabase ,

In the end, I have opted to use a custom catalog because stream maps have limitations with nested fields:
https://sdk.meltano.com/en/latest/stream_maps.html#no-nested-property-declarations-or-removals

I've added a modified tap-shopify.catalog.json file under the extract folder and set it up in the plugin configuration.

plugins:
  extractors:
    - name: tap-shopify
      variant: matatika
      catalog: extract/tap-shopify.catalog.json
      pip_url: git+https://github.com/Matatika/tap-shopify.git
      config:
        store: ....
        start_date: ...
        is_plus_account: false

from tap-shopify.

farzeni avatar farzeni commented on June 3, 2024 1

Thank you for your quick response.

I appreciate your suggestions and will try using a custom catalog and stream maps to handle this issue.

Looking forward to the permanent fix

Thanks again

from tap-shopify.

ReubenFrankel avatar ReubenFrankel commented on June 3, 2024

Hi @farzeni, thanks for raising this. We've had a couple of reports of null data in fields causing the tap to break (e.g. #7). We've been thinking about doing a refactor to map all fields as nullable by default (excluding primary keys), but for now you may be able to work around it by passing the tap a custom catalog or using stream maps to coalesce None to an empty string ('').

We'll keep this issue open until we have a fix, so you should see updates here when we create a PR to address all these.

from tap-shopify.

mike-luabase avatar mike-luabase commented on June 3, 2024

@farzeni @ReubenFrankel did you end up fixing this with a stream map? It wasn't clear to me how to map nested field. I was trying:

plugins:
  extractors:
  - name: tap-shopify
    variant: matatika
    pip_url: git+https://github.com/Matatika/tap-shopify.git
    config:
      stream_maps:
        abandoned_checkouts:
          phone: __NULL__

but getting:

jsonschema.exceptions.ValidationError: None is not of type 'string' cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.463820Z [info     ]                                cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.463907Z [info     ] Failed validating 'type' in schema['properties']['billing_address']['properties']['phone']: cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.463999Z [info     ]     {'type': 'string'}         cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.464083Z [info     ]                                cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.464169Z [info     ] On instance['billing_address']['phone']: cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-08-24T02:16:15.464293Z [info     ]     None

from tap-shopify.

ReubenFrankel avatar ReubenFrankel commented on June 3, 2024

Various schema issues addressed in v0.2.0

from tap-shopify.

Related Issues (8)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.