Giter VIP home page Giter VIP logo

Comments (8)

tywalch avatar tywalch commented on May 24, 2024 2

@calebwilson706 I took the steps to reproduce, installing electrodb at the version you specified (npm install [email protected]) and made a file with your model above, but was unable to recreate the exception you shared. In looking at the error stack and its associated code, I would expect this to be throwing when your model is loaded.

In the cloudwatch logs you shared it looks like the indexes are finding async functions for forEachAsync, mapAsync, and removeNull, which suggest some of the values on the model might not be what is expected. I assume these logs are the result of printing the i, accessPattern and index values found at this line?

I completely understand the need to sanitize models, though I think some of the dynamic implementation of the models might be at play here. The playground uses the same version as you have reported, so if you are able to recreate the issue there (even with "fake" attributes) that would be very helpful to help diagnose further.


Regarding the ElectroValidationError you posed, would you be able to share the attribute definition used for this attribute? Additionally, here is a link to the code associated with identifying/validating the type of data received.

I cannot speak for the type of data being passed (the value is not logged or captured in the error to avoid leaking private data) but it does seem to be working as designed: the value received was not of the expected type. For example, a function could be flagged as unknown given the above rules.

The best course of action would be to add logging or further validation near the use of this put.

I'll leave this ticket open to maintain communication with you 👍

from electrodb.

tywalch avatar tywalch commented on May 24, 2024

Hi @calebwilson706

Thank you for reaching out! Does your model throw when using the electrodb playground? If so, would you be able to provide a playground like that demonstrates the issue? If not, would you be able to provide the version you are using, and a model that causes this exception?

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

It doesn't throw out when using the playground, I'll get you my model and version now, thanks!

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

We are on version 1.8.4

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

Our model is as follows (i have taken out any attribute which is not used for our indexes)

export const buildPartitionKey = (docmailMailingGuid: string): string => `${MAILING_PREFIX}${docmailMailingGuid}`;
export const buildSortKey = (): string => `${MAILING_PREFIX}`;
export const buildGsi1PartitionKey = (policyId: string): string => `${POLICY_PREFIX}${policyId}`;
export const buildGsi1SortKey = (effectiveDate: string): string => `${EFFECTIVE_DATE_PREFIX}${effectiveDate}`;
export const buildGsi2PartitionKey = (requestedAt: string): string => `${REQUESTED_AT_PREFIX}${requestedAt}`;
export const buildGsi2SortKey = (): string => `${REQUESTED_AT_PREFIX}`;

export const documentPrintRequest = new Entity(
  {
    model: {
      entity: "document print requests",
      version: "1",
      service: "docmail"
    },
    attributes: {
      partitionKey: {
        type: "string",
        required: true,
        readOnly: true
      },
      sortKey: {
        type: "string",
        required: true,
        readOnly: true,
        default: MAILING_PREFIX
      },
      gsi1PartitionKey: {
        type: "string",
        required: true,
        readOnly: true
      },
      gsi1SortKey: {
        type: "string",
        required: true,
        readOnly: true,
      },
      gsi2PartitionKey: {
        type: "string",
        required: true,
        readOnly: true
      },
      gsi2SortKey: {
        type: "string",
        required: true,
        readOnly: true,
        default: REQUESTED_AT_PREFIX
      },
      policyId: {
        type: "string",
        required: true
      },
      effectiveDate: {
        type: "string",
        required: true
      },
      docmailMailingGuid: {
        type: "string",
        required: true
      },
      requestedAt: {
        type: "string",
        required: true
      },
    },
    indexes: {
      byDocmailMailingGuid: {
        pk: {
          field: "partitionKey",
          composite: ["docmailMailingGuid"],
          casing: "none",
          template: buildPartitionKey("${docmailMailingGuid}")
        },
        sk: {
          field: "sortKey",
          composite: [],
          casing: "none",
          template: buildSortKey()
        }
      },
      byPolicyIdAndEffectiveDate: {
        index: "gsi1",
        pk: {
          field: "gsi1PartitionKey",
          composite: ["policyId"],
          template: buildGsi1PartitionKey("${policyId}"),
          casing: "none"
        },
        sk: {
          field: "gsi1SortKey",
          composite: ["effectiveDate"],
          template: buildGsi1SortKey("${effectiveDate}"),
          casing: "none"
        }
      },
      byRequestedAtDate: {
        index: "gsi2",
        pk: {
          field: "gsi2PartitionKey",
          composite: ["requestedAt"],
          template: buildGsi2PartitionKey("${requestedAt}"),
          casing: "none"
        },
        sk: {
          field: "gsi2SortKey",
          composite: [],
          template: buildGsi2SortKey(),
          casing: "none"
        }
      }
    }
  },
  {
    table: documentPrintRequestTableName,
    client: documentClient
  }
);```

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

Im also having an issue when trying to complete a put operation with an attribute which is a list i'm getting a validation error

error {
    "errorType": "ElectroValidationError",
    "errorMessage": "Invalid value type at entity path \"documentsRequested[*]. Received value of type \"unknown\", expected value of type \"object\", Invalid value type at entity path \"documentsRequested[*]. Received value of type \"unknown\", expected value of type \"object\", Invalid value type at entity path \"documentsRequested[*]. Received value of type \"unknown\", expected value of type \"object\" - For more detail on this error reference: https://github.com/tywalch/electrodb#invalid-attribute",
}

once again this doesn't occur in the electrodb playground, the only differences between the playground and our code being

1.) ours is running in an actual lambda
2.) our code is split across a few files

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

Hello, sorry for the delay - we have found this source of our issue. We have some helper methods which modify the array prototype to give us some extension method helpers

interface Array<T> {
  forEachAsync<T>(this: Array<T>, action: (value: T) => Promise<void>): Promise<void>;
  mapAsync<T, U>(this: Array<T>, map: (value: T) => Promise<U>): Promise<U[]>;
  removeNull(this: Array<T>): Array<NonNullable<T>>;
}

Array.prototype.forEachAsync = async function forEachAsync<T>(this: Array<T>, action: (value: T) => Promise<void>): Promise<void> {
  for (const element of this) {
    await action(element);
  }
};

Array.prototype.mapAsync = async function mapAsync<T, U>(this: Array<T>, mapFunction: (value: T) => Promise<U>): Promise<U[]> {
  const result: U[] = [];
  await this.forEachAsync(async (x) => {
    result.push(await mapFunction(x));
  });
  return result;
};

Array.prototype.removeNull = function removeNull<T>(this: Array<T>): Array<NonNullable<T>> {
  return this.filter((value): value is NonNullable<T> => value != null);
};

because there is a for in loop in line 2193 of entity.js these are being found when looping. Changing this to a forEach would most likely solve the issue. I am imagining something similar is happening with the validation error discussed above also.

from the mozilla docs

Array indexes are just enumerable properties with integer names and are otherwise identical to general object properties. The for...in loop will traverse all integer keys before traversing other keys, and in strictly increasing order, making the behavior of for...in close to normal array iteration. However, the for...in loop will return all enumerable properties, including those with non–integer names and those that are inherited. Unlike for...of, for...in uses property enumeration instead of the array's iterator. In sparse arrays, for...of will visit the empty slots, but for...in will not.

It is better to use a for loop with a numeric index, Array.prototype.forEach(), or the for...of loop, because they will return the index as a number instead of a string, and also avoid non-index properties.

Thanks!

from electrodb.

calebwilson706 avatar calebwilson706 commented on May 24, 2024

i can update anywhere the for in loop is used if you would like, i'm not seeing any guide on contributing

from electrodb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.