aws-samples / amazon-sagemaker-generativeai Goto Github PK

dataset.map() function maps a function to apply it to every record in the dataset. You do not apply a function here, you pass sample['text'] and consequently all your tokens you use for fine-tuning are exactly the same

lm_train_dataset = train_dataset.map(
    lambda sample: tokenizer(sample["text"]), batched=True, batch_size=32, remove_columns=list(train_dataset.features)
)


lm_test_dataset = test_dataset.map(
    lambda sample: tokenizer(sample["text"]), batched=True, remove_columns=list(test_dataset.features)
)

Hello,

In your example notebook studio-notebook-fine-tuning/falcon-40b-qlora-finetune-summarize.ipynb, once the model is trained, how do you save and load the model for inference in the future?

I used trainer.save_model('local path'), but it doesn't save the config.json file, saving the config file manually doesn't help either. The code below has an error

trainer.save_model('local path')
model.config.to_json_file('local path')
generator = pipeline('text-generation', model='local path')
generator(prompt)
tokenizer = AutoTokenizer.from_pretrained('local path')

Any help is appreciated!

When running the "Upload to model Artifacts" section of "/train-and-deploy-gptj/deployment/deploy_gptj_DJLModel.ipynb' I get the following error when trying to create the 'model_config':

RuntimeError Traceback (most recent call last)
in
40 with smart_open(load_path, "rb") as f:
41 buffer = io.BytesIO(f.read())
---> 42 model.load_state_dict(torch.load(buffer))
43
44 tokenizer = AutoTokenizer.from_pretrained(model_id)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
1602
1603 if len(error_msgs) > 0:
-> 1604 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
1605 self.class.name, "\n\t".join(error_msgs)))
1606 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for GPTJForCausalLM:
Unexpected key(s) in state_dict: "transformer.h.0.attn.bias", "transformer.h.0.attn.masked_bias", "transformer.h.1.attn.bias", "transformer.h.1.attn.masked_bias", "transformer.h.2.attn.bias", "transformer.h.2.attn.masked_bias", "transformer.h.3.attn.bias", "transformer.h.3.attn.masked_bias", "transformer.h.4.attn.bias", "transformer.h.4.attn.masked_bias", "transformer.h.5.attn.bias", "transformer.h.5.attn.masked_bias", "transformer.h.6.attn.bias", "transformer.h.6.attn.masked_bias", "transformer.h.7.attn.bias", "transformer.h.7.attn.masked_bias", "transformer.h.8.attn.bias", "transformer.h.8.attn.masked_bias", "transformer.h.9.attn.bias", "transformer.h.9.attn.masked_bias", "transformer.h.10.attn.bias", "transformer.h.10.attn.masked_bias", "transformer.h.11.attn.bias", "transformer.h.11.attn.masked_bias", "transformer.h.12.attn.bias", "transformer.h.12.attn.masked_bias", "transformer.h.13.attn.bias", "transformer.h.13.attn.masked_bias", "transformer.h.14.attn.bias", "transformer.h.14.attn.masked_bias", "transformer.h.15.attn.bias", "transformer.h.15.attn.masked_bias", "transformer.h.16.attn.bias", "transformer.h.16.attn.masked_bias", "transformer.h.17.attn.bias", "transformer.h.17.attn.masked_bias", "transformer.h.18.attn.bias", "transformer.h.18.attn.masked_bias", "transformer.h.19.attn.bias", "transformer.h.19.attn.masked_bias", "transformer.h.20.attn.bias", "transformer.h.20.attn.masked_bias", "transformer.h.21.attn.bias", "transformer.h.21.attn.masked_bias", "transformer.h.22.attn.bias", "transformer.h.22.attn.masked_bias", "transformer.h.23.attn.bias", "transformer.h.23.attn.masked_bias", "transformer.h.24.attn.bias", "transformer.h.24.attn.masked_bias", "transformer.h.25.attn.bias", "transformer.h.25.attn.masked_bias", "transformer.h.26.attn.bias", "transformer.h.26.attn.masked_bias", "transformer.h.27.attn.bias", "transformer.h.27.attn.masked_bias", "transformer.h.28.attn.bias", "transformer.h.28.attn.masked_bias", "transformer.h.29.attn.bias", "transformer.h.29.attn.masked_bias", "transformer.h.30.attn.bias", "transformer.h.30.attn.masked_bias", "transformer.h.31.attn.bias", "transformer.h.31.attn.masked_bias", "transformer.h.32.attn.bias", "transformer.h.32.attn.masked_bias", "transformer.h.33.attn.bias", "transformer.h.33.attn.masked_bias", "transformer.h.34.attn.bias", "transformer.h.34.attn.masked_bias", "transformer.h.35.attn.bias", "transformer.h.35.attn.masked_bias", "transformer.h.36.attn.bias", "transformer.h.36.attn.masked_bias", "transformer.h.37.attn.bias", "transformer.h.37.attn.masked_bias", "transformer.h.38.attn.bias", "transformer.h.38.attn.masked_bias", "transformer.h.39.attn.bias", "transformer.h.39.attn.masked_bias", "transformer.h.40.attn.bias", "transformer.h.40.attn.masked_bias", "transformer.h.41.attn.bias", "transformer.h.41.attn.masked_bias", "transformer.h.42.attn.bias", "transformer.h.42.attn.masked_bias", "transformer.h.43.attn.bias", "transformer.h.43.attn.masked_bias", "transformer.h.44.attn.bias", "transformer.h.44.attn.masked_bias", "transformer.h.45.attn.bias", "transformer.h.45.attn.masked_bias", "transformer.h.46.attn.bias", "transformer.h.46.attn.masked_bias", "transformer.h.47.attn.bias", "transformer.h.47.attn.masked_bias".

How can this be fixed?

We run Generative AI on SageMaker workshop and unfortunately, Lab 1 & 2 created loads of issues due to some recent changes were pushed to change from Flan T5 to Falcon 7b.
Lab1: In text-to-text the response from Falcon on customer service example didn’t work as well as Flan without users starting to adjust some configuration parameters (not as good as Flan out of the box)
Lab2: lab 2 changed model under the hood including changes to the streamlit application, while flan is still referenced in workshop. I think it is a good idea to add another model, but that should be an option not mess up the existing model. Consistency is the biggest issue here.
please can you fix these ASAP?

Error :
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (422) from primary with message "Failed to deserialize the JSON body into the target type: inputs: invalid type: sequence, expected a string at line 1 column 11".

I'm getting an above error for "Multilingual chatbot using E5 multilingual embeddings model and Meta llama2 7-b chat LLM in Sagemaker Jumpstart" notebook

I'm getting an error at section : [Create payload function. Based on the language of the input question ( English, Spanish, or Italian) we are using Amazon Comprehend to idenitfy the language so the prompt for llama2 can be crafted dynamically to ask it to respond in the same language as the question)]

Unable to get where exactly wrong is happening , even after checking payload , it seems fine as below :
{'inputs': [[{'role': 'system',
'content': 'Your are a friendly multilingual Assistant chatbot. You can speak in English, Spanish and Italian. Answer the following QUESTION based only on the CONTEXT given.Respond using the language of the QUESTION and the CONTEXT.If you do not know the answer and the CONTEXT doesn't \n contain the answer truthfully say "I don't know".\n\n CONTEXT:\n Online features are used in applications required to make real-time predictions. Online features are served from a high-throughput repository with single-digit millisecond latency for fast predictions.\nOffline features are used for training because you need access to very large volumes over a long period of time. These features are served from a high-throughput, high-bandwidth repository.\n\n\n ANSWER:\n '},
{'role': 'user',
'content': 'Respond in English.what are online features?'}]],
'parameters': {'max_new_tokens': 1024,
'top_p': 0.9,
'temperature': 0.1,
'return_full_text': False}}

Any clue ???