This project aims at providing an ecosystem of intelligent and user-friendly services based on modern smart home devices and other technologies that can help patients adhere better to their medicines and become more self-aware about their side effects and usage.
The project also includes a subscriber-based mechanism which will provide relatives and doctors quick and easy access to patient's medical history in case of emergency.
- A Java/Kotlin based Android App as the frontend
- Google Firebase Realtime Database for backend
- Deep Learning based Optical Character Recognition with OpenCV & Java
- Custom built Alexa Skill in python for Amazon Echo Device and other Amaxon Alexa based devices.
The alexa skill is a custom-hosted skill built with python using ask-sdk-core
and flask-ask-sdk
(to route the requests sent by alexa to our skill using a flask app). The endpoint for the skill is obtained by deploying it with ngrok. The skill makes use of firebase_admin
python module to get data from firebase backend. The interaction model for the alexa skill including all the intents, their utterances, slots, slot types and dialog model can be found in remede-alexa-skill/assets/interaction_model.json
-
Create a new skill on amazon developer console and use the
assets/interaction_model.json
to create the interaction model for the skill using the JSON Editor. -
Create a new python virtual environment, navigate to
remedE-alexa-skill
directory and install all the dependencies withpip install -r requirements.txt
-
Run the
remede-skill.py
file which runs as a flask app on port 5000 on your localhost. -
Download ngrok and run it with protocol set to http and port number 5000.
-
Copy the generated url and paste it in the https endpoint for the skill and then you are ready to test the skill.
-
LaunchRequest Intent: The LaunchRequest Intent invokes when the user says "Launch Remedy Helper". This intent is handled by
LaunchRequestHandler
which returns a JSON response to alexa cloud with a welcome message including a brief description of the skill and how to begin. -
LoginIntent: The LoginIntent invokes when the user asks the skill to "login for user USERNAME" or "provide medical adherence for USERNAME". This intent is handled by
LoginIntentHandler
which fetches the user key from the realtime database using theUtils.get_user_key()
method. This method then stores the user name and key in json file for later reference. The handler also checks for existing medicine details using theUtils.get_user_medicine_details()
method. If medicines are found in the user's data, then the skill returns JSON response to alexa cloud with a welcome message and a question asking permission to set reminders for all the medicines else the skill returns a response with just a welcome message. -
AMAZON.YesIntent: The Amazon.YesIntent is built-in intent which is invoked when the user says "Yes" or its synonyms. This intent is handled by the
CreateMedicineReminderHandler
which makes an API call to the alexa Reminders API to set daily reminder for all the medicines at the times entered by the user. The skill then returns a JSON response with a confirmation message which then spoken by alexa device. -
GetMedDataIntent: The GetMedDataIntent is invoked when the user asks the skill to "give a brief description of MEDANME" or "what is MEDNAME used for?". This intent is handled by the
GetMedDataIntentHandler
which recieves the medicine name asmed_name
slot value from the JSON request recieved by the skill. The handler then calls theUtils.get_med_json_data()
to get the "generic_name" and "description" from theassets/med_data.json
. The skill then prepares a speech output string for alexa and returns a JSON response to alexa cloud which is then spoken by alexa. -
GetSideEffectsIntent: The GetSideEffectsIntent is invoked when the user asks the skill "what can be the side effects fo MEDNAME". This intent is handled by the
GetSideEffectsIntentHandler
which recieves the medicine name asmed_name
slot value from the JSON reques recieved by the skill. The handler then calls theUtils.get_med_json_data()
method to get the "side_effects" from theassets/med_data.json
for that medicine. The skil then prepares a speech output string for alexa using the side effects and returns a JSON response to alexa cloud which is then spoken by alexa. -
GetNextDoseIntent: The GetNextDoseIntent is invoked when the user asks the skill "when is the my next dose of MEDNAME". This intent is handled by the
GetNextDoseIntentHandler
which recieves the medicine name asmed_name
slot and the calls theUtils.get_next_dose()
method. This method then uses theUtils.get_user_med_data()
method to get user medicine details from firebase realtime database. From the recieved medicine data, the method then finds closest dose time to the current time for which the user has not taken the medicine and returns it to the handler. The handler then creates a speech output string and returns the JSON response. -
GetRemainingStockIntent: The GetRemainingStockIntent is invoked when the user asks the skill "which of my medicines are running out" or "which of my medicines are about to finish". This intent is handled by the
GetRemainingStockIntentHandler
which calls theUtils.get_remaining_stock()
method which in turn uses theUtils.get_days_left()
method to calculate the number of days in which a particular medicine will run out. The handler then creates an ouput speech which tells the user the number of days all the medicines will last and if any of the medicines will last for only 3 days or less, then those medicines are suggested for reordering to the user. The skill then uses "Intent Confirmation" usingDelegateDirective
to ask if alexa should place an order for the medicines. Depending on the user's response, ifintent.confirmation_status
of the request is equal toIntentConfirmationStatus.CONFIRMED
, this intent is chained toReorderMedicinesIntentHandler
. Else ifintent.confirmation_status
in the request is equal toIntentConfirmationStatus.DENIED
, this intent is chained toAMAZON.StopIntent
. The skill then returns the appropriate intent as aDelegateDirective
object with the help ofadd_directive()
method in the response. -
ReorderMedicinesIntent: This intent does not contain any sample utterances in the skill interaction model, but is chained to
GetRemainingStockIntent
and is invoked when the user replies in affirmation to the its Intent Confirmation. This intent is handled by theReorderMedicinesIntentHandler
which returns the order placement confirmation message and sends a card to the alexa mobile app of the user showing the deatils of order. (Currently this handler does not actually place any orders due to lack of availablity of any trusted online medicines ordering API) -
AMAZON.StopIntent: This intent is invoked whenever the user says "stop" and is also chained to the
GetRemainingStockHandler
through which it is invoked when the user denies its Intent Confirmation. It is handled by theCancelOrStopIntentHandler
which just returns a goodbye message to the alexa cloud which is then spoken by the alexa device.