An Android App for Detecting Sleep and Pausing Media
Project Overview
This project was developed as the capstone for my Master’s degree in Software Engineering at the University of Wisconsin La Crosse, under the guidance of Professor Mao Zheng. The Android app is designed to detect when users fall asleep and automatically pause media playback. This functionality aims to help users avoid missing out on podcasts, audiobooks, or music when they unintentionally fall asleep.
Introduction
The purpose of this project to develop an Android app that will detect when a user has fallen asleep and pause any media (music, podcast, video, etc.) that is playing. To do this, the phone must be placed near the user’s head on their bed. Then using a combination of the phone’s accelerometer, light sensor, proximity sensor, microphone and Android’s Sleep API, this app will detect when the user has fallen asleep.
Stages of Sleep
The app recognizes four stages of sleep: Light, Deep, REM, and Awake. It focuses on pausing media when the user first enters light sleep and uses a smart alarm feature to wake the user during light sleep, which is known to be the optimal stage for feeling refreshed upon waking.
Design
The Design of the app is quite simple. Users can simply start the sleep tracking feature of the app. Look into their past sleep data, or go into their settings to make adjustments.
The phone detects when the user is asleep by using the sensors on the smartphone. This is done in 2 ways depending on the version of android the phone is running. Phones running Android API 29 and higher have access to Sleep API. When a phone is detected to have sleep API, we will use this preferred method of detecting sleep. If a phone on a version of android below API 29 is using the app, then the phone will use an AI trained model to detect if the user is asleep or not. To detect if the user is asleep the phone must be placed on the bed near the user’s head, then using various sensors on the phone the app will feed that data to sleep API or the AI model, then to decide if the user is asleep every so often.
Detecting Sleep
The app utilizes smartphone sensors and the Android Sleep API for devices running Android 10 and above. For older versions, an AI model trained with TensorFlow and Keras is used to detect sleep. The sleep detection requires the phone to be placed on the bed near the user's head, leveraging the phone's accelerometer, light sensor, and proximity sensor to gather data.
Implementation
The implementation uses TensorFlow and Keras for training the AI model. The sleep detection relies on data from various sensors to determine the user's sleep state. During development, data was collected through Google Firebase, which was essential for training and refining the sleep detection algorithms.
Sensors and Data Collection
The app collects data from motion sensors, light sensors, and proximity sensors. A dedicated app was developed for data collection, storing the information in Google Firebase. A long with this data the SleepAPI sleep confidence level was collected. This data was then used to train the Random Forest AI.
AI and Algorithms
Due the limitations of Sleep API only being available on an Android SDK 29 and later, as of November 2021 over 69% of android devices are running on SDK 28 or earlier. To allow this app to run on older devices I also trained an AI, using data gathered with Sleep API and other sensors on the phone. To determine the best AI to use for this situation I decided to test a bunch of them on data I had already collected. I split my data into 70% testing, 20% cross validation, and 10% testing data, I used 28 thousand rows of data collected on myself, using a Google Pixel 4a’s accelerometer, light sensor, proximity sensor and microphone; it on data collected from Sleep API’s sleep confidence level. The models I tested were a Support Vector Machine (SVM), Naïve Bayes classification algorithm, K-nearest neighbor algorithm, Logic Regression, Gradient Boosting, decision tree, and random forest. I tested all of these for their accuracy at predicting Sleep API’s sleep confidence level. Through this testing I found that K-nearest neighbor, Gradient Boosting, Decision tree, and Random Forest where the most accurate of the models that I tested. I also tested the processing speedo of each algorithm on a large amount of data. Due to its accuracy and speed, the Random Forest algorithm was chosen to be used.
Database Design
Data is stored locally on the phone in a SQLite database. Accessing this database is done using Android’s Room library. Room is used because it makes it incredibly easy to access the SQLite database. Room runs at a much higher level than the SQLite database’s API’s so we have to spend less time writing queries and worrying about the intricacies of the database. Room allows us to simply connect the database and send and receive the data we want in a quick and efficient manner.
The database’s layout is simple. There are no complex relations between tables that we would need to write advanced queries for so simply accessing the database through Room will be more than efficient enough for the design of this app. The most important table is the Night table that stores all the sleep data. We are storing the time the user the user started tracking, the time they fell asleep, and the time they woke up in the morning. This allows the app to easily calculate how much sleep the user got on a given night and let the user view this data. We also store alarms in the database, simply storing what day the alarm is for, if it is a smart alarm or not, and what time the alarm should go off. If the alarm is not a smart alarm, the smartTimeStart and smartTimeEnd fields will be empty. In the case of a smart alarm the time field will be empty, the smartTimeStart is the earliest time the user would like to be woken up and the smartTimeEnd is the latest time they would like to be woken up. Finally, we store the sensitivity settings in the database as well.
Controlling Media
Once the app determines that the user is asleep, the app will then pause the user’s media. To achieve this, we use androids Audio Manager. With the audio manager we can request focus of the phone’s audio, this pauses any other media playing on the android devices, from video to podcasts. This also allows us to relinquish audio focus allowing the previously playing media to resume.
Determining When to Pause Media
The decision to pause the user’s media is made based off the sleep confidence level, light level, and the motion level reported by either Sleep API or the Random Forest AI with the light sensor and the acceleration variance formula used for the light and motion levels. If a large motion level is detected the app will wait another 10 minutes before pausing the media even if the sleep confidence level is above the threshold before pausing the media. We are assuming if a large movement is detected then the user is likely not asleep yet, as large movements are common during light stages of sleep. The motion level needed to allow the app to pause the media is determined by the sensitivity settings of the app. A maximum light level will also need to be reached to determine that the user is asleep. This allows us to differentiate between a user who is asleep and one who is still awake reading a book or watching a movie but not moving at all. The light level needed to pause the media is similarly determined by the user’s sensitivity settings allowing a user who sleeps in a lit room to still use the app. Finally, the most important factor in pausing the media is the sleep confidence level. The sleep confidence level is a value between 0 and 100 provided by Sleep API, the lower the value the more likely that the user is asleep. Once the required sleep confidence level is reached, and the users motion level and light levels are not too high, the app will pause the media.
Past Data and Alarms
Additionally, the app stores users sleep data, and allows them to set alarms. Users can look back at their previous sleep schedules and see how much REM, deep, and light sleep they got. Additionally users can set alarms in the app. Allowing users to have all their sleep related tracking and alarms in one spot.
Security and Permissions
All data is stored locally on the device, with no personal data attached to ensure privacy. The app relies on Android’s security features and app sandboxing for data protection. The necessary permissions include Activity Recognition and Set Exact Alarms to function correctly.