A new approach leads to more accurate data classification with less training data and more privacy.
by Stuti Gupta
Sensors surround us in our daily lives in forms such as smartphones, wearable devices, cameras, and wireless routers. Words and other information flutter around us and into these devices. This data is used for applications such as environment monitoring, healthcare, education, and workplace management. A new study by Yujing Chen improves data classification performance through federated multi-task learning, making use of the local storage of each task’s data. To break it down, this research combines federated learning, multi-task learning, and attention mechanisms.
Federated Learning
Federated learning allows data to be stored locally without the need to be stored in the cloud. This works by each individual device downloading the current model then improving that current model based on the needed changes on each device. The changes from each individual device are stored only on that device and not the cloud, then summed or aggregated, and averaged to become the new current model. From this the cycle starts all over again, all the while ensuring privacy of the devices. Challenges to overcome that are introduced along with this innovative machine learning approach include bandwidth, meaning wireless communication, and unreliability of the end devices.
Multi-task Learning
Multi-task learning (MTL) determines the outcome of a task after taking into account related tasks as not to overfit the interested task; in simpler terms, if the outcome of task A takes into account the representation of related task B, a more inclusive result of task A can be produced. In a great example given by Sebastian Ruder’s article, An Overview of Multi-Task Learning in Deep Neural Networks, the movie The Karate Kid presents a way to understand MTL. Sensei Mr. Miyagi trains Daniel to become the karate kid through tasks including cleaning the car and sanding the floor, which at the time do not seem useful, but are actually related tasks giving him the skill to become the karate champion he becomes.
MTL can be viewed as a type of inductive transfer, helping the model select some hypotheses over others in the form of inductive bias. Two methods exist for MTL: hard parameter sharing and soft parameter sharing. Starting with hard parameter sharing, this method has a single hidden shared layer with many output task layers, which decreases the chance of overfitting. By contrast, in soft parameter sharing each task has its own hidden layer and own output layer. Within the hidden layers there is sharing between the tasks. Using unrelated tasks to determine the outcome of the original task can actually hurt the reliability of the outcome. For that reason, another unique approach includes finding which tasks are related enough to share. One important thing to remember is that MTL is not binary.
Attention Mechanisms
Finally, this study aims to improve performance for data from several sensors using attention mechanisms addressed by Recurrent Neural Networks (RNN). Being a big part of deep learning, RNN is able to work with sensor data such as audio, text, and video. This network stores memory in the form of an array, and receives information using attention, focusing on a subset of the information. In an example given by An Introduction to Recurrent Neural Networks, as you read this sentence you are able to process each word’s meaning based on previous words. Unlike most networks, RNN is a special type of network that uses loops and can do this; for example, given a string of characters this network will be able to predict the next character of the sequence.
Integrating these methods and concepts allowed Chen and her colleagues to build a learning model with data from several tasks, each with several sensors, while protecting each task’s privacy and using attention mechanisms to improve model classification and performance for input data from several sensors. In the future she also hopes to experiment with other concepts such as asynchronous learning multiple tasks while protecting local data privacy; but for now this study is an incredible stride for the environment, health, education, and work. The world of words as voices and texts surrounds us once again.