Towards Equitable Privacy Preserving Machine Learning

With applications beyond the scope of Machine Learning (ML), Federated Learning (FL) is an important framework that respects data privacy and security. It is a ML technique that trains an algorithm across multiple decentralized edge devices or servers holding local data samples, without exch

Project Title

Towards Equitable Privacy Preserving Machine Learning

Project Area of Specialization

Artificial Intelligence

Project Summary

With applications beyond the scope of Machine Learning (ML), Federated Learning (FL) is an important framework that respects data privacy and security. It is a ML technique that trains an algorithm across multiple decentralized edge devices or servers holding local data samples, without exchanging them. The emergence of data protection laws have facilitated the exploration of this new framework and has also made it a very popular term amongst the research community. To facilitate the introduction of FL to a community that is part of a developing region- where not all edge devices have adequate idle resources to train a model- we hope to make FL more friendly to the diverse variety of devices that exist. 

In this project, we introduce a novel solution to dealing with systems heterogeneity among devices, subsequently allowing low-end devices to be included in the FL process. Our approach aims to adaptively serve each mobile device a model per its computational resources. To do this, we perform experiments to explore different techniques that can help make a model less computationally expensive to train and to develop a predictive model that allows us to serve different models to different devices by reducing the computational complexity of a model along different dimensions (model memory consumed, number of operations performed, etc) per the device’s specifications.

We aim to produce results and make deductions based on software simulating the Federated Servers and Clients using LEAF, a benchmarking framework for learning in federated settings. These results will also be verifiable on real devices, which to the best of our knowledge, is the first time a federated context paper discusses this kind of real-world experimentation.

This real-world experimentation is to gain insights as to the factors that affect the performance of a model on actual devices. By training different models on a variety of physical devices, we aim to collect data and insights that can allow us to configure a model at run time (via techniques such as pruning) before sending the model for training to a specific device. These techniques used to configure a model are to be derived from our simulation experiments, that are mentioned in the previous paragraph.  

Project Objectives

The main overarching aim of this project is to be able to adaptively serve models to different devices based on their computational resources. With regard to this main goal, we further divide the project into two main objectives, where both objectives are necessary to achieve our goal. 

The first one of these main objectives focuses on understanding the impact of “adaptive” strategies (example: model pruning, varying batch size, etc) on different Machine Learning models’ efficacy in terms of model’s loss and accuracy in the federated context, and devising strategies that can be used to give us results that are as close as possible to the ideal/optimal case. To this end, we devise and perform different experiments to identify different techniques that would help us serve a wide range of heterogeneous devices. For example, one such technique is model pruning. Model pruning helps reduce the algorithmic cost of the model, as well as the memory space consumed by the model. We prune models based on different metrics (example: weights, gradients, etc), and try determining the optimal choice. We perform other similar experiments to explore other adaptive strategies. We are currently looking into more techniques, whilst performing experiments to get even better results. 

The second main objective is building a predictive model that allows us to adaptively serve a device a ML model, based on its system specifications (eg: RAM, CPU power, etc). To do this, it is required that we understand how different model characteristics affect a model’s training performance on devices with different resources. To achieve this we collect benchmarking and stress testing data from multiple different devices concerning different characteristics (such as the model’s total number of operations, parameters, memory consumed) and analyze this data. This would allow us to develop a predictive model that when given a device’s specifications and a model can adaptively configure the model such that it executes within its allotted round time. 

The achievement of our main aim relies on the achievement of both of the aforementioned main objectives. Insights gained from either of these objectives are valuable and often very helpful in actualizing the other objective, hence complementing each other. The above only very briefly covers some of the sub-objectives that are a part of these two objectives, since doing otherwise is not within the scope of this section.

Project Implementation Method

With the introduction of Federated Learning (FL) came the 'federated context'. Although similar to the Machine Learning (ML) environment, this context differs because of the distributed nature of the framework, which in turn adds a degree of uncertainty to the conclusions drawn from traditional ML research. To ensure that we abided by this federated context, our implementation to this framework was incorporated into a benchmarking framework for learning in federated settings called LEAF, developed at Carnegie Mellon University. 

Designing, Testing, and Comparisons were crucial parts of our journey to pick the best model adaptive technique that fulfills our purpose. To carry out these tasks and to note the outcomes of our results, software simulations were carried out through LEAF, which has codebases for the federated server, federated clients, and all the required algorithms in their original form of Federated Averaging. LEAF has six datasets on which we trained our model: Federated Extended MNIST, Sentiment140, Shakespeare, Celebrity Dataset, Reddit, and a Synthetic Dataset. Running these simulated model training experiments using the LEAF datasets required us to use high-performance servers and GPUs that facilitated the training of models as large as approximately two million parameters. With the textual datasets (like Reddit), we required more processing power because of the experiment's nature.

As common to every project, a literature review of the Federated Learning (FL) technology is one of the first tasks we did when we began this project. Along with learning about what areas had already been explored and what modifications have been suggested, one common observation throughout this task was the lack of the FL model testing on real devices in a real-world environment. 

This lack of work on real devices testing in the federated context and our project's adaptive nature deemed it necessary to test ML models on actual physical devices. To carry out this task, we performed various experiments using the aforementioned benchmarking framework LEAF's models. These on-device experiments included stress testing experiments concerning different model characteristics, experiments to understand the effect of pruning and other adaptive techniques on the model's performance in terms of training time, and how different devices perform with regards to their computational resources. These experiments require us to use multiple devices with varying computational resources to judge fairly whether our modifications to FL behaved as they did in the simulation. We use devices ranging from high-end flagship devices to some of the most resource-constrained low-end devices to ensure our additions to FL make it as heterogeneous as required in the real-world deployment scenario.

Benefits of the Project

Machine Learning (ML) is a handy tool that can be applied to numerous disciplines and problems to make robust educated predictions and automate many tasks. Despite its benefits, the ML framework is troublesome because of its key ingredient - the dataset needed to train a model. What is concerning about it is that for the ML framework to work, the model must be trained on large amounts of data- this means that data is gathered from multiple origins, not necessarily belonging to the developer of the model. This may have data privacy-related concerns, especially after the introduction of laws like the General Data Protection Regulation (GDPR) enforced by the European Union. 

 

Federated Learning (FL) is, simply put, the next generation of ML. The main distinguishing factor between FL and ML is that FL does not require data collection at a single location. Devices, along with the data they contain, contribute to the model's learning despite the physical separation. This way, personal data never has to be shared in its raw form to train a model, and data privacy is maintained. FL also gives the data owner more control over their data. As the awareness about data privacy increases, especially after recent large data breaches and leaks, FL is a necessary direction to explore to continue using ML tools to our benefit.

As always, a new paradigm comes with its own set of problems. Likewise, FL faces incompatibility issues with devices that are limited in terms of computational power, like low-end mobile devices that may not be able to train a model. These devices are a fairly common sight in developing countries, and so, for a framework like FL to succeed in a developing country, FL needs to be made more heterogeneous. Successfully deploying FL in a developing region has many benefits, many of which are revolutionary and can contribute significantly to virtual development. A paradigm like FL holds immense value in Healthcare, where data privacy is of utmost importance. ML's application in this field is currently restricted by limited dataset availability model training because of the legal and ethical requirements to protect patient privacy. With FL, a digital healthcare system can be facilitated that can entirely change the modern medical system. 

By working on our project, we hope to make significant contributions given the context of a developing region. Our primary goal is to bring a country like Pakistan to the same level of ML learning and advancements as the rest of the world. By increasing the heterogeneity of FL, we also hope to expand its use to a developing country's different sectors. We believe that these goals are worth our time because FL itself is revolutionary, with applications exceeding the reach of ML. Further, the emergence of laws like the GDPR has led many countries to follow the same path, and so, by making FL more accessible, we hope to take a step closer to the introduction of laws safeguarding data and user privacy.

Technical Details of Final Deliverable

Throughout this project, we performed various experiments. Some details about the technicalities of these experiments are explained in the Implementation Method section. In this section we further explain these details, alongside the tools and frameworks used in implementing these experiments.

To choose and analyze the closest to the optimal adaptive model serving techniques in the context of Federated Learning, we performed a series of simulations using the LEAF benchmarking framework. This framework was developed as part of a research paper and is widely used by other research works related to Federated Learning. This framework is implemented in Python and uses TensorFlow version 1.13.1. It has the implementation of 6 datasets and their respective models. This includes two image datasets (Celebrity and Federated Extended MNIST) both having their own Convolutional Neural Networks, three text-based datasets (Shakespeare, Reddit, Sentimental140) with each having its own Long Short Term Memory (LSTM) model, and the Sentiment140 dataset having an additional logistic regression model, and a Synthetic dataset that is generated using a random probability distribution and trains using a logistic regression model. Furthermore, the framework provides flexibility to deal with other parameters such as the distribution of data between devices, among other things. Taking advantage of the flexibility of this framework, we added our adaptive serving implementation to the framework. This allowed us to compare our adaptive serving approach with other existing techniques, in terms of different aforementioned models’ loss, and accuracies. It also allowed us to compare, and improve upon our adaptive serving techniques.

For on-device training and understanding the factors that affect the performance of a model’s training on an actual physical device, as well as how different models perform on different devices, we first chose to implement LEAF’s models in a different framework. Our end goal is to be able to adaptively serve a model given a device’s specifications, and model characteristics. We choose to implement LEAF’s models for this task for multiple reasons. Firstly, many other research works in the area of Federated Learning have made use of this framework. Secondly, it would be easier to compare our simulation results with our on-device testing/benchmarking results. Lastly, the framework provides various kinds of datasets with different distributions of data, which is crucial for our task. For these experiments, we use PySyft, PyGrid, and KotilinSyft frameworks, which work together to provide Federated Learning on Android mobile devices. These frameworks are specifically designed for Federated Learning and are being used by other projects - both in the industry, and academia. For our models, we use PyTorch, which is another popular framework for Machine, and Deep Learning models.

Final Deliverable of the Project

Software System

Core Industry

IT

Other Industries

Security

Core Technology

Artificial Intelligence(AI)

Other Technologies

Others

Sustainable Development Goals

Industry, Innovation and Infrastructure, Partnerships to achieve the Goal

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com