The following projects are available for Master and Bachelor students. They are performed in close collaboration with an experienced member of the lab. Apply for a project by sending an email to the contact mentioned for the project.
You may also suggest new projects, ideally close enough to our ongoing, or previously completed projects. In that case, you will have to convince Anne-Marie Kermarrec that it is worthwhile, of reasonable scope, and that someone in the lab can mentor you!
Projects available for Spring 2025.
Scalable and Distributed LoRA Adapter Selection and Serving
Master’s Thesis or MSc semester project
Contact: Martijn de Vos (martijn.devos@epfl.ch)
The fine-tuning of LLMs to specialize its performance on a particular task is essential for unlocking their
full capabilities. It has become an important paradigm in the field of LLMs. LoRA has recently gained much attention [1]. LoRA introduces trainable parameters, or adapters, that interact with the pre-existing ones through low-rank matrices, allowing the model to adapt to new tasks without retraining it fully. It is based on the assumption that the differences between the pre-trained and fine-tuned model exhibit low-rank properties. LoRA keeps the pre-trained model parameters frozen and uses auxiliary low-rank matrices that are randomly initialized.
In some settings, there may be many LoRA adapters for diverse types of downstream tasks, such as text translation or summarization. Our goal is to build a disaggregated, distributed system architecture for fetching and utilizing the right adapters for a particular query input while minimizing the end-to-end inference latency. Related work can be found in [2] and [3]. The main goal of this project is to design, implement and evaluate (parts of) a scalable and distributed system that efficiently selects and serves LoRA adapters based on specific query inputs, aiming to reduce the end-to-end inference latency for various downstream tasks. Previous expertise with distributed ML systems is highly recommended. Potential research questions:
- How can a disaggregated system architecture speed up inference with LoRA and multiple adapters?
- How can we build a distributed adapter routing and selection mechanism?
- How can we adapt our system architecture to support batched requests?
[1] Hu, Edward J., et al. “Lora: Low-rank adaptation of large language models.” arXiv preprint arXiv:2106.09685 (2021).
[2] Sheng, Ying, et al. “S-lora: Serving thousands of concurrent lora adapters.” arXiv preprint arXiv:2311.03285 (2023).
[3] Zhao, Ziyu, et al. “LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild.” arXiv preprint arXiv:2402.09997 (2024).
Boosting Decentralized Learning with Bandwidth Pooling
Master’s Thesis or MSc semester project
Contact: Martijn de Vos (martijn.devos@epfl.ch)
Decentralized Learning (DL) is a relatively new class of ML algorithms where the learning process takes place on a network of interconnected devices with no central server that supervises the training. While DL has initially been applied within data centers to improve the efficiency and scalability of large-scale ML tasks in homogeneous environments, it is increasingly being used to train ML models between end-user devices in heterogeneous environments. With DL, each device in the network independently updates its own model based on the data available locally and directly shares the updated model with other clients. Then, each client periodically aggregates received models. DL uses a peer-to-peer communication topology that prescribes which clients share their model with which other clients.
As DL moves beyond homogeneous data centers to large-scale, heterogeneous end-user environments such as smartphone networks, the variability in computational and communication resources becomes a substantial issue [1]. The discrepancies in bandwidth among nodes can lead to inefficiencies in model dissemination, which is critical to the DL process and directly affects the duration of a round. This project aims to design and evaluate a bandwidth pooling strategy where nodes with surplus bandwidth can assist other nodes in the dissemination of their models. The main research question we seek to address is: “How can nodes in DL effectively utilize the surplus bandwidth of other nodes to accelerate model dissemination?”. Some related work on this topic can be found in references [2-4].
[1] Lai, Fan, et al. “Fedscale: Benchmarking model and system performance of federated learning at scale.” International conference on machine learning. PMLR, 2022.
[2] Chen, Yifan, Shengli Liu, and Dingzhu Wen. “Communication Efficient Decentralized Learning over D2D Network: Adaptive Relay Selection and Resource Allocation.” IEEE Wireless Communications Letters (2024).
[3] Yemini, Michal, et al. “Robust Semi-Decentralized Federated Learning via Collaborative Relaying.” IEEE Transactions on Wireless Communications (2023).
[4] Tang, Zhenheng, et al. “Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication.” IEEE Transactions on Parallel and Distributed Systems 34.3 (2022): 909-922.
Optimizing the Simulation of Decentralized Learning Algorithms
Master’s Thesis or MSc semester project
Contact: Martijn de Vos (martijn.devos@epfl.ch)
The simulation of distributed learning algorithms is important for understanding their achievable model accuracy and the overhead in terms of total training time and communication cost. For this purpose, our lab has recently designed and implemented a distributed simulator specifically devised to simulate decentralized learning (DL) algorithms [1].
The main idea of this simulator is that first, the timestamps of all events in the system (training, model transfers, aggregation, etc.) are determined using a discrete-event simulator. Meanwhile, the simulator devises a compute graph containing all compute tasks (train, aggregate, or test). Then, the compute graph is solved in a distributed manner, possibly using different machines and multiple workers. An advantage of this simulator over others is that it supports the integration of real-world mobile traces that include each node’s training and network capacity.
Because of the discrete-event simulation, we maintain full control over the passing of time, enabling the evaluation of DL algorithms with nodes with different hardware characteristics. The FedScale simulator [2] uses a similar idea but only supports Federated Learning, which is generally easier to simulate than decentralized learning. While the first version of our simulator is already in use for various projects involving DL, there is a significant opportunity to enhance its scalability regarding the number of nodes it can support.
This project aim to improve the scalability of our simulator by identifying and implementing various optimization techniques. For instance, the current simulator is limited by the available memory, as many DL algorithms induce a memory footprint that scales linearly with the number of nodes in the DL network. One potential approach is to reduce the simulator’s memory footprint by strategically aggregating models as soon as they arrive in a machine. Other optimizations can revolve around the manipulation of the compute graph that is generated during the discrete-event simulation.
Affinity and experience with implementing distributed systems are required for this project. Since the project primarily focuses on the performance of simulation rather than on the ML algorithms, familiarity with ML/DL algorithms is optional but can be helpful during the project.
[1] Source code: https://github.com/sacs-epfl/decentralized-learning-simulator
A Comparative Evaluation of Decentralized Learning Algorithms using Realistic Real-world Traces
Master’s Thesis or MSc semester project
Contact: Martijn de Vos (martijn.devos@epfl.ch)
Decentralized Learning (DL) has gained significant attention in recent years due to its potential to enhance the privacy, fault tolerance, and scalability of machine learning compared to centralized settings. Various algorithms have been proposed for DL, such as Decentralized Stochastic Gradient Descent (D-SGD), AllReduce-SGD (in which nodes are connected in a ring topology and use AllReduce to average their models), Asynchronous Distributed Parallel Stochastic Gradient Descent (AD-PSGD) and Gossip Learning (GL). While these algorithms all share the same objective – collaboratively train a model without sharing data – they make different trade-offs, e.g., whether there is round synchronisation and how models are averaged across peers. To better understand the trade-offs of these algorithms, a comprehensive experimental evaluation is needed. In this project, we propose a detailed comparative analysis of these DL algorithms using realistic real-world traces that capture compute power, network capacity, data heterogeneity, and node availability in realistic FL settings.
The primary goal of this project is to compare and evaluate state-of-the-art DL algorithms under realistic conditions. You will implement these algorithms in an existing simulator [1]. This simulator already has different traces integrated that mimic the actual behaviour of compute power, network capacity, data heterogeneity, and node availability in real-world settings. You will leverage these traces and implement different DL algorithms. By doing so, we provide a more accurate and practical assessment of these algorithms’ strengths and weaknesses, which will guide future research and development in the field of DL.
[1] Source code: https://github.com/sacs-epfl/decentralized-learning-simulator
