TY - JOUR
T1 - TinyFEL: Communication, Computation, and Memory Efficient Tiny Federated Edge Learning via Model Sparse Update
AU - Chen, Qimei
AU - Cheng, Han
AU - Liang, Yipeng
AU - Zhu, Guangxu
AU - Li, Ming
AU - Jiang, Hao
PY - 2025/4/1
Y1 - 2025/4/1
N2 - Federated edge learning (FEL) is regarded as a promising distributed machine learning paradigm to reduce transmission latency and resources as well as preserve raw data privacy by collaboratively training local deep learning models across multiple edge devices. However, with the development of artificial intelligence (AI) technologies, the size of neural network models grows exponentially with their parameters to meet variable application requirements, which poses significant challenges to the computation, communication, and memory abilities of edge devices. Existing designs typically focus on either communication or computation efficiency without caring each device's memory ability. To deal with the above issues, we first introduce a novel model sparse update enabled tiny FEL (TinyFEL) architecture, which terminates the backpropagation early in local model training processes. Therefore, the proposed TinyFEL can reduce local memory occupation and lessen the communication-and-computation burden. Furthermore, we propose a parameter splitting mechanism instead of transmitting the full model, only a part of updated layers of parameters is transmitted for aggregation, which significantly reduced the communication overheads. Thereafter, we develop a communication and computation latency minimization problem to accelerate the training of TinyFEL. To this end, we theoretically analyze the convergence performance of TinyFEL, which unveils the mathematical relationship among sparse update ratio assignment, device selection, and learning performance. Then, a joint sparse update ratio assignment, device selection, and resource allocation strategy is introduced based on the alternating direction method of multipliers (ADMMs) and block coordinate descent (BCD) algorithms. Numerical results indicate that our proposed TinyFEL can reduce training memory occupation by over 40% than the traditional FEL at the cost of negligible accuracy loss.
AB - Federated edge learning (FEL) is regarded as a promising distributed machine learning paradigm to reduce transmission latency and resources as well as preserve raw data privacy by collaboratively training local deep learning models across multiple edge devices. However, with the development of artificial intelligence (AI) technologies, the size of neural network models grows exponentially with their parameters to meet variable application requirements, which poses significant challenges to the computation, communication, and memory abilities of edge devices. Existing designs typically focus on either communication or computation efficiency without caring each device's memory ability. To deal with the above issues, we first introduce a novel model sparse update enabled tiny FEL (TinyFEL) architecture, which terminates the backpropagation early in local model training processes. Therefore, the proposed TinyFEL can reduce local memory occupation and lessen the communication-and-computation burden. Furthermore, we propose a parameter splitting mechanism instead of transmitting the full model, only a part of updated layers of parameters is transmitted for aggregation, which significantly reduced the communication overheads. Thereafter, we develop a communication and computation latency minimization problem to accelerate the training of TinyFEL. To this end, we theoretically analyze the convergence performance of TinyFEL, which unveils the mathematical relationship among sparse update ratio assignment, device selection, and learning performance. Then, a joint sparse update ratio assignment, device selection, and resource allocation strategy is introduced based on the alternating direction method of multipliers (ADMMs) and block coordinate descent (BCD) algorithms. Numerical results indicate that our proposed TinyFEL can reduce training memory occupation by over 40% than the traditional FEL at the cost of negligible accuracy loss.
KW - Computational modeling
KW - Training
KW - Performance evaluation
KW - Memory management
KW - Servers
KW - Data models
KW - Optimization
KW - Numerical models
KW - Resource management
KW - Mathematical models
KW - Communication resources
KW - computation resources
KW - federated edge learning (FEL)
KW - model sparse update
KW - parameter splitting
KW - Computational modeling
KW - Training
KW - Performance evaluation
KW - Memory management
KW - Servers
KW - Data models
KW - Optimization
KW - Numerical models
KW - Resource management
KW - Mathematical models
KW - Communication resources
KW - computation resources
KW - federated edge learning (FEL)
KW - model sparse update
KW - parameter splitting
UR - https://openir.whu.edu.cn/AchievementDetail/Scopus_222c505d-6d65-4715-9619-7f0ea8c0e8ba
U2 - 10.1109/jiot.2024.3499375
DO - 10.1109/jiot.2024.3499375
M3 - Article
SN - 2327-4662
VL - 12
SP - 8247
EP - 8260
JO - IEEE INTERNET OF THINGS JOURNAL
JF - IEEE INTERNET OF THINGS JOURNAL
IS - 7
ER -