F-Transformer: a federated transformer for efficient and privacy-preserving sequence generation.

Transformer models have demonstrated remarkable success in natural language processing (NLP) tasks, but their deployment in distributed environments faces critical challenges including high computational demands, large memory footprints, and privacy concerns when handling sensitive data. Existing federated learning (FL) implementations with transformers often sacrifice either model performance for privacy or resource efficiency for accuracy. We propose the F-Transformer, a lightweight federated transformer framework that addresses these limitations through integrated privacy-preserving mechanisms and architectural optimizations. Our framework employs a compact architecture with 4 attention heads, 4 layers, and 64 embedding dimensions, achieving only 0.87 million trainable parameters. We implement an incremental FL strategy where local clients continuously train on newly arriving data while the global model aggregates updates through the FedAvg algorithm. The framework integrates privacy objectives directly into the optimization process through a novel regularization formulation. We evaluate the F-Transformer on the WikiText-2 dataset using validation perplexity as the primary metric, along with central processing unit (CPU) utilization, memory consumption, and training loss convergence. Our results demonstrate a validation perplexity of 5.9894, surpassing state-of-the-art (SOTA) models including BERT-Large, GPT-2, and SparseGPT while using significantly fewer parameters. The framework achieves 40% reduction in CPU utilization and 34% reduction in memory consumption compared to centralized training. These results establish the F-Transformer as an effective solution for privacy-preserving sequence generation in resource-constrained federated environments.