Breaking the Privacy Promise: Gradient Inversion Attacks on Federated Low-Rank Adapted Vision Transformers

Abstract

Parameter-efficient fine-tuning (PEFT) has been increasingly adopted to fine-tune pretrained models with low computational cost and is therefore also preferred in the federated learning paradigm to reduce computational and communication overhead. In this work, we investigate the sparsely explored domain of the risk of private data leakage in LoRA-adapted Vision Transformers (ViTs), where low-rank updates weaken the gradient signal available to an honest-but-curious server. We show that reconstruction is limited by two coupled ambiguities: reduced information in LoRA gradients and missing spatial cues from frozen positional embeddings. To address this, we propose a practically plausible spatially anchored generative inversion framework that incorporates trainable positional embeddings into transformer blocks during local fine-tuning. The resulting positional gradients act as layer-wise spatial anchors, while the generative prior keeps reconstructions on a realistic image manifold, reducing twin-data and patch-permutation artifacts. Adding our anchoring to GIAS improves PSNR by 10.14 dB, SSIM by 0.45, and LPIPS by 0.098 on CIFAR-10, with PSNR gains of 7.83 and 11.74 dB on high-resolution CelebA and CARS, respectively. The proposed strategy provides gains at higher batch sizes and in FedAvg settings. These results indicate that LoRA-based PEFT does not, by itself, prevent data leakage in FL and that carefully incorporated positional embeddings can help in exposing private information.

Method: Spatial Anchoring

The server prescribes a ViT in which a trainable positional embedding $p_l$ is added to the input of every transformer encoder layer (except the first), alongside the LoRA adapters. This doesn't change the clients' local training procedure. The architecture also improves task performance and robustness, which gives clients a genuine reason to adopt it. The overhead is small: GFLOPs increase by only 0.016% for ViT-Small.

The gradients of $p_l$ now carry explicit, layer-wise spatial information. The attacker builds on the generative GIA framework GIAS. It optimizes a latent code $z$ and then the generator $G_\omega$ so that the gradients of the candidate image $\hat{x} = G_\omega(z)$ match the client's gradients, for both the LoRA/classifier parameters and the positional embeddings:

Contributions

• We show that LoRA-induced parameter sparsity increases the non-identifiability of gradient inversion by reducing the information available to the attacker.
• We identify frozen positional embeddings as a fundamental source of duplicate reconstructions under PEFT.
• We design layer-wise trainable positional embeddings for ViTs that expose positional gradients during federated PEFT and consistently improve reconstruction.
• We evaluate extensively across LoRA ranks, batch sizes, FedSGD and FedAvg, defences and high-resolution datasets.

Quantitative results

CIFAR-10, single step, batch size 1

Mean over 10 users with three reconstructions each; standard deviation in parentheses. Green marks an improvement.

High-resolution datasets

Multi-step updates (FedAvg), CIFAR-10, PSNR ↑

Each client holds 16 private samples per round, giving E×16/B local steps. The GIAS gain is statistically significant (Welch's t-test, p < 0.05). SSIM and LPIPS are in the paper.

Ablations

Privacy–utility trade-off

BibTeX

@InProceedings{Badiya_2027_WACV,
    author    = {Badiya, Ritik K. and Manna, Siladittya and Sinha, Ashmit and
                 Mandula, Yashwanth and Chakraborty, Anirban},
    title     = {Breaking the Privacy Promise: Gradient Inversion Attacks on
                 Federated Low-Rank Adapted Vision Transformers},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {},
    year      = {2027},
    pages     = {}
}