SwiftSNNI: Optimized Scheduling for Secure Neural Network Inference (SNNI) on Multi-Core Systems

Secure Neural Network Inference (SNNI) enables privacy-preserving inference on encrypted data with strong cryptographic guarantees. However, practical deployments suffer from high preprocessing overhead, significant communication costs, and sequential execution. These limitations lead to low throughput, underutilized system resources, long queueing delays, and poor scalability. This work introduces \textit{SwiftSNNI}, a unified, resource-aware scheduling framework for SNNI. It implements a hybrid offline–online strategy that orchestrates offline preprocessing ($T_{\text{pre}, i}$) and online inference ($T_{\text{on}, i}$) jobs to maximize parallelism. By formulating SNNI scheduling as a constrained optimization problem, \textit{SwiftSNNI} overlaps $T_{\text{pre, i}}$ phase execution of future requests with active $T_{\text{on, j}}$ jobs. \textit{SwiftSNNI} also incorporates optional advance notices to enable proactive $T_{\text{pre}, i}$, which further reduces average input delay ($D$). Evaluations using five benchmark neural networks (M1, M2, HiNet, AlexNet, VGG-16) under diverse workloads and stochastic arrival rates confirm substantial performance gains. Compared to a parallelized sequential baseline (MS-SHARK), \textit{SwiftSNNI} achieves up to 97\% lower average input delay ($D$), a 81\% reduction in makespan ($\approx 5.4 \times$ speedup), and delivers $5.6 \times$ increase in throughput. Furthermore, \textit{SwiftSNNI} reduces average waiting time ($W$) by over 99\%, demonstrating robust starvation prevention for high-concurrency workloads. \textit{SwiftSNNI} supports concurrent execution, scales to larger neural networks, and provides an efficient runtime for SNNI deployments. The \footnote{https://github.com/KanwalBat00l/SwiftSNNI}{\textit{SwiftSNNI}} implementation is available online.