Aurora: A Leverage-Aware Optimizer for Rectangular Matrices | Tilde

High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking

We study high-probability (HP) convergence guarantees in decentralized stochastic optimization, where multiple agents collaborate to jointly train a model over a network. Existing HP results in decentralized settings almost exclusively focus on the Decentralized Stochastic Gradient Descent ($\mathtt{DSGD}$) algorithm, which requires strong assumptions, such as bounded data heterogeneity, or strong convexity of each agent's cost. This is contrary to the mean-squared error (MSE) results, where methods incorporating bias-correction techniques are known to converge under relaxed assumptions and achieve better practical performance. In this paper we provide the first step toward bridging the gap, by studying HP convergence of $\mathtt{DSGD}$ incorporating the gradient tracking technique, in the presence of noise satisfying a relaxed sub-Gaussian condition. We show that the resulting method, dubbed $\mathtt{GT-DSGD}$, achieves order-optimal HP convergence rates for both non-convex and Polyak-Łojasiewicz costs, of order $\mathcal{O}\Big(\frac{\log(1/δ)}{\sqrt{nT}}\Big)$ and $\mathcal{O}\Big(\frac{\log(1/δ)}{nT}\Big)$, respectively, where $n$ is the number of agents, $T$ is the time horizon and $δ\in (0,1)$ is the confidence parameter. Our results establish that $\mathtt{GT-DSGD}$ converges in the HP sense under the same conditions on the cost as in the MSE sense, while achieving comparable transient times. To the best of our knowledge, these are the first HP guarantees for decentralized optimization methods incorporating bias-correction. Numerical experiments on real and synthetic data verify our theoretical findings, underlining the superior performance of $\mathtt{GT-DSGD}$ and highlighting that the benefits of incorporating bias-correction are also maintained in the HP sense.

Sea of Nodes

A Sea of Nodes IR Tutorial. Sea of Nodes has 5 repositories available. Follow their code on GitHub.

Removing fsync from our local storage engine | FractalBits Blog

How we used pre-allocation, O_DIRECT, and an SSD-aware journal to keep our local storage engine's writes crash-consistent without fsync.

Making Julia as Fast as C++ · FLOW Lab

Vending Machines / 42+ Assets | 3D Props | Unity Asset Store

Elevate your workflow with the Vending Machines / 42+ Assets asset from PackDev. Find this & other Props options on the Unity Asset Store.

Sudo su (@sudoingX) on X

it's now writing CUDA C++ dispatch hooks for llama.cpp. the agent is patching mmqcu to route Q8 matmul through its own optimized kernel. a 27B model on a dgx spark is rewriting its own inference engine. i'm just watching.

Вторая жизнь MacBook Air 2011: Как Linux Mint, ZRam и магия Swapspace сотворили чудо

У каждого из нас есть старый верный друг. Мой друг — это  MacBook Air 2011 года . Тонкий алюминиевый корпус, стильный и красивый дизайн, с отличной клавиатурой...