arxiv:2602.05258

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs

Published on Feb 5

· Submitted by

Haoran Li on Feb 6

Carnegie Mellon University

Upvote

Authors:

Haoran Li ,

Abstract

CoPE introduces a soft clipping method for Rotary Positional Embedding that unifies out-of-distribution mitigation and semantic modeling while enabling effective long-context processing up to 256k length.

AI-generated summary

Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts, their guiding principles generally fall into two categories: (1) out-of-distribution (OOD) mitigation, which scales RoPE frequencies to accommodate unseen positions, and (2) Semantic Modeling, which posits that the attention scores computed with RoPE should always prioritize semantically similar tokens. In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely CoPE: soft clipping lowfrequency components of RoPE. CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping. Extensive experiments demonstrate that simply applying our soft clipping strategy to RoPE yields significant performance gains that scale up to 256k context length, validating our theoretical analysis and establishing CoPE as a new state-of-the-art for length generalization. Our code, data, and models are available at https://github.com/hrlics/CoPE.

View arXiv page View PDF GitHub 5 Add to collection

Community

haoranli-ml

Paper author Paper submitter about 12 hours ago

•

edited about 3 hours ago

[Paper] [HF checkpoints]

CoPE is a plug-and-play enhancement of RoPE that softly clips the unstable low-frequency components, delivering consistent gains both within the training context and during long-context extrapoaltion.

With a simple yet effective soft clipping strategy, CoPE

1️⃣ Eliminates severe OOD outliers, whose periods exceed the pre-training context window and are the primary cause of OOD extrapolation.

2️⃣ Refines Long-range Semantic Signals by alleviating the secret long-term decay of semantic attention introduced by RoPE.

3️⃣ Prevents Spectral Leakage induced by hard frequency truncation, which otherwise leads to long-range oscillatory ringing in the attention scores across relative token distances and introduces spurious correlations.