New Yuan Intelligence Report



Imagine a customer service system where hundreds or thousands of user requests come in every second.

Each request has to go through a "triage desk": is it a refund, a logistics inquiry, or a complaint? Should it be transferred to a human agent? Which tool should be called?

These types of questions have one common feature: the answer is among a few given options .

For this purpose, the industry has developed specialized decision models , such as Jev and the open-source Kev. They read in the request and the options, and provide the probability for each option in a single forward pass, returning in dozens of milliseconds without generating any text.

However, problems also arise.

"I want a refund" and "I bought something last month, returned it once, and this time it broke again. Can I directly get a refund, and who will cover the shipping cost?" In the eyes of the decision model, these two questions cost the same amount of computation .

Simple requests are "over-serviced", while complex requests can't be given more thought.

What if the model first writes a reasoning segment and then answers? It's possible, but each request has to pay for the reasoning text , even if it's just a single sentence "I want a refund".

Is there a model that can be as fast as a small model for simple questions and as thoughtful as a large model for complex ones?

To answer this question, a research team from Lehigh University, University of British Columbia, and Vector Institute proposed Lev: a decision model that deepens thinking by using cycles rather than tokens. Paper link: https://arxiv.org/abs/2610.20951 Project homepage: https://openlair.github.io/lev/ Code link: https://github.com/OpenLAIR/lev Model and dataset: https://huggingface.co/collections/songdj/lev Lev answers in cycles: In example 1, the correct answer B is selected in the first cycle, and the stopper determines it can stop; in example 2, 4 is selected in the first cycle, and then changed to the correct answer 6 in the fourth cycle. (Source: redrawn from the per-cycle readings recorded on the project homepage, replaying experimental data, not real-time reasoning) Paper figure 1: The number of cycles used by Lev-2.6B on JevBench simple, standard, and hard tasks, as well as accuracy and latency. Lev's core idea can be summarized in one sentence: the same group of network layers is repeatedly run, and each cycle reads out an answer. The authors refer to this mode of computation as System 1.5: it retains the interface of System 1, which "directly gives a decision," while also spending more computation on difficult tasks like System 2, but these computations occur in the hidden states rather than being written out in text. The results are very intuitive: On the 231 public questions of JevBench, the same Lev-2.6B checkpoint increased from 63.2% in one round to 76.2% in four rounds, an increase of 13.0 percentage points; The stopper cannot see the difficulty of the questions, but on average used 1.2, 2.4, and 3.0 rounds respectively for simple, standard, and difficult questions; On-demand depth in offline statistics calculated 15% to 24% less of the loop computation, with accuracy loss less than 1 point; Within the latency interval of under 50ms, Lev-1.4B and Lev-3B were 5.4 to 8.8 points higher than the non-loop models and cascading with the same latency (measured on single H200, one request at a time). The additional computing power is only useful on a few questions.

The author took two publicly available decision-making models Kev-0.8B and Kev-4B . Both models have the same training data source, and on the 231 public questions in JevBench, the large model answered correctly on 28 more questions .

These 28 questions were all from standard questions and difficult ones . The simple questions were all correctly answered by both models.

The questions where the two models gave different results were only 42 questions , yet the large model spent extra computation on all 231 questions.

If we use the confidence level of the small model to select its half least certain questions and pass them to the large model, the accuracy almost catches up with the large model's full answer .

Conclusion: Additional computation only provides returns on a few questions, and a cheap initial read can identify them.


Figure 2 of the paper: (a) The per-question results of Kev-0.8B and Kev-4B; (b) the effect of passing only the half least certain questions to the large model; (c) comparison between generated reasoning and single-time reading; (d) the change in accuracy when adding a loop to a regular pre-trained model .

Why are the existing three "add computational power" approaches insufficient?

  • Cascading: The median latency of Kev-0.8B is already 79ms, while Kev-4B is 108ms. The transferred applications need to be read from the beginning again, which is slower than directly using a large model and requires deploying an additional model. Generating inference text: On two open-source Jev-style services based on the same 26B foundation, enabling thinking improves the accuracy of difficult questions by 9 to 13 points, but the median time for difficult questions becomes three to four times longer. Adding loops to the pre-trained model: The author connected the middle 8 layers of Qwen3.5-0.8B into gated loops, and the accuracy change was less than half a point, with the gate always close to zero, making the loop effectively unused. The last point is particularly crucial. It indicates that loops should be learned during pre-training rather than added temporarily during decision training. Lev's three key designs Recently, the cyclic Transformer has once again attracted attention from the industry. Based on the above judgment, Lev selected native loop language model as the base: Byte Seed's Ouro (1.4B, 2.6B, loop 4 times) and

    Nanbeige4.2-3B

    (loop 2 times).

    Paper Figure 3: Lev's overall structure. The same group of decoding layers run in a loop, and the shared pointer head reads an option each round; the stopping mechanism determines when to exit for each question.

    Design One: Each round outputs an answer.

    The request and all options are packed into a single sequence. The same group of decoding layers runs repeatedly, and after each round, the same pointer head reads the options and position's hidden state, outputting the option probabilities once.

    Design Two: All rounds are jointly trained.

    The pointer head and LoRA adapters are shared across rounds, loss is averaged over all rounds . Thus, the 1st round, 2nd round, and 4th round are all independently usable decision models.

    When using k rounds, only the first k rounds are calculated; when needing an extra round, it continues calculation from the hidden state of the previous round, the request does not need to be read again .

    Design Three: The stopping mechanism predicts whether it is consistent with the deepest round.

    Common early stopping rule is 'stop when confidence is high enough'. But the author found that shallow reading was severely overconfident on new data: Lev-2.6B's questions that stopped in the first round achieved 94.8% accuracy on the calibration set, but only 78.9% on new data sources, with confidence levels almost the same. Moreover, confidence scores couldn't distinguish errors that could be corrected by one more round of calculation, with AUROC only ranging from 0.46 to 0.61. So Lev's stopper changed its target: instead of predicting whether the current answer is correct, it predicts whether the current answer differs from the deepest round. The "standard answer" for this target is the model's own deepest output, and fitting the stopper and selecting thresholds don't require manual annotations. Deployers can directly use unlabeled requests from their own business for calibration. Experimental results One weight, multiple operating points The author trained three Lev models and three non-cyclic counterparts (Qwen3-1.7B, Qwen3.5-2B, Qwen3.5-4B) using exactly the same 31,329 training data and recipe. Evaluations included three tuning sets, two held-out test sets (LevBench, built by the author, was frozen before any model read it), and 231 public questions from JevBench. Conclusion 1: The deeper the layer, the more accurate, but the gains diminish.

    Lev-2.6B on JevBench from one round to four rounds sequentially are 63.2%、71.0%、75.3%、76.2%; on held-out average sequentially are 51.3%、62.0%、64.8%、65.7%. Most of the gains come from the first additional round, with the fourth round adding at most 1 more point.

    The further away from the training data, the more useful depth is: improving by about 7 points on the training data source, and by 9.0 to 19.5 points on more unfamiliar test sets.

    Conclusion two: improvements are concentrated on difficult problems.

    From one round to four rounds, Lev-2.6B on 111 difficult problems net correct 19 more, on 72 standard problems net correct 10 more. Among the difficult problems, 29 were corrected to correct, 10 were corrected to wrong.


    Paper Figure 5: Where does depth contribute, including JevBench's various difficulty levels, modifications between adjacent rounds, and improvements across different application domains.

    Different tasks benefit differently: on Kev-Breadth, depth allows language understanding tasks to improve 19.0 points, while on 'taste' tasks like humor and color preference there is no gain. The authors speculate that the additional rounds provide the serial computation needed to combine multiple facts.

    Conclusion three: reducing depth by 15% to 24% on demand, accuracy almost doesn't drop.

    On two held-out test sets, pruning only used the deepest one round 76% to 85% of computation, with an accuracy loss of 0.3 to 0.8 points. In the locking test where the recipe and stopper are fully determined and read only once, the loss is at most 1.0 points.


    Paper Figure 6: Accuracy and computational cost. Gray represents fixed depth and its frontier, blue line represents the stopper, and star represents the locking test.

    Conclusion Four: The key is to calibrate using target traffic.

    As long as the threshold is selected on unlabeled target requests in the same way as the stopper, with the criterion of ``matching the deepest answer'', a simple maximum probability threshold can differ from the stopper by less than 0.2 points. 500 unlabeled questions per fold are sufficient.

    The authors clearly explain the applicability of the results in the paper: within a latency of 39 to 46ms , the on-demand Lev-1.4B and Lev-3B models outperform the only available cascading model with the same latency by 7.0 to 8.8 points, and surpass the best non-cyclic scheme by 5.4 to 7.5 points

    .

    Paper Figure 10: Accuracy and average latency on LevBench locking tests. Star represents the on-demand depth Lev, and numbers indicate the gap compared to the best non-cyclic scheme.

    Compared to fixed depth with the same cost, the advantage of on-demand depth is not significant: Lev-2.6B is only 0.2 points higher on held-out, and Lev-3B is 1.0 points higher; Lev-2.6B performs equally to the fixed depth frontier on the locked test. "Less thinking" is offline statistics: the saved computation is converted into online throughput or latency, requiring the service side to support the bundling scheduling of requests with different depths. The advantage lies in latency, not computation: when calculated by floating-point operations, each Lev model performs equally or worse than the non-cyclic scheme, because each loop runs the entire base. Some of the latency advantage also comes from the gaps between 22 to 79ms in the non-cyclic scheme. Delay measurement conditions: single H200, bf16, one request at a time, and no fusion operators, graph capture, or state caching are used in all systems. Multi-thinking also makes mistakes: on LevBench, the first additional loop corrects 20.4% of the questions but also makes 8.6% mistakes; 12.6% of the questions are correctly answered in shallow layers but wrong in the deepest loop. Summary and outlook: Lev provides the insight that the native cyclic base can turn "how deep to think" from a constant during training into a configurable setting during service. The same decision model can make trade-offs between latency and accuracy for each question, instead of for each deployment. The author proposes two subsequent directions: Design new stopping targets, recover the "shallow correct, deep wrong" questions, and thus surpass the accuracy of the deepest layer;

  • Test on a larger circular base to see whether the gains on difficult questions will continue to grow with scale.

Currently, the paper, code, three models, and three datasets are all publicly available. The datasets are subsets filtered according to the redistribution terms of their sources, and sources that cannot be hosted are accompanied by reconstruction instructions.




Team Introduction


  • Dingjie Song (Dingjie Song, co-first author): PhD student at Lehigh University.

  • Zhiling Yan (co-first author): PhD student at Lehigh University.

  • Yuxuan Zhang: PhD student at the University of British Columbia, Vector Institute.

  • Lichao Sun (corresponding author): Assistant Professor at Lehigh University




References:

https://openlair.github.io/ 

https://arxiv.org/abs/2610.20951

https://openlair.github.io/lev/


Editor: LRST



Second Chase ASI
⭐Like, share, and mark as read in one click⭐
Light up the star, lock in the fast push from New Intelligence Yuan!