DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI business “devoted to making AGI a reality” and open-sourcing all its models. They started in 2023, but have been making waves over the past month approximately, and particularly this previous week with the release of their two latest reasoning models: DeepSeek-R1-Zero and the more sophisticated DeepSeek-R1, also known as DeepSeek Reasoner.

They’ve released not just the designs but likewise the code and assessment prompts for public use, together with an in-depth paper detailing their method.

Aside from creating 2 highly performant models that are on par with OpenAI’s o1 design, the paper has a lot of valuable info around support knowing, chain of idea thinking, timely engineering with thinking models, and more.

We’ll start by focusing on the training process of DeepSeek-R1-Zero, which uniquely relied solely on support knowing, instead of traditional monitored knowing. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some timely engineering best practices for reasoning designs.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s latest model release and comparing it with OpenAI’s thinking designs, specifically the A1 and A1 Mini designs. We’ll explore their training procedure, reasoning capabilities, and some essential insights into timely engineering for thinking models.

DeepSeek is a Chinese-based AI company devoted to open-source advancement. Their recent release, the R1 thinking design, is groundbreaking due to its open-source nature and ingenious training techniques. This includes open access to the designs, prompts, and research papers.

Released on January 20th, DeepSeek’s R1 achieved remarkable efficiency on various standards, rivaling OpenAI’s A1 designs. Notably, they likewise launched a precursor model, R10, which functions as the foundation for R1.

Training Process: R10 to R1

R10: This design was trained specifically using reinforcement learning without monitored fine-tuning, making it the very first open-source design to attain high efficiency through this method. Training included:

– Rewarding proper responses in deterministic tasks (e.g., math problems).
– Encouraging structured reasoning outputs using design templates with “” and “” tags

Through countless versions, R10 developed longer thinking chains, self-verification, and even reflective behaviors. For instance, during training, the model demonstrated “aha” minutes and self-correction habits, which are uncommon in standard LLMs.

R1: Building on R10, R1 added a number of improvements:

datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human preference alignment for refined responses.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at numerous sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 models across many thinking benchmarks:

Reasoning and Math Tasks: R1 competitors or surpasses A1 designs in precision and depth of thinking.
Coding Tasks: A1 designs typically perform better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 frequently surpasses A1 in structured QA jobs (e.g., 47% precision vs. 30%).

One noteworthy finding is that longer reasoning chains usually enhance efficiency. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some limitations:

– Mixing English and Chinese reactions due to a lack of monitored fine-tuning.
– Less refined actions compared to chat designs like OpenAI’s GPT.

These issues were attended to throughout R1’s refinement process, consisting of supervised fine-tuning and human feedback.

Prompt Engineering Insights

A remarkable takeaway from DeepSeek’s research study is how few-shot prompting degraded R1’s performance compared to zero-shot or concise tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s recommendations to limit context in thinking models. Overcomplicating the input can overwhelm the design and minimize accuracy.

DeepSeek’s R1 is a significant advance for open-source thinking models, showing abilities that equal OpenAI’s A1. It’s an exciting time to experiment with these models and their chat user interface, which is totally free to utilize.

If you have questions or wish to find out more, inspect out the resources connected below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only method

DeepSeek-R1-Zero sticks out from many other advanced designs due to the fact that it was trained using just support learning (RL), no monitored fine-tuning (SFT). This challenges the present traditional approach and opens up new opportunities to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the very first open-source design to verify that advanced thinking capabilities can be established purely through RL.

Without pre-labeled datasets, the design learns through trial and mistake, improving its habits, specifications, and weights based solely on feedback from the services it creates.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero included presenting the design with various thinking jobs, varying from mathematics problems to abstract reasoning difficulties. The model produced outputs and was evaluated based on its performance.

DeepSeek-R1-Zero received feedback through a benefit system that assisted direct its knowing process:

Accuracy rewards: Evaluates whether the output is correct. Used for when there are deterministic results (mathematics problems).

Format rewards: Encouraged the design to structure its reasoning within and tags.

Training timely template

To train DeepSeek-R1-Zero to generate structured chain of idea series, the scientists used the following timely training template, changing timely with the thinking concern. You can access it in PromptHub here.

This design template prompted the model to clearly detail its idea process within tags before delivering the final answer in tags.

The power of RL in thinking

With this training procedure DeepSeek-R1-Zero began to produce sophisticated reasoning chains.

Through countless training steps, DeepSeek-R1-Zero evolved to resolve increasingly intricate problems. It learned to:

– Generate long thinking chains that allowed much deeper and more structured problem-solving

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own errors, showcasing emergent self-reflective behaviors.

DeepSeek R1-Zero efficiency

While DeepSeek-R1-Zero is mainly a precursor to DeepSeek-R1, it still attained high efficiency on numerous benchmarks. Let’s dive into some of the experiments ran.

Accuracy improvements during training

– Pass@1 precision started at 15.6% and by the end of the training it improved to 71.0%, similar to OpenAI’s o1-0912 model.

– The red solid line represents efficiency with bulk voting (similar to ensembling and self-consistency techniques), which increased accuracy even more to 86.7%, going beyond o1-0912.

Next we’ll take a look at a table comparing DeepSeek-R1-Zero’s efficiency across multiple reasoning datasets against OpenAI’s reasoning models.

AIME 2024: 71.0% Pass@1, slightly listed below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a rating of 73.3%.

– Performed much even worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll look at how the response length increased throughout the RL training procedure.

This graph shows the length of responses from the design as the training procedure advances. Each “action” represents one cycle of the design’s knowing process, where feedback is provided based on the output’s efficiency, evaluated utilizing the prompt template talked about earlier.

For each question (corresponding to one action), 16 reactions were tested, and the typical accuracy was determined to make sure steady examination.

As training advances, the model produces longer thinking chains, enabling it to resolve significantly intricate thinking jobs by leveraging more test-time compute.

While longer chains don’t always guarantee better results, they usually associate with improved performance-a pattern also observed in the MEDPROMPT paper (find out more about it here) and in the initial o1 paper from OpenAI.

Aha moment and self-verification

One of the coolest aspects of DeepSeek-R1-Zero’s development (which also applies to the flagship R-1 model) is just how great the model ended up being at reasoning. There were advanced reasoning habits that were not explicitly programmed however developed through its support finding out process.

Over countless training steps, the design began to self-correct, review flawed logic, and confirm its own solutions-all within its chain of idea

An example of this kept in mind in the paper, described as a the “Aha minute” is below in red text.

In this circumstances, the design literally stated, “That’s an aha minute.” Through DeepSeek’s chat feature (their version of ChatGPT) this kind of thinking normally emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and obstacles in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to perform at a high level, there were some drawbacks with the design.

Language mixing and coherence concerns: The design occasionally produced actions that mixed languages (Chinese and English).

Reinforcement learning trade-offs: The lack of monitored fine-tuning (SFT) meant that the model lacked the improvement needed for totally polished, human-aligned outputs.

DeepSeek-R1 was established to deal with these problems!

What is DeepSeek R1

DeepSeek-R1 is an open-source thinking design from the Chinese AI lab DeepSeek. It builds on DeepSeek-R1-Zero, which was trained totally with reinforcement learning. Unlike its predecessor, DeepSeek-R1 includes monitored fine-tuning, making it more refined. Notably, it outperforms OpenAI’s o1 design on a number of benchmarks-more on that later.

What are the main distinctions between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 constructs on the foundation of DeepSeek-R1-Zero, which works as the base model. The two vary in their training approaches and overall performance.

1. Training technique

DeepSeek-R1-Zero: Trained totally with reinforcement knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes supervised fine-tuning (SFT) initially, followed by the same reinforcement finding out process that DeepSeek-R1-Zero damp through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Battled with language blending (English and Chinese) and readability concerns. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these concerns with cold-start fine-tuning, making responses clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still a really strong thinking design, sometimes beating OpenAI’s o1, but fell the language blending concerns decreased usability greatly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on the majority of reasoning benchmarks, and the actions are a lot more polished.

Simply put, DeepSeek-R1-Zero was a proof of idea, while DeepSeek-R1 is the completely enhanced variation.

How DeepSeek-R1 was trained

To tackle the readability and coherence concerns of R1-Zero, the scientists included a cold-start fine-tuning stage and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a top quality dataset of long chains of thought examples for preliminary monitored fine-tuning (SFT). This information was gathered using:- Few-shot triggering with comprehensive CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 went through the exact same RL procedure as DeepSeek-R1-Zero to improve its thinking capabilities further.

Human Preference Alignment:

– A secondary RL stage enhanced the model’s helpfulness and harmlessness, making sure better alignment with user requirements.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller sized, efficient designs like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 standard efficiency

The researchers tested DeepSeek R-1 across a variety of criteria and versus top designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into numerous classifications, shown below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied throughout all models:

Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 outshined o1, Claude 3.5 Sonnet and other designs in the bulk of thinking benchmarks.

o1 was the best-performing design in 4 out of the five coding-related criteria.

– DeepSeek performed well on creative and long-context job job, like AlpacaEval 2.0 and ArenaHard, surpassing all other models.

Prompt Engineering with reasoning models

My favorite part of the short article was the scientists’ observation about DeepSeek-R1’s sensitivity to triggers:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which recommendations Microsoft’s research study on their MedPrompt structure. In their research study with OpenAI’s o1-preview design, they found that frustrating reasoning models with few-shot context broken down performance-a sharp contrast to non-reasoning designs.

The crucial takeaway? Zero-shot triggering with clear and concise instructions seem to be best when utilizing reasoning models.