GPT-OSS vs. Qwen vs. Deepseek: Comparing Open Source LLM Architectures

0:00OpenAI recently dropped GPT-OSS, its first open weights model since GPT2 in 2019. It's one of the highest-profile open-source model launches since DeepSeek R1 made waves back in January. But how does GPT-OSS compare to the other top open-source models out there architecturally? Let's find out.

0:23GPT-OSS is one of OpenAI's most anticipated recent launches, a large, fully open-weights model from one of the leading American AI labs. Let's take a closer look at the paper to find out how it was actually engineered and trained. GPT-OSS is a mixture of experts model available in two sizes, 120 billion parameters and 20 billion parameters. Each token activates the top four experts, meaning only a portion of the total parameters are used at any given time. This allows for efficient inference without sacrificing the benefits of a larger model. Trained as a decoder-only transformer, GPT-OSS incorporates plenty of features typical to modern LLMs.

0:57This includes grouped query attention, a modified attention mechanism that lets multiple query heads share the same key value pairs to reduce memory use and speed up inference. It also includes Swiglu activations in the feedforward network layers, which allow for more nuanced transformations than simpler activations like ReLU, as well as rotary positional impeddings, or ROPE, which encode token position directly into the attention mechanism to support longer contacts. Finally, the model also makes use of RMSNORM with pre-normalization, a normalization method that scales inputs by their root mean square for more stable training.

1:29One standout capability of the model is its 131 ,000 token context window, which it achieves by applying yarn scaling during pre-training rather than as an inference time adjustment. We'll touch on what this means a little bit later in the video. For GPT-OSS, OpenAI makes use of their open-source O200K Harmony tokenizer. This byte-pair encoding tokenizer has over 200 ,000 tokens and builds on the O200K tokenizer used in models like GPT-40. As for the dataset GPT-OSS was trained on, OpenAI has only disclosed the broad strokes. The model was trained on a text-only corpus in the trillions of tokens with a focus on STEM, coding, and general knowledge.

2:04Harmful content was filtered out for safety, but beyond that, there's little else known publicly. Once training was complete, the model was released in a quantized format by default, making it lightweight enough for deployment on modest hardware. This allows it to be run on consumer-grade GPUs, laptops, or other resource-limited hardware. However, there's no unquantized version available. GPT-OSS also underwent substantial post-training for safety and alignment, shaping its default behavior for more controlled outputs. It's worth noting that some in the open-source community are experimenting with reducing or removing these layers in order to explore the raw model's capabilities.

2:37In the broader landscape of open-source AI, GPT-OSS arrives as a fully equipped, long-context model ready for immediate use. As impressive as it is, however, it's just one of several models in a rapidly expanding field of open source LLMs. QEM3, the newest family of models developed by Alibaba Cloud, dropped this past April to considerable hype, with benchmark scores that rivaled those of leading open source-based models like DeepSeq V3 or Llama 4. The QEM3 family includes both dense models, which activate all of their parameters for each query, and mixture of expert models, which only activate a small subset of their parameters for each query.

3:11The dense models come in seven different size classes, including a 0.6 billion parameter model, one of the smallest current generation open-weight models around, while the MOE models come in two different size classes. Architecturally, QEN 3 dense models are very similar to the QEN 2.5 models, Alibaba's previous releases. Like QEN 2.5 and GPT-OSS, QEN 3 incorporates features like group query retention, SWE glue, ROPE, and RMS norm. QUIN3's sparse models share the same fundamental architecture as its dense models, but add a mixture of experts layer with 128 total experts, of which 8 are activated per token.

3:43All QUIN3 models also use the same tokenizer used in previous QUIN models, which implements byte-level byte-parent codings that allow it to handle any text or symbol without special preprocessing, unlike word or character-based tokenizers. One of the main things that sets QUIN3 apart from previous QUIN models is the way it controls the scale of the key query and value projections to keep attention scores stable at scale. It replaces QKV bias, a static offset that shifts KQ view projections in previous models, with QKNORM, a normalization step that dynamically rescales that query and key vectors to maintain constant magnitudes.

4:17Dataset-wise, QUEN3 was trained on 36 trillion pre-training tokens, twice as many as the QUEN 2.5 models. In addition to pulling data from multilingual text, STEM, and coding sources and reasoning tasks, QEM3 also uses QEM 2.5 models to generate trillions of tokens of synthetic data in different formats like textbooks, instructions, and code snippets. QEM3's pre-training occurred in three stages. In stage one, the general stage, models were trained on over 30 trillion tokens covering 119 languages at a sequence length of 4096 tokens. In stage 2, the reasoning stage, models were trained on an additional 5 trillion higher quality tokens featuring more STEM, reasoning, and coding problems.

4:53And in stage 3, which the Quen team calls the long context stage, context length was extended to over 32 ,000 tokens using a bunch of clever algorithmic optimizations, including ABF, a technique to adjust rope so positional signals remain accurate over much longer sequences, yarn to further scale for longer inputs, and dual chunk attention to process sequences efficiently. Together, all of these optimizations allow the model to reason over much longer inputs at inference. Finally, Quen uses a four-step post-training pipeline with two goals, giving users more control over how much reasoning to use for a given query and letting them efficiently distill larger model capabilities into smaller models.

5:29The first step in the post-training pipeline is a long chain-of-thought cold start stage, which involves feeding a model a curated dataset of challenging reasoning problems from math, logic, and STEM with verifiable reference answers, and then filtering outputs to ensure quality. This is followed by a reasoning RL stage using GRPO, an RL algorithm originally developed by DeepSeq researchers, on roughly 4 ,000 query verifier pairs to strengthen complex problem solving. Personally, I think it's fascinating that it only takes 4 ,000 pairs to get great results. The third step in the post-training pipeline, thinking mode fusion, is a key QN3 innovation that integrates reasoning and non-reasoning into a single model, letting users switch modes without changing models.

6:08Essentially, what developers did in this step was fine-tune the model on a mix of thinking data, which includes intermediate reasoning steps, and non-thinking data, which omits them, and then build a chat interface to let users toggle modes. Though this was unique to Quinn when the model first launched, GPT-5 now features a similar toggle. The final step, General RL, broadens capabilities in instruction following, formatting, preference alignment, tool use, and specialized scenarios. Quinn's developers then use strong-to-weak distillation, which allows for the training of smaller models from larger ones.

6:38All in all, QEM3's performance is very impressive, especially given its relatively small size. But just months earlier, a different model had already raised the stakes in open source. Released in December of last year, DeepSeek's V3 model was one of the most ambitious open source LLMs to come out of a major lab in recent years. A chatbot developed in China. It's called DeepSeek. DeepSeek is such a fundamental change to the economics of what's going on. The most downloaded free app in the US. This is an update in what people think is possible. At 671 billion parameters, it's a massive general purpose base model designed for efficiency as much as capability, laying the groundwork for the reasoning focused R1 model that would follow.

7:16We're not going to get into a ton of detail about V3's architecture or training pipeline here, because we put out a comprehensive deep dive into it back in February. But high level, the thing to know about V3 is that it's a mixture of experts model with several hardware and algorithmic optimizations, including training v3 natively in 8-bit rather than 16 or 32-bit, a huge unlock for cutting training costs. And just recently, DeepSeek pushed v3 even further with an updated version. The newly released v3.1 builds directly on the original v3-based checkpoint, extending it with a two-phase long-context training approach and adding a hybrid thinking mode that lets the same model switch between reasoning-heavy and lightweight inference.

7:53It also improves tool use and agent performance thanks to a more advanced post-training. In practice, this means v3.1 keeps the same core architecture as v3, but delivers stronger reasoning, smarter tool use, and greater performance. One thing that sets v3 apart is that it uses a different attention mechanism than GPT-OSS and QEM3. In modern LLMs, a lot of the compute and memory is tied up in the KV cache, and so v3 makes use of MLA, which compresses keys and values into a smaller latent space before caching them, then decompresses them during inference. Although MLA is a bit more complex to implement, the previous DeepSeq v2 paper found it delivers greater memory savings and better modeling performance than GQA, especially in huge long-context models like this one.

8:35And that's just one of several areas where DeepSeq v3 takes a different path. With all that in mind, let's take a step back. From v3 to QUEN to GPT-OSS, how should we think about, at a high level, the differences between these models? One big difference is size. The QUEN 3 model family is the only one of the three to offer both dense and mixture of expert variants, with dense models from 0.6 billion to 32 billion parameters in a mixture of experts lineup that includes a 30 billion parameter model and a 235 billion parameter model. Notably, QN's mixture of experts base models match the dense model's performance with only a fifth as many active parameters.

9:09On the other hand, DeepSeq v3 only comes in a mixture of experts architecture with 671 billion parameters, of which 37 billion are activated for a given token prediction, so considerably larger than even the biggest QN3 model. GPT-OSS sits in the middle. It offers two MOE models, one with 117 billion parameters, of which 5.1 billion are activated for a given token, and a smaller one with 21 billion parameters, of which 3.6 billion are activated for a given token. One of the most interesting technical differences lies in how each model extends its context length. YARN, short for Yet Another Rope Extension, is a technique for stretching the model's rotary positional embeddings so that it can handle far longer sequences than it was originally trained on.

9:46Normally, rope starts to break down when you feed it more tokens than its base frequency was set for, But Yarn tweaks that frequency so the same embedding space covers much more ground. What's interesting is how the three models here use it differently. GPT-OSS applies Yarn right from pre-training, so its weights have learned to work natively with 131 ,000 token contexts. DeepSeq takes a staged approach, fine-tuning after pre-training to first reach 32 ,000 tokens, then further training to achieve 128 ,000. Quen also fine-tunes to 32 ,000, but skips that additional retraining step. Instead, at inference time, they apply yarn scaling again, increasing the rope base frequency by a factor of four to reach 128 ,000 tokens without extra retraining.

10:27In other words, GPT-OSS is born with long-context ability. DeepSeq is trained into it step-by-step, and Quen pushes the limits of what a 32 ,000-trained model can do without more long-context training. Personally, I think one of the most interesting things about these papers, and the state-of-the-art in deep learning more generally, is that a lot of these read as empirical findings. Each lab describes a combination of tools that works well for them, but almost no one gives a first principles justification of why one tool is better than the other. For instance, why MLA is better than GQA, full stop.

10:55This is much different from domains like math or theoretical physics, which are all about providing first principles explanations that derive results from axioms or laws. Also, it's interesting that even though most of these models have similar top-line benchmark statistics and use broadly the same tools, like attention mechanisms, activation functions, positional embeddings, and so on, they achieved these similar results using often very different techniques. This is quite surprising. You'd expect that very different training methods would lead to very different results. Also, all of the major models heavily use reinforcement learning as part of the post-training and reasoning portions of their model training efforts.

11:28And it's fascinating and pretty surprising how some of these RL efforts require very little amounts of data, just 4 ,000 data pairs in the case of Quinn. Another point here is that it's very opaque what the differences in data sets are between the labs. It's clear from the papers that there's an enormous amount of work happening behind the scenes in dataset engineering. This work is probably a significant aspect of the moat that makes these companies comfortable releasing their models. It's very difficult to replicate what they're releasing. So the big takeaway when reading these papers is you shouldn't focus too much on just the benchmark performance or top-line stats like context size.

11:58Instead, look at the specific methods that these labs are using to achieve those results. There are tons of high performing open source models that we didn't discuss in this video, like Kimi K2 or Google Gemma 3, but when you peek under the hood of many of these, you'll find nuanced differences that I find really interesting. I hope this gives you a framework for how to understand the latest open source releases and gives you a toolkit to start tinkering with them yourself. Thanks for watching. See you in the next episode.