top of page

The AI Chip Race Heats Up Yet Again: Etched's Transformer Gambit

6/30/24

Editorial team at Bits with Brains

The battle for AI chip supremacy is intensifying. Etched.ai, a plucky startup founded by Harvard dropouts, has thrown down the gauntlet to challenge the reigning king of AI hardware - Nvidia.

Key Takeaways
  • Etched's Sohu chip promises 20x faster AI processing than Nvidia GPUs

  • Custom-built for transformer models, but limited to that architecture

  • Potential game-changer for AI costs and capabilities if claims hold true

  • Executives should watch closely but proceed cautiously with adoption


Etched.ai Takes Aim at Nvidia

A new contender has emerged to challenge Nvidia's dominance in AI chips. Etched, a startup founded by Harvard dropouts, recently unveiled its Sohu chip - a custom-built application-specific integrated circuit (ASIC) designed exclusively for transformer models.


With bold claims of 20x faster processing than Nvidia's top GPUs, Etched has captured the attention of the AI industry and investors alike. But what exactly is this new chip, how does it work, and what are the potential implications for businesses looking to harness the power of generative AI?


Let's break down the key aspects of Etched's Sohu chip and examine how it stacks up against the competition.


The Transformer-Only Approach

At its core, the Sohu chip takes a radically different approach compared to general-purpose GPUs. While chips like Nvidia's H100 are designed to handle a wide variety of AI workloads, Etched has gone all-in on transformer models - the architecture behind today's most advanced language models like GPT-4.


By "burning" the transformer architecture directly into silicon, Etched claims to achieve dramatically higher efficiency for these specific workloads. The chip strips away components needed for other AI tasks, dedicating more of its transistors to the matrix multiplication operations that form the backbone of transformer computations.


This laser focus allows for some impressive theoretical gains:

  • 20x faster processing than Nvidia H100 GPUs

  • 10x performance boost over Nvidia's next-gen Blackwell GPUs

  • 90% FLOP utilization vs 30-40% for general-purpose chips

If these claims hold up in real-world testing, the Sohu chip could significantly reduce the hardware costs and energy consumption associated with running large language models.


The Pros and Cons

As with any specialized technology, Etched's approach comes with both advantages and limitations. 


Here are some key pros and cons:


Advantages:

  • Dramatically faster inference for transformer models

  • Potential for major cost savings on AI infrastructure

  • Enables new real-time AI applications previously infeasible

  • Open-source software stack for customization

Disadvantages:

  • Limited to transformer architecture - can't run other AI models

  • Unproven technology from a young startup

  • Potential obsolescence if transformer models fall out of favor

  • Less flexible than general-purpose GPUs

For businesses heavily invested in transformer-based AI, the potential upsides are significant. The ability to run models like GPT-4 and Claude Opus at a fraction of the current cost could open up new use cases and make AI more accessible. However, the risks of betting on an unproven technology shouldn't be overlooked.


Price-Performance Comparison

While exact pricing for the Sohu chip hasn't been revealed, Etched claims it will offer major cost savings compared to current GPU-based solutions.


Here’s a hypothetical comparison: An 8 node cluster of Nvidia H100 chips costs around $200,000+ and generates approximately 25,000 tokens per second. The power draw is high. By comparison, an 8 node cluster of Etched Sohu chips generates approximately 500,000 tokens per second at a much lower power draw.


The potential for 20x performance at lower power consumption is certainly appealing. However, without concrete pricing information, it's difficult to assess the true price-performance ratio.

Organizations should keep a close eye on real-world benchmarks and pricing details as they emerge.


Implementation Challenges

Businesses considering adopting Etched's technology, need to be aware of the following several hurdles:

  1. Software ecosystem: While Etched promises an open-source stack, the ecosystem around their chip is still nascent compared to established platforms.

  2. Integration: Adapting existing AI pipelines to work with the new hardware could require significant engineering effort.

  3. Talent: Finding developers experienced with this new architecture may prove challenging in an already tight AI labor market.

  4. Risk management: Relying on a single startup for critical AI infrastructure carries inherent risks that must be carefully weighed.


The Road Ahead

Etched's entry into the AI chip market highlights the rapid pace of innovation in this space. While their transformer-only approach is intriguing, it's too early to declare victory over established players like Nvidia.


For executives keeping tabs on AI developments, the Sohu chip represents an interesting case study in specialization vs. flexibility. The potential for massive efficiency gains is tantalizing but comes at the cost of versatility.


As generative AI continues to evolve at breakneck speed, maintaining a balanced perspective is key. Etched's technology could be a game-changer, but it's just one piece of a complex puzzle. Successful AI implementation requires more than just raw computing power - it demands an overall  strategy encompassing data, talent, ethics, and business alignment.


The coming months will be critical as Etched moves from bold claims to real-world deployments. Savvy leaders should watch closely, but proceed cautiously.


FAQs


Q: Can the Sohu chip be used for AI training, or only inference?

A: Currently, Etched is focusing on inference workloads. AI training would likely still require traditional GPUs.


Q: How does Etched's chip compare to other AI-focused startups like Cerebras or Graphcore?

A: While those companies offer more general-purpose AI accelerators, Etched has taken a hyper-specialized approach focusing solely on transformers. This allows for potentially greater efficiency, but less flexibility.


Q: What types of businesses could benefit most from this technology?

A: Companies heavily invested in natural language processing, chatbots, and other transformer-based applications could see the biggest gains. However, the technology is still unproven in production environments.


Q: Are there any regulatory concerns around adopting this new chip architecture?

A: As with any AI technology, companies need to ensure compliance with data privacy and security regulations. The specialized nature of the chip doesn't inherently create new regulatory hurdles.


Q: How soon could we see Etched's technology available for commercial use?

A: Etched aims to launch their Sohu Developer Cloud for testing in the coming months. Wider commercial availability will likely depend on the success of these initial deployments.


Sources:

[1] https://www.etched.com

[2] https://www.reddit.com/r/LocalLLaMA/comments/1dq8o02/sohu_ai_chip_how_does_it_work/

[3] https://www.techpowerup.com/323887/ai-startup-etched-unveils-transformer-asic-claiming-20x-speed-up-over-nvidia-h100

[4] https://techcrunch.com/2024/06/25/etched-is-building-an-ai-chip-that-only-runs-transformer-models/

[5] https://www.tomshardware.com/tech-industry/artificial-intelligence/sohu-ai-chip-claimed-to-run-models-20x-faster-and-cheaper-than-nvidia-h100-gpus

[6] https://www.theregister.com/2024/06/26/etched_asic_ai/

[7] https://cryptoslate.com/is-the-nvidia-top-in-as-etched-launches-asic-for-llms-10x-faster-than-h100-gpus/

[8] https://techfundingnews.com/ai-chip-race-heats-up-etched-secures-120m-to-take-on-nvidia/

[9] https://www.signitysolutions.com/blog/challenges-in-generative-ai-implementation

[10] https://aws.amazon.com/ai/generative-ai/use-cases/

[11] https://www.economist.com/business/2024/06/08/the-war-for-ai-talent-is-heating-up

[12] https://www.eweek.com/artificial-intelligence/generative-ai-enterprise-use-cases/

[13] https://www.computerworld.com/article/2086920/the-ai-talent-shortage-can-companies-close-the-skills-gap.html

Sources

© 2023 Analytical Outcomes LLC, All Rights Reserved

bottom of page