Beyond GPUs: The Future of AI Compute

Before 2020, a speaker notes that Jensen Huang prioritized maximizing floating-point operations per joule for cryptocurrency mining and convolutional neural networks rather than AI.

The emergence of large language models like GPT-2 and GPT-3 subsequently underscored a critical need for immense memory capacity and bandwidth in AI hardware.

This shift in AI requirements led Nvidia to pivot its hardware development strategy after 2020, moving beyond pure computational efficiency to prioritize architectures that specifically support transformer models.

Data centers currently incur significant energy waste, with an estimated 99.99% of energy lost during the conversion of signals between light and electricity.

Performing computation directly in light offers a solution, as it would be nearly instantaneous and consume almost no power.

However, a major technical challenge remains in finding effective methods for storing bits directly within light.

The SOMA (Sharded Optimization Mixture of Assemblies) architecture involves clustering large datasets and training smaller, specialized LSTMs on individual GPUs.

Traditional gradient estimation methods face scalability issues because error rates tend to grow linearly with the size of the model.

By sharding the model, the SOMA approach caps gradient noise, leading to more stable training even as models scale.

Photons offer a substantial advantage over electrons, exhibiting 10,000 times lower loss and significantly higher bandwidth, making them ideal for high-efficiency data processing.

Light propagation inherently supports massive parallelism because photons do not interact with one another, allowing for simultaneous data processing without interference.

The objective is to replace energy-intensive electronic matrix-vector multiplication in AI systems with passive optical modulation, significantly reducing power consumption.

Ilker's team demonstrated a proof-of-concept system that uses light propagation to perform denoising, a critical step in diffusion models for image generation.

One significant issue is the high energy cost associated with converting data between digital electronic memory and the optical domain.

Programming and calibrating physical optical devices demand active energy expenditure, which adds to the operational cost.

Implementing nonlinear activation functions, which are easily handled by digital transistors, presents a complex challenge in passive optical setups.

Initial experiments to induce nonlinearities involved intense light pulses in multimode fibers, a method that introduces additional system complexity.

Instead of dynamic weight updates during inference, the algorithm employs a static, multi-stage architecture using a fixed schedule of sub-units.

A 1,000-step diffusion process is segmented into 100-step subunits, where parameters remain constant for each segment.

This approach allows the system to operate as a diffusion model without the need for high-speed hardware weight updates.

The system demonstrated performance scaling following a power-law, consistent with observations in digital neural networks.

Currently, digital electronics remain the primary bottleneck in this setup, despite the optical components.

There is potential to explore recurrent neural network (RNN) structures to minimize the frequency of digital interface interactions.

Minimizing digital interfaces could alleviate the computational strain associated with frequent digital-to-analog conversions.

This complex organ operates on only about 20 watts of power, comparable to an iPhone charger, showcasing remarkable energy efficiency.

The brain can learn sophisticated tasks from just a few examples, a stark contrast to the vast data requirements of current AI systems, such as those used in autonomous vehicles.

Key attributes include intertwining memory and computation, using event-based communication through spikes, and enabling structural and functional adaptability in hardware.

Spiking neurons function similarly to leaky capacitors, accumulating an electrical charge until it reaches a specific threshold.

Upon reaching this threshold, the neuron 'fires' a discrete digital spike, transmitting information.

This mechanism indicates that the system is neither purely analog nor purely digital, uniquely employing both time and signal amplitude as critical variables.

This event-driven processing fundamentally differs from conventional digital logic, allowing for constant, dynamic computational activity.

More experimental, 'sci-fi' approaches, such as coupled oscillators in CMOS technology, are currently ambitious and require further research and development before widespread adoption.

In modern silicon, interconnect capacitance now surpasses gate capacitance, indicating that moving data across wires consumes more energy than the actual transistor switching.

The majority of energy expenditure in current hardware is dominated by the movement of bits rather than their processing.

Optics offers a promising solution to bypass these traditional wiring constraints, potentially revolutionizing energy efficiency in computing.

The speaker personally believes that optics will be the critical technology to address this challenge in hardware packaging.

The continued reliance on silicon as a primary compute medium stems from the industry's extensive mastery and expertise in working with it for electronic circuits.

While exotic materials like diamond substrates are being explored, silicon's dominance is more a result of established engineering capabilities than its inherent superiority over other materials like carbon.

The critical focus should be on the manifestation and implementation of electronic principles, rather than an exclusive debate between silicon and carbon.

Therefore, the preference for silicon in circuit design is largely pragmatic, leveraging decades of accumulated knowledge and infrastructure.

The encoder network in the biological system takes image data processed by a Convolutional Neural Network (CNN) and concatenates it with scalar observations.

This combined input is then passed through a Multi-Layer Perceptron (MLP) which outputs the specific frequency and amplitude values required for stimulating each neural channel.

Initially, the encoder was limited to selecting intensity parameters, but later iterations expanded its capability to select specific electrodes for stimulation.

To prevent the decoder from 'cheating' by playing the game on its own, the researchers massively undersized it, ensuring that the biological system's neural spikes were essential for gameplay.

Extensive ablation studies were conducted to confirm that the spikes themselves genuinely carried information relevant to playing the game.

An oversized silicon decoder can inadvertently learn to play the game independently, which would prevent the biological cells from actually acquiring the necessary skills.

The speaker highlights a common issue in current fly brain studies, where oversized decoders often lead to silicon-driven performance rather than demonstrating true biological intelligence.

The system never reaches a steady state of equilibrium because it involves two dynamic systems that are in constant competition for control.

In a departure from typical reinforcement learning, the researchers explicitly penalized entropy, finding it to be a helpful mechanism for guiding the neural cells' behavior.

The culture demonstrates self-organization, actively minimizing the rate of high-entropy, asynchronous stimulations.

This active control over entropy helps maintain a dynamic and responsive system rather than allowing it to settle into a static state.