Why are cpus so much "slower" than gpus

CPUs (Central Processing Units) are general purpose processors.  They can do 5 different things (5 types of operations).  They can do arithmetic, logic, jumps, reads from memory and writes to memory.  With the combination of those 5 operations, you can do anything.  CPU's have to balance the performance of those 5 operations.  If any one of those types runs slow, then the entire computer runs slow.

GPUs (Graphics Processing Units) are domain-specific processors.  Their only goal is to accelerate 2D and 3D graphics.  3D graphics are based on matrix multiplication.  Matrix multiplication can be very slow.  A single matrix multiplication requires a lot of floating point multiplication and addition operations.  In the pre-GPU era, the CPU had to perform all the individual float operations.  If you are pre-rendering a 3D movie, then the CPU limitations are not that big of a deal.  For video games, you need to render the 3D in realtime.  It is not a surprise that video game companies started to push to make it easier to do 3D realtime graphics.

The key to making 3D rendering faster is the matrix multiplication.  If you analyse the formula, you will see that many of the float operations don't actually depend on each other.  This means you can perform the float operations in parallel.  This is the key to a GPUs performance.  Instead of feeding two numbers and an instruction to a CPU over and over again, you feed two 4x4 matrices and an instruction to the GPU and the GPU performs the calculation in parallel.

As you make your 3D scene more complex, you need to perform more and more matrix operations.  You generally need to render at least 30 images every second.  To increase the quality of the 3D scene, you need a faster GPU.  It turns out that making a GPU is faster is a lot easier than making a CPU faster.  Remember those 5 types of operations that a CPU needs to keep in balance?  If you noticed, the GPU only needs to do one of those operations: the arithmetic.  Since GPU's are not general purpose processors, they can be highly optimized. 

The side effect of GPU's fast math performance is that it works for other applications as well.  It is becoming more and more common to use the GPU's power to encode 2D video or perform encryption.