33.Cache 3

Here’s an experiment you can try. Your computer caches your floppy drive with main memory, and you can actually see it happening. Access a large file from your floppy — for example, open a 300-kilobyte text file in a text editor.

The first time, you will see the light on your floppy turning on, and you will wait. The floppy disk is extremely slow, so it will take 20 seconds to load the file. Now, close the editor and open the same file again. The second time (don’t wait 30 minutes or do a lot of disk access between the two tries) you won’t see the light turning on, and you won’t wait.

The operating system checked into its memory cache for the floppy disk and found what it was looking for. So instead of waiting 20 seconds, the data was found in a memory subsystem much faster than when you first tried it.

One access to the floppy disk takes 120 milliseconds, while one access to the main memory takes around 60 nanoseconds — that’s a lot faster. You could have run the same test on your hard disk, but it’s more evident on the floppy drive because it’s so slow.

To give you the big picture of it all, here’s a list of a normal caching system:

 L1 cache – Memory accesses at full microprocessor speed (10 nanoseconds, 4 kilobytes to 16 kilobytes in size)

 L2 cache – Memory access of type SRAM (around 20 to 30 nanoseconds, 128 kilobytes to 512 kilobytes in size)

 Main memory – Memory access of type RAM (around 60 nanoseconds, 32 megabytes to 128 megabytes in size)

 Hard disk – Mechanical, slow (around 12 milliseconds, 1 gigabyte to 10 gigabytes in size)

 Internet – Incredibly slow (between 1 second and 3 days, unlimited size)

As you can see, the L1 cache caches the L2 cache, which caches the main memory, which can be used to cache the disk subsystems,
and so on.

One common question asked at this point is, “Why not make all of the computer’s memory run at the same speed as the L1 cache, so no caching would be required?” That would work, but it would be incredibly expensive.

The idea behind caching is to use a small amount of expensive memory to speed up a large amount of slower, less-expensive memory. In designing a computer, the goal is to allow the microprocessor to run at its full speed as inexpensively as possible.

A 500-MHz chip goes through 500 million cycles in one second (one cycle every two nanoseconds). Without L1 and L2 caches, an access to the main memory takes 60 nanoseconds, or about 30 wasted cycles accessing memory.

When you think about it, it is kind of incredible that such relatively tiny amounts of memory can maximize the use of much larger amounts of memory. Think about a 256-kilobyte L2 cache that caches 64 megabytes of RAM. In this case, 256,000 bytes efficiently caches 64,000,000 bytes.

Why does that work? In computer science, there is a theoretical concept called locality of reference. It means that in a fairly large program, only small portions are ever used at any one time. As strange as it may seem, locality of reference works for the huge majority of programs.

Even if the executable is 10 megabytes in size, only a handful of bytes from that program are in use at any one time, and their rate of repetition is very high.