Try typing 'free' at a command prompt, you should get output like
[aravi@ironclad Fishing]$ free
total used free shared buffers cached
Mem: 515848 510832 5016 0 20504 253168
-/+ buffers/cache: 237160 278688
Swap: 1847424 70080 1777344
The Linux kernel caches recently accessed files in RAM so that they're faster to load. It will free up this space if a current process requires it (as of kernel 2.6, this is slightly different - more on that later). This means that memory usage may appear higher than it is. The -/+ Buffers/Cache line reports how much memory is used by actual processes and how much is ACTUALLY free.
Now, to how kernel 2.6 is slightly different. When new memory is needed the Linux kernel has two options - free up some space used for cached files or swap data in memory out to disk. In 2.4 it would resort to swapping only after it had pushed out all cached files, in 2.6 it may do one or the other (controlled via /proc/vm/swappiness).
You might want to check if that is your issue, or if an app really is consuming that much memory. In my case, an application failed to allocate memory - so I know there is another issue at work. Running top and pressing shift-M will give you a listing of which applications are currently consuming the most memory.