Using swap isn't always a sign you need more RAM. Typically, if you use a computer for a while or have a lot of IO operations going on, Linux will decide to swap some things to make more room for cache.
Sometimes Linux just finds that you have a bunch of inactive app memory and it can swap that out to cache way more stuff. That's just good memory management, but it's not worth buying more RAM over