Welcome to geocaching! As Mongoose39uk said, you did the right thing by leaving the container in position.
You asked, "What is the point?" Well, it is rather like an organised treasure hunt, except there's no real treasure at the end, just a box of swap items and a logbook. For many cachers, the pleasure is in the walk and the location. Each new cache takes you somewhere new, often to the sort of places that only a local would know.
If you've looked at geocaching.com, you'll probably have a much better idea of what's involved now. If you fancy trying it, but don't have a gps unit, don't let that put you off. Many cachers start without one, and in fact I found my first six using just maps and the encrypted hint which most caches have. Start with easy ones - you'll see that caches are rated on geocaching.com for difficulty and terrain, one star being easiest and five stars hardest. Go for traditionals, which are a single hidden container, rather than multis, which have a number of stages - what you found is one stage of a multi. And go for regular containers, rather than micros, which are often 35mm film containers like the one you found, and so harder to find than a tupperware box or similar.
If you need any help just ask.
Do not go gentle into that good night.
Rage, rage against the dying of the light. (Dylan Thomas)