The historic fort and surrounding neighborhood are interesting to walk around if you like history.
Will you have a car? I recommend getting one for at least 2 days and going east on the island to El Yunque Nat'l Forest (http://www.fs.usda.gov/elyunque). Beautiful hiking, waterfalls, swimmable streams, lots of nature.
From there you can rent snorkeling gear from one of the beach kiosks (there are tons) at Fajardo. At night the kayak tour of the bioluminescent bay is really something.
A day at Fajardo and a day at El Yunque sounds about right.