Yes, plastic will take on the smell of fermenting wort. Think about pickle buckets. Be careful when scrubbing out your buckets, you create scratches that can be the perfect hiding place for undesireable little monsters....beer spoilers.
The easiest thing to do is just to soak in PBW or something like that over night, then spray out. The vast majority of trub and stuff stuck the sides with just wash out. Then you can wipe out any remaining stuff with a soft cloth or paper towels.
I'll also add that you may want to buck up and buy one carboy to use as a secondary. Plastic breathes and is not the best thing for conditioning or aging beer.