There are only 5 important things in fishing:
1. Location
2. Location
3. Location
4. Location
5. Location
4 inch trout are not very clever, bigger ones are! The bigger ones are easily scared, will have the best lies that offer food and safety, and will be harder to fool with your bait and the way it is presented.
First you need to find out if there are bigger trout present. Then find out if there are spots that they prefer; could be in fast water, beneath undercut banks, overhanging trees etc. Then you need to look at how you approach the water; if you just walk up to the stream and the last you see of the bigger trout is them fleeing at high speed then this needs a much more stealthy approach.
It may be that as Marcus suggests, using a bigger bait like a worm might find bigger trout but only if you know where the trout are, and haven't scared them.