Daniel, I'm 6ft tall and 14st, and I know how to look after myself, but I wouldn't bother with the scum. If possible report them to the fishery owner or club responsible for the fishery. If they appear regularly try ringing the local police station, they should have a community officer for your area, and the chances are he will know who the scrotes are. He probably won't be able to do much, but might be able to warn them off. The risk of taking action yourself and getting into trouble isn't worth it, you'll come off worse in the end.
Try to get some friends to go fishing with you, or see if anyone one in your club (or on this site) lives nearby and can meet you there. These kind of bullies are usually cowards, and any sign of resistance frightens them off.
I only wish I and a few of my mates could help - I know about 30 Karate blackbelts!