Can I ever become a competent golfer?

I am 53 and have played golf most of my life, but never seriously, 5 or 6 times a year at most. In the last 3 years I have taken it up and have been playing weekly, sometimes twice a week. I have taken lessons from 3 different, highly regarded teachers in my area. I practice at the range at least once a week, usually more.

I cannot improve. My last score (yesterday 103 -) was worse than my first (over 3 years ago - 98). It's getting very frustrating and I'm thinking of just stoping playing as it's beginning to not be fun anymore. I have always been very good at sports, but I'm just not getting this one.

My short game and putting are ok, but when it comes to iron play and off the tee, I just stink. Maybe every third shot is decent. Every once in a while I will have a good round where I think I am getting it, but the next 10 rounds are abysmal. I almost never break 100.

I realize without seeing me swing it's hard to advise, but any have any ideas on why I can't improve? Is 53 just too late to try? Will some people just never get it?

Thanks!