In recent months as I've gotten back into the game, I haven't worried much about keeping score, so what I did was probably more defined as "recreation"... On the other hand, I feel once you keep score, then it becomes a measuring device of you against other golfers or against the course itself. That is a sport.
Quite a number of years ago the same argument came up. I remember Frank Beard being the leading money winner that year and to some extent, agreeing that he himself wasn't an athlete. His enormous respect for football, baseball and basketball players was such that he couldn't see himself as their physical equal, thus he defined himself as one who played a "game" and differentiated golf from sport on those grounds.
These days, with fitness having become so important to the PGA Tour players, the fitness trailer is used as frequently as the practice range at every tour stop. I think it's safe to say the physical capability of golfers has increased over the years to the point that Frank Beard would admit the current crop of players are, in fact, athletes, thus they play a sport.