Technical and practical answers
Technically, on your downswing, the weight of the head actually forces the shaft to flex downwards, so that the toe of the club is closer to the ground than at address (hope that makes sense). In fact, I had my irons adjusted for the correct lie. Funny thing was, at address, the toe pointed up (just like you), however, when I swung, the impact point on the sole was nearer to the toe. So I had my clubs adjusted to 2* upright.
On a practical matter, this really shouldn't matter too much, since lie angle is more important when the club hits the turf, which you shouldn't be doing with a driver. However, there might be different results on mis-hits.
I wouldn't worry about it. I would rather focus on getting the right length and stiffness of the shaft for your swing.