I don’t know much about Bard specifically, but I would second sferik’s comment to apply to many schools. I know the average is 11, but that’s over the top to me. I was surprised by how my top choices shifting during the final year (learning more detail about each of the schools, visits, and as above, the financial piece can play a role), so I was glad I had applied to places to leave the options open. If you haven’t already, perhaps you can try to identify what you love about Bard (the people/culture, the campus layout, the flexibility, the kinds of courses, the size, or whatever combination of factors it is), and then ask a college counselor about similar schools (with similar/higher rankings) you might consider.
The fact that you’re questioning whether it’s a ‘good enough’ school, and considering people’s impressions that you’re undercutting yourself, says to me that that might be important to you. Maybe there are other schools you’d love that will be more challenging or rigorous? Sometimes reputation is bull, but sometimes it does reflect certain elements/truths about places. You sound like you are a hard-working/high-scoring student, so it may be important to find a place where you can really grow, shine, reach your potential…... I applied to a couple schools that were long-shots, a couple that seemed just about right, and a couple ‘safeties’ (where I had a solid chance of acceptance based on my grades/scores) – all of which I’d want to attend, of course. I got in to half of the places, so I knew my aim was right (not undercutting nor overestimating myself), and I attended my first choice. Everyone is different, but that strategy worked for me.
As for reputation, it’s true that every school has it’s different connotations to people, but overall there are schools considered better/stronger/more competitive/more rigorous/etc and the opposites. Having attended several schools, I have experienced the positive and negative reactions and effects of both kinds of reputations. Having gone to a very high-ranked undergrad did help me not only get a great education with great resources, but years later has helped with jobs, first impressions, applications, etc. But this all depends on your field—a school that may not be very well known/well ranked may have a huge strength in the department you want to major in, and that should carry a lot of weight. I can imagine it’s more difficult if you have no idea about your major (which is how I narrowed down my choices), but then I guess it has to do with finding a place with many options and generally strong programs in many departments (not limited or uneven areas). Even if there are a handful of subjects that interest you, I would strongly recommend looking at schools strengths/rankings in those areas to get a better sense of the programs and where you might fit best.
All that said, go to the school you love if at all possible!