Here is my perspective.

There are some Major doctrines and Minor doctrines. The Major doctrines are the dividing line. One of them is the Trinity or the deity of Christ. If you do not believe in that, It does not make sense to attend a church that does and vice versa.

However, "6000" years for the age of the earth is truly a minor doctrine. Each christian in the church can have his or her opinion. In fact, the pastor too is allowed to have his opinion about this as well and it should not bother you at all. This brings me to another point.

Why were you bothered by the 6000 speech? If you know that the evidence are empty, you should smile and forget about it. But if it makes you uncomfortable, shouldn't you pay even closer attention?

Having said this, I strongly believe 6000 years for the age of the earth is neither scientific nor biblical. This is true even for the Young Earth view. If one added all the years in the bible, one would get far more years than 6000 years even with the literal 24 hours 6-days of creation.