It's apparently the chlorophyll that gives celery the bitter taste.

Maybe that's why it stops tasting so vile when it gets overcooked.