I'm in exactly the same situation. Yiddish was the native tongue of my Great-Grandparents. My Grandparents spoke it fluently, as a second language, but crucially, 'NOT IN FRONT OF THE CHILDREN'. My parents only picked up a few common phrases, and I only know a few words.
This is a very sad situation and it is even sadder if other languages go the same way as Yiddish.
Incidentally, I don't know if you're aware of an excellent book: 'The Joys of Yiddish' by Leo Rosten.