My rule for podcast listening speeds:

2x: just the information please
1x: entertain me!

Or put another way:

2x: Charlie Rose
1x: Louis CK