Learning from the Past while Looking Ahead