Enormous amounts of ever-changing  knowledge are available online in diverse textual styles and diverse formats. Recent advances in deep learning algorithms and large-scale datasets are spurring progress in many Natural Language Processing (NLP) tasks, including question answering. Nevertheless, these models cannot scale up when task-annotated training data are scarce. This talk presents my lab’s work toward building general-purpose models in NLP and how to systematically evaluate them. First, I present a general model for two known tasks of question answering in English and multiple languages that are robust to small domain shifts.  Then, I show a meta-training approach that can solve a variety of NLP tasks with only using a few examples and introduce a benchmark to evaluate cross-task generalization. Finally, I discuss neuro-symbolic approaches to address more complex tasks by eliciting knowledge from structured data and language models.