resources

Links

Textbooks

• Mathematics for Machine Learning. Marc Peter Deisenroth, Aldo Faisal, and Cheng Soon Ong. (2020).
• A First Course in Machine Learning. Simon Rogers and Mark Girolami. (2016).
• Deep Learning. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. (2016).
• Dive into Deep Learning. Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. (2021).
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. Aurelien Geron (2017).
• Speech and Language Processing. Dan Juraksky and James Martin. (2025).

Neural Networks

Papers

• Learning Representations by back-propagating errors. Rumelhart, David E; Hinton, Geoffrey E.; and Williams, Ronald J. (Sept. 1985).
• Automatic differentiation in machine learning: a survey. Atilim Gunes Baydin et al. (2018).

Best Practices in Deep Learning

Papers

• Layer Normalization. Jimmy Lei Ba et al. (2016).
• Distilling the Knowledge in a Neural Network. Hinton, Geoffrey et al. (2015).

Other

• Regularization. Lecture notes.
• Regularization and lagrange multipliers. Stack overflow response. (mathematical explanation).

Language and Sequence Modeling

Papers

• Finding Structure in Time. Elman, Jeffrey L. (1990).
• Learning Long-Term Dependencies with Gradient Descent is Difficult. Bengio, Yoshua et al. (1994).
• Backpropagation through time: what it does and how to do it. Werbos, Paul J. (1989).

Other

• Understanding LSTM Networks. Neat blog post.
• The Unreasonable Effectiveness of Recurrent Neural Networks. Nice visuals for RNNs.

• A Neural Probablistic Language Model. Bengio, Yoshua et al. (2003).
  
  NOTES: Proposed using NNs on a large scale for language modeling instead of n-grams. Does some tricks with the direct concantenation of embeddings before the softmax layer and uses a fixed sliding window.

Transformers and Pretraining

Papers

• Attention is All You Need. Vaswani, Ashish et al. (2017).
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin, Jacon et al. (2018).
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Raffel, Colin et al. (2019).

Other

• The Annotated Transformer.
• Stanford CS224N Notes on Transformers. Very good lecture notes to follow the attention mechanisms leading to "self-attention."
• The Illustrated BERT, ELMo, and co. Popular blog by Jay Alammar.

Algorithms for Natural Language Processing

Papers

• A Simple LSTM model for Transition-based Dependency Parsing. Elkaref, Mohab and Bohnet, Bernd. (2017).
• Deep Unordered Composition Rivals Syntactic Methods for Text Classification. Iyyer, Mohit. (2015).
• Convolutional Neural Networks for Sentence Classification. Kim, Yoon. (2014).

Courses

• NLP Course | For You. Excellent NLP course for fundamental topics; draws heavily from research papers.
• Modern Approaches in Natural Language Processing.

Evaluation Methods in NLP

Papers

• An Empirical Investigation of Statistical Significance in NLP. Berg-Kirkpatrick, Taylor et al. (2012).

Programming

Other

• Unix for Poets.
• Scala tutorials for computational linguistics. Jason Baldridge's blog.

Deep Computer Vision

Papers

• Deep Residual Learning for Image Recognition He, Kaiming et al. (2015).

Machine Translation

Models

• Sequence to Sequence Learning with Neural Networks. Sutskever, Ilya et al. (2014).
  
  NOTES: Used 2 separate LSTMs (4 layers) and reversed the order of the source sequence for better performance. **Architecture still encodes source sequence into a fixed-length vector.**
• No Language Left Behind: Scaling Human-Centered Machine Translation. NLLB Team and Marta R. Costa-jussà et al. (2022).
• MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. Sneha Kudugunta et al. (2023).
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. Yonghui Wu et al. (2016)
• mT5: A massively multilingual pre-trained text-to-text transformer. Xue, Linting et al. (2020).
  NOTES: Extended T5 model to a multilingual setting by training on a variant of common crawl (C4). The Small and Base models may fall short on tasks on given languages compared to dedicated models. However larger models are shown empirically to close the gap.
• Neural Machine Translation by Jointly Learning to Align and Translate. Bahdanau, Dzmitry et al. (2014).
• Effective Approaches to Attention-based Neural Machine Translation. Luong, Thang et al. (2015).

Papers

• Low-resource Languages: A Review of Past Work and Future Challenges. Magueresse, Alexandre et al. (2020).
• Sequence-Level Knowledge Distillation. Kim, Yoon and Rush, Alexandar M. (2016). *pending read*

Other

• The Bender Rule. Justification for stating the languages you're working on. Fair point.
• A Neural Network for Machine Translation at Production Scale.

Large Language Models

Models

• Language Models are Few-Shot Learners. Brown, Tom B. et al. (2020).

Retrieval-Augmented Generation

Papers

• Dense Passage Retrieval for Open-Domain Question Answering. Karpukhin, Vladimir et al. (2020).

Reasoning and Agents

Papers

• Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Yao, Shunyu et al. (2023).
• Respone: Emergent Analogical Reasoning in Large Language Models. Webb, Taylor et al. (2022).
• ReAct: Synergizing Reasoning and Acting in Language Models. (PENDING READ) Yao, Shunyu et al. (2022).

Human-Computer Interaction

Other

• Value Sensitive Design.

Knowledge Representation

Papers

• What is an ontology? Guarino, Nicola et al. (2009).