Crafting a parser, part 1: from source to tokens

Before any grammar rule runs, a lexer turns characters into spanned tokens — and that token shape decides everything downstream.