Smallest unit of top-level text in an HTML page; that is, a token of text that lives outside of
results from an html parse.
Add the directory that this URL references to the traversable set; that is, to the bounding set
of path prefixes that we are willing to download from, given "limit traversal." This is called
automatically, as well as through traversable|; thus it parses to the directory level, removing
any filename portion of the URL.