Natural Language Processing (NLP)
Natural language processing is the branch of computing concerned with making machines read, interpret and produce human language. Search engines, translation, spam filtering and the large language models behind AI assistants are all applications of it.
Three generations of method
The field began with hand-written rules: grammars and dictionaries encoding how a language works, brittle against everything the rules did not anticipate. Statistical methods replaced them, learning probabilities from large collections of text so that the most likely reading of an ambiguous sentence could be computed rather than declared. Neural networks, and then the transformer architecture, made the statistics deep enough that a single model could be trained on text at scale and then asked to do most of the older tasks at once. A large language model is the current form of that third generation.
The tasks that hide under the name
Splitting text into tokens. Tagging parts of speech. Named entity recognition, which finds the people, places, organizations, dates and amounts in a passage. Sentiment analysis. Summarisation. Question answering, which locates the span of text that answers a question. Generation, which produces new text. Two of these do most of the work when an assistant reads a business website. Entity recognition is what pulls a business name, an address and a phone number out of a page and decides whether the three pages of a site are describing one business or several. Question answering is what finds, or fails to find, the sentence that answers what the customer asked.
What follows for a page
Write for the entity recognizer and the question answerer, because they are what will read you. One name, spelled one way, with one address and one phone number, repeated identically, gives the recognizer nothing to reconcile. A heading that asks the customer’s question with a plain answer in the paragraph beneath gives the answerer a span to lift. Neither task rewards adjectives.
A common question
Which NLP tasks actually matter when an assistant reads a business website?
Two of them carry most of the load. Entity recognition pulls the name, the address and the phone number out of a page and settles whether several pages describe one business or more than one. Question answering looks for the span of text that answers what the customer asked, and either finds it or does not. Writing for those two means one name spelled one way, with one address and one number repeated without variation, and headings that pose the question with the answer stated plainly underneath, and neither task pays anything for adjectives.