NLP Landscape from 1990s to 2020s

I was watching all the chatbots around and was wondering how chatbots are working, how SIRI is responding in such an accurate way day by day. As a software engineer, I got a chance to learn some COTS tools and APIs like AWS Bedrock, Assembly AI, PrivateGPT, etc. And I used them to create some POC (Proof of Concepts) like https://www.youtube.com/watch?v=Qq1GHnCkX24 and https://www.youtube.com/watch?v=tYugWsD-PoM.

But it was not enough, as I was using just APIs but not understanding the basic concepts. After that I was searching for the right start and basic foundation. Based on some initial research and learning, here is the landscape that I understood from the learning.

What is NLP?

NLP stands for Natural Language Processing, It consists of using Computer Science along with Artificial Intelligence to understand and interact with Human Language.

Need of NLP: Since humanity started using machines of any kind, there have been ways to interact with them, for example some mechanical machines where in order given them instruction you start to do certain tasks for example levers shifting etc.

Now, when we evolved we started using more easier to use methods to interact with machines like remote control based or control circuit based machines.

In the later evolution, we want to interact with machines just as we are talking in a language where we talk to each other. And this is where NLP comes into the picture!

Real World Examples:

The most common example would be contextual advertisements, where based on your hobby, that might have been read from your interaction with instagram, twitter and etc, you will see ads, and more in deep, sometimes you see ads for which you are just talking to you friends 1:1, and when you pick your phone, you see the ad in front, and get wonder that how these apps are tracking you, so scary!!

Other very common examples would be SIRI, Google one etc.

The Era

NLP has 3 main approaches in the current world:

1. Heuristic Approach, its ever existed since past 50 years and still being used
2. Machine Learning, start from 1990s and still being used
3. Deep Learning, started from 2015 and still being used

Common NLP Tasks

1. Text/Document Classification: suggesting news, cricket scores etc
2. Sentiment Analysis: Based on the tweets in twitter understand what is going on for example what people are feeling about a product or company, or in more deep sometimes you hear that elections are also forecasted.
3. Information Retrieval
4. Parts of Speech Tagging
5. Language Detection and Machine Translation: example google translate etc
6. Conversational Agents: example Swiggy chatbot, mainly of 2 types, text based and voice based.
7. Knowledge Graph and QA Systems

Challenges in NLP

1. Ambiguity : I saw a boy on the beach with my binoculars.
2. Contextual Word: I ran to the store because we ran out of milk.
3. Colloquialism and slang: Piece of cake, pulling your legs
4. Synonyms
5. Irony, Sarcasm and tonal diff: Tones, that's just what I needed today
6. Spelling Errors
7. Creativity