A. Introduction
As a student of artificial intelligence who has only been studying the field for 3 months, I am fascinated by OpenAI’s ChatGPT, which is known as the world’s most advanced chatbot. The use of the Transformer algorithm in ChatGPT makes it possible to process continuous data structures with ease, leading to easier training and usage.
In my own personal project, called Corgiman, I have created a simple chatbot model that utilizes a classification model as its primary function. It is important to note that computers do not have the ability to understand natural language like humans do. Instead, they understand sequences of numbers and words. Therefore, a unique sequence of numbers must be created for every word in any language in order to facilitate communication. This process of converting words into vectors is called word-to-vec.
In this post, I would like to share with you my experience in creating a simple chatbot model using Python and apply it in my project, Corgiman. These knowledge I have learned by myself and besides the enthusiastic help of Developer-AI Club. I believe that this will provide valuable insight into the process of creating a chatbot and the challenges that come with it.
B. Model overview with Flask and MongoDB
Our model is clear and easy to understand, with the Front-end and Back-end communicating with each other through APIs. To support real-time communication, we used socketio library. The Back-end accesses data through MongoDB, and a JSON file is used to provide training data when a database is not available.
We will come to the actual example of this model. For the Corgiman project, we used Javascript for the Front-end and the Window’s built-in speechSynthesis engine to convert chatbot responses into speech (text-to-speech). Additionally, we employed the SpeechRecognition API to transform voice into text and send it to the server. Javascript also integrates the socketio library to enable message-sending without traditional methods like GET or POST. For the Back-end, we utilized the Flask library, a widely-used framework for deploying apps to localhost. MongoDB was selected as the database due to its use of the JSON data format, making data storage and retrieval more efficient.
Our focus here is not to delve deep into this model as it is straightforward. The purpose of this post is simply to provide an introduction to the chatbot model, so let’s move on to the next section.
C. Chatbot model
The training data will have the following structure:
C1. Training model
All steps
Prepare data for word-to-vector conversion
- Before we proceed, let’s analyze the data. The data consists of an array of information written in JSON format with the structure “Name”: “Value”. There are three components to consider: “tag” – this represents the subject of each object, such as greeting, occupation, or age; “patterns” – this is where the questions or statements for the chatbot to recognize are stored; “responses” – this is where all the answers are stored.
- Next we will have a variable named documents. It will have data as an array of tuples, in the tuples will include all the patterns we have and its tags. Patterns will be split literal into array and remove special symbols then lowercase. For example we will have an element like ([“hi”, “there”], “greeting”) or ([“morning”], “greeting”)
- Through the data that we have, we will separate all the patterns we have and put in the words variable. Of course it must be stripped of the previous special character and lowercase.
- Finally, all tags will also be saved into a variable called classes, after being cleaned
1 2 3 4 5 6 7 8 | <span class="token keyword">for</span> intent <span class="token keyword">in</span> intents<span class="token punctuation">[</span><span class="token string">"intents"</span><span class="token punctuation">]</span><span class="token punctuation">:</span> <span class="token keyword">for</span> pattern <span class="token keyword">in</span> intent<span class="token punctuation">[</span><span class="token string">'patterns'</span><span class="token punctuation">]</span><span class="token punctuation">:</span> word_list <span class="token operator">=</span> nltk<span class="token punctuation">.</span>word_tokenize<span class="token punctuation">(</span>pattern<span class="token punctuation">)</span> words<span class="token punctuation">.</span>extend<span class="token punctuation">(</span>word_list<span class="token punctuation">)</span> documents<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">(</span>word_list<span class="token punctuation">,</span> intent<span class="token punctuation">[</span><span class="token string">'tag'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token keyword">if</span> intent<span class="token punctuation">[</span><span class="token string">'tag'</span><span class="token punctuation">]</span> <span class="token keyword">not</span> <span class="token keyword">in</span> classes<span class="token punctuation">:</span> classes<span class="token punctuation">.</span>append<span class="token punctuation">(</span>intent<span class="token punctuation">[</span><span class="token string">'tag'</span><span class="token punctuation">]</span><span class="token punctuation">)</span> |
Convert word to vector
- For computers to understand, we need to convert words into vectors. The crucial aspect of this model lies in this step, so it is imperative that we present it clearly and effectively.
- First we will iterate through all the elements in the documents.
- Next we will have a vector of 0 elements whose length is equal to the length of the words variable. In places where the words variable matches the words in the pattern, it will be changed to 1. For ease of understanding we will come to the example, we will have the words variable as [“hi”, “morning”, “there” “, “bye”] then we will first create a vector of [0,0,0,0], compare with pattern [“hi”, “there”] then at index position 0 and 2 in words will coincide together, we will have a new vector is [1,0,1,0] (*)
- The same goes for tags, the tag is the label for each pattern that we give the machine to determine. Similar to the previous step, the position where the tag matches in the classes variable will have the value 1. For example, we have classes as [“greeting”, “occupation”, “age”, “name”], so tag “age” will be vector [0,0,1,0] (**)
- So we have training data as vectors, it has a structure of two-dimensional array [[vector_pattern, vector_tag],…]. Where vector_pattern is the vector created at (*) and vector_tag is created at (**)
1 2 3 4 5 6 7 8 9 10 11 | <span class="token keyword">for</span> document <span class="token keyword">in</span> documents<span class="token punctuation">:</span> bag <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> word_patterns <span class="token operator">=</span> document<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> word_patterns <span class="token operator">=</span> <span class="token punctuation">[</span>lemmatizer<span class="token punctuation">.</span>lemmatize<span class="token punctuation">(</span>word<span class="token punctuation">.</span>lower<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token keyword">for</span> word <span class="token keyword">in</span> word_patterns<span class="token punctuation">]</span> <span class="token keyword">for</span> word <span class="token keyword">in</span> words<span class="token punctuation">:</span> bag<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span> <span class="token keyword">if</span> word <span class="token keyword">in</span> word_patterns <span class="token keyword">else</span> bag<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">)</span> output_row <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span>output_empty<span class="token punctuation">)</span> output_row<span class="token punctuation">[</span>classes<span class="token punctuation">.</span>index<span class="token punctuation">(</span>document<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">1</span> training<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">[</span>bag<span class="token punctuation">,</span> output_row<span class="token punctuation">]</span><span class="token punctuation">)</span> |
Training
- After we have finished preparing the data and turning it into vectors, we will train the model. The type of training will depend on each model and individual requirements. For Corgiman, I use a Sequential model which consists of 4 layers, the first layer has 256 neurons, the next layer has 128 neurons, the third layer has 64 neurons and the last layer has the number of neurons equal to the length of the output.
1 2 3 4 5 6 7 8 9 10 11 12 | model <span class="token operator">=</span> Sequential<span class="token punctuation">(</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dense<span class="token punctuation">(</span><span class="token number">256</span><span class="token punctuation">,</span> input_shape<span class="token operator">=</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>train_x<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">,</span> activation<span class="token operator">=</span><span class="token string">'relu'</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dropout<span class="token punctuation">(</span><span class="token number">0.5</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dense<span class="token punctuation">(</span><span class="token number">128</span><span class="token punctuation">,</span> activation<span class="token operator">=</span><span class="token string">'relu'</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dropout<span class="token punctuation">(</span><span class="token number">0.5</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dense<span class="token punctuation">(</span><span class="token number">64</span><span class="token punctuation">,</span> activation<span class="token operator">=</span><span class="token string">'relu'</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dropout<span class="token punctuation">(</span><span class="token number">0.5</span><span class="token punctuation">)</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>Dense<span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>train_y<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span> activation<span class="token operator">=</span><span class="token string">'softmax'</span><span class="token punctuation">)</span><span class="token punctuation">)</span> sgd <span class="token operator">=</span> gradient_descent_v2<span class="token punctuation">.</span>SGD<span class="token punctuation">(</span>lr<span class="token operator">=</span><span class="token number">0.01</span><span class="token punctuation">,</span> decay<span class="token operator">=</span><span class="token number">1e-6</span><span class="token punctuation">,</span> momentum<span class="token operator">=</span><span class="token number">0.9</span><span class="token punctuation">,</span> nesterov<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span> model<span class="token punctuation">.</span><span class="token builtin">compile</span><span class="token punctuation">(</span>loss<span class="token operator">=</span><span class="token string">'categorical_crossentropy'</span><span class="token punctuation">,</span> optimizer<span class="token operator">=</span>sgd<span class="token punctuation">,</span> metrics<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'accuracy'</span><span class="token punctuation">]</span><span class="token punctuation">)</span> |
Evaluate
Result
Visual
C1. Prediction model
All steps
Clean up sentence
- Remove special characters (., !, @, #, ?, …). For example, “how old are you ?” => “how old are you”
- Eliminate repeated words. For example, “he isn’t a doctor, is he” => “he isn’t a doctor, is”
- Split sentences into arrays of words. For example, “what is your name ?” => [“what”, “is”, “your”, “name”]
1 2 3 4 5 | <span class="token keyword">def</span> <span class="token function">clean_up_sentence</span><span class="token punctuation">(</span>sentence<span class="token punctuation">)</span><span class="token punctuation">:</span> sentence_words <span class="token operator">=</span> nltk<span class="token punctuation">.</span>word_tokenize<span class="token punctuation">(</span>sentence<span class="token punctuation">)</span> sentence_words <span class="token operator">=</span> <span class="token punctuation">[</span>lemmatizer<span class="token punctuation">.</span>lemmatize<span class="token punctuation">(</span>word<span class="token punctuation">)</span> <span class="token keyword">for</span> word <span class="token keyword">in</span> sentence_words<span class="token punctuation">]</span> <span class="token keyword">return</span> sentence_words |
Text to vector
- Create an array with n elements 0, where n is the length of the array containing all the trained words. For example, we have to 6 words trained, we will have an array like [0,0,0,0,0,0]
- At positions where the words in that sentence match in the array of all words, the value will be updated to 1. For example, the array will be [0,1,0,0,1,0]
1 2 3 4 5 6 7 8 9 | <span class="token keyword">def</span> <span class="token function">bag_of_word</span><span class="token punctuation">(</span>sentence<span class="token punctuation">)</span><span class="token punctuation">:</span> sentence_words <span class="token operator">=</span> clean_up_sentence<span class="token punctuation">(</span>sentence<span class="token punctuation">)</span> bag <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">*</span> <span class="token builtin">len</span><span class="token punctuation">(</span>words<span class="token punctuation">)</span> <span class="token keyword">for</span> w <span class="token keyword">in</span> sentence_words<span class="token punctuation">:</span> <span class="token keyword">for</span> i<span class="token punctuation">,</span> word <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>words<span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token keyword">if</span> word <span class="token operator">==</span> w<span class="token punctuation">:</span> bag<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">1</span> <span class="token keyword">return</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>bag<span class="token punctuation">)</span> |
Predict
- In the previous section we did the model training. Now we just need to take that model out to use
- We will have an array of values like the figure above, which is the exact proportion of the sentence in which tag
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | <span class="token keyword">def</span> <span class="token function">predict_class</span><span class="token punctuation">(</span>sentence<span class="token punctuation">)</span><span class="token punctuation">:</span> bow <span class="token operator">=</span> bag_of_word<span class="token punctuation">(</span>sentence<span class="token punctuation">)</span> res <span class="token operator">=</span> model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>np<span class="token punctuation">.</span>array<span class="token punctuation">(</span><span class="token punctuation">[</span>bow<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">;</span> ERROR_THRESHOLD <span class="token operator">=</span> <span class="token number">0.25</span> results <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">[</span>i<span class="token punctuation">,</span> r<span class="token punctuation">]</span> <span class="token keyword">for</span> i<span class="token punctuation">,</span>r <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>res<span class="token punctuation">)</span> <span class="token keyword">if</span> r <span class="token operator">></span> ERROR_THRESHOLD<span class="token punctuation">]</span> <span class="token comment"># => result = the value > 0.25</span> results<span class="token punctuation">.</span>sort<span class="token punctuation">(</span>key <span class="token operator">=</span> <span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> reverse<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span> <span class="token comment"># get the highest value [[2, 0.9970567]]</span> return_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token comment"># check meaningful sentences</span> <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>results<span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">:</span> return_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token string">'intent'</span><span class="token punctuation">:</span> <span class="token string">'no answer'</span><span class="token punctuation">}</span><span class="token punctuation">]</span> <span class="token keyword">elif</span> results<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator"><</span> <span class="token number">0.6</span><span class="token punctuation">:</span> return_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token string">'intent'</span><span class="token punctuation">:</span> <span class="token string">'no answer'</span><span class="token punctuation">}</span><span class="token punctuation">]</span> <span class="token keyword">else</span><span class="token punctuation">:</span> <span class="token keyword">for</span> r <span class="token keyword">in</span> results<span class="token punctuation">:</span> return_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">'intent'</span><span class="token punctuation">:</span> classes<span class="token punctuation">[</span>r<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'probability'</span><span class="token punctuation">:</span> <span class="token builtin">str</span><span class="token punctuation">(</span>r<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token comment"># r[0] is index, r[1] is value.</span> <span class="token comment"># example: [{'intent': 'greetings', 'probability': '0.9970567'}]</span> <span class="token keyword">return</span> return_list |
Get response
- We will find a way to get the maximum value in that array, and its index
- Its index position is also the predicted tag position in the array containing all the tags, we will get the correct tag.
- With the tag found, we will find the appropriate responses in the data
- Randomly select an answer from the responses array
1 2 3 4 5 6 7 8 | <span class="token keyword">def</span> <span class="token function">get_response</span><span class="token punctuation">(</span>intents_list<span class="token punctuation">,</span> intents_json<span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token comment"># get the first value ( the value has probability highest)</span> tag <span class="token operator">=</span> intents_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'intent'</span><span class="token punctuation">]</span> <span class="token keyword">for</span> intent <span class="token keyword">in</span> intents_json<span class="token punctuation">:</span> <span class="token keyword">if</span> intent<span class="token punctuation">[</span><span class="token string">'tag'</span><span class="token punctuation">]</span> <span class="token operator">==</span> tag<span class="token punctuation">:</span> <span class="token keyword">return</span> random<span class="token punctuation">.</span>choice<span class="token punctuation">(</span>intent<span class="token punctuation">[</span><span class="token string">'responses'</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token keyword">return</span> <span class="token string">"I can't get it"</span> |
D. Advantages and Disadvantages
Advantages
- Flask provides a simple and clear model for building chatbots, making it easy for developers to understand and implement.
- The chatbot runs stably on i5 chip configuration or higher, ensuring good performance for your users.
- The training data is stored in JSON format, which is a plain text format that is easy to use and send to any server. JSON is also commonly used to exchange text data between the browser and the computer.
- The chatbot training process is efficient and doesn’t take much time, allowing you to get your chatbot up and running quickly.
- As the chatbot is deployed on Flask, it’s easy to extend and add new features or customizations.
- MongoDB, a NoSQL database, provides the flexibility and ease of storing data, allowing you to manage and organize your data in a way that meets your needs.
- MongoDB also offers robust security features to protect your data, ensuring that your chatbot and its data are safe and secure.
Disadvantages
- Lack of diverse and comprehensive training data
- Overloading the model with too much training data can decrease its accuracy
- Longer sentences for the chatbot to analyze and predict will negatively impact its response accuracy
- Chatbots built on Flask may require more resources and potentially impact system performance
- MongoDB, being a NoSQL database, offers flexible data storage but also requires careful data management and security measures
- As Flask is an open-source platform, there may be security vulnerabilities or missing features, requiring extra caution when implementing.
Conclusion
In this post, I hope to introduce you to a useful tool, a chatbot. I have built the chatbot in the simplest way possible to help you understand better. However, to achieve higher accuracy, each step needs to be optimized further.
This is my first article and there may be mistakes or misunderstandings, I welcome feedback and corrections from all of you. Thank you very much for reading and providing feedback.
Corgiman is the project where I applied this model, You can find it here
If you are interested or have any questions about the article, please leave your comments below the comment section ^^. Thank you very much for your follow up.