The Theta Developers Forum recently integrated some AI features such as AI Chatbot, AI Helper and Topic Summarization. In this guide, I will be testing out the AI Chatbot features, specifically the quality of the AI responses, the limitations of the AI model, and the overall experience of using these AI features.
You can read more about the AI features here.
Let’s get right into the research.
The AI Chatbot
When using the AI chatbot feature, there are about four AI options to choose from. These options are located at the top right-hand side when the feature is triggered. We call these large language models (LLMs).
Theta also took the AI chatbot integration a step further by bringing in AI personas that are more streamlined and specific in their responses and usage. It is believed that their responses are more human and context-aware.
Back to the purpose of this section. I will walk you through my experience with AI Chatbots (LLM), specifically the GPT-4 Omni and Claude 3.5 Sonnet. Here, I am basically going to rate them on the quality of their responses.
Previously, I had the opportunity to work with React Native and integrate 360-degree image viewers like Pannellum Viewer, where I wrote this guide. The Viewers are mostly made for web use, so you need the WebView component to display them on mobile devices. I will be asking five questions that have been more like blockers to me when I had the privilege to use WebViews with React Native.
The Questions:
- Why is my React Native WebView not displaying any content, even when I have loaded my Local HTML file?
- My WebView is crashing. What could be the cause?
- Why is my WebView not executing JavaScript even after I have set javaScriptEnabled={true}?
- I am trying to send data from my React Native to WebView via props, but it is not working. How can I properly send data from React Native to my WebView?
- WebView is loading my local HTML, but when I click on the links inside the content, it isn’t working. It looks like navigation has been blocked. What can I possibly do to fix this?
The Responses:
First Question: Why is my React Native WebView not displaying any content even when I have loaded my Local HTML file?
GPT-4 Omni:
My Thoughts on GPT-4 Omni:
- The response is well-structured as it gave the responses in a step-by-step manner, highlighting all possible causes of the issue.
- I also love that GPT-4 Omni explicitly stated the platform compatibility issue. WebView behaves differently across various platforms, which is the reason for the first question.
- The part suggesting using debugging tools to ascertain if the WebView is really working and that the issue is from the local HTML content. This, to me, is more like an actionable step, and the last step developers might finally resort to if the other options don’t work. It also includes the part directing users to GitHub to get a broader view of how to solve the problem.
- Flaw: I expected that it would mention using the
onErrorattribute, which is the custom method for tracking errors in the WebView. - Overall, GPT-Omni did a great job in responding to this question.
Claude 3.5 Sonnet:
My Thoughts on Claude 3.5 Sonnet:
- First, I love that it reassures the users that it understands the question by restating the question before answering it.
- It highlighted the two options by which HTML content can be loaded into the WebView. This is an important step in debugging—trying various options to see if one method works when another doesn’t can easily help the user detect where the real issue is.
- The responses were overly personalised. Making it streamlined in its response to the question.
- Flaw: It didn’t provide a well-structured approach to detecting/solving the issue. It was more like it got the information from the Theta Forum, which can be good if the main purpose of the AI is to give information based on the Theta Forum data and not an external data source.
Second Question: My WebView is crashing. What could be the cause?
GPT-4 Omni:
My Thoughts on GPT-4 Omni:
- Just like expected, it listed out various causes of WebView crashing.
- Flaw: No code sample. It would be good if there were code samples attached to each response.
Claude 3.5 Sonnet:
My Thoughts on Claude 3.5 Sonnet:
- Listed out the possible cases but with a code sample.
- The response is personalised searching through the Theta Forum.
- Lastly, it added a call to action (CTA) asking users to give more clarifying questions since the question: WebView crashing isn’t specific; in a way, the question is vague, and a little specificity might be needed to really know why the WebView is crashing.
Third Question: Why is my WebView not executing JavaScript even after I have set javaScriptEnabled={true}?
GPT-4 Omni:
My Thoughts on GPT-4 Omni:
- It is good that it started by telling the user to confirm if they have set the
javaScriptEnabledtotrue. - Where it said to check WebView permissions, it should have suggested some possible WebView permissions that could be an issue regarding this scenario.
- It is an awesome approach to debugging using the
console.logfeature. - Lastly, the CTA added in the concluding part was great. This lets users know they can also consult platform-specific docs to grasp the concept fully.
Claude 3.5 Sonnet:
My Thoughts on Claude 3.5 Sonnet:
- Explained the common causes of JS not executing and even attached sample codes to them.
- It spots the platform/device’s additional configuration for executing JS in WebView.
- It also cited the use of the
require()method when loading the local HTML file. - Lastly, it asks the user if there are specific elements (like error messages) for a better debugging experience.
Fourth Question: I am trying to send data from my React Native to WebView via props, but it is not working. How can I properly send data from React Native to my WebView?
GPT-4 Omni:
My Thoughts on GPT-4 Omni:
- Well-structured response
- I love the fact that it took it from scratch by explaining the first step—installing webView. But there again, this might be unnecessary as I definitely know that and have passed that step.
- Flaw: I expect that it should not only suggest an approach but first try to understand my present approach so it doesn’t give me an approach I have already examined. Though there might be errors in my present code, in such cases, it will just help me debug and update it.
Claude 3.5 Sonnet:
My Thoughts on Claude 3.5 Sonnet:
- The response was very clear, explaining the key methods of communication between React Native and WebView.
- Including code samples with best practices is also very valid in tackling the problem.
Fifth Question: WebView is loading my local HTML, but when I click on the links inside the content, it isn’t working. It looks like navigation has been blocked. What can I possibly do to fix this?
GPT-4 Omni:
My Thoughts on GPT-4 Omni:
- I resonate with the introduction in the response. It shows that GPT really understands WebView. This particular question is a common issue when using WebView.
- The responses are isolated, and the responses are categorised into two based on the platform (Android and IOS). That is a really good way of tackling the issue. Plus, it added working code to each platform response it gave.
Claude 3.5 Sonnet:
My Thoughts on Claude 3.5 Sonnet:
- The response isn’t great. First, it assumed I was working on a Flutter application. It should have asked about my stack to know how to respond to me.
- But I believe it would have done better if my question had been rephrased, and I had explicitly stated that I was working with React Native.
Quality of AI Experience
Here, I will touch on user experience (UX), speed, and performance
- From the look of things, Claude 3.5 Sonnet is more personalised compared to GPT-4 Omni. Claude 3.5 Sonnet is using Theta forum information as its source of data.
- I love that you can share your AI conversations. This is a very good feature as it helps users seamlessly share responses. Also, the feature where you can update a particular response is great.
- Due to the fact that Claude 3.5 Sonnet searches before it responds, GPT-4 is faster than it when responding to prompts.
- It is very easy to make use of the AI features.
- The AI responses are very accurate. A little update or addition is needed. Also, it is very good that the AI models add CTAs at the end (the concluding part) of their responses.
Suggestion: The UX could be improved by giving users the option of choosing if they want to use the AI models with the Theta Personas or if they want to use the AI models alone. But since this AI’s purpose is to align responses to the forum data, then having the Theta Personas is great.
Limitation on AI Usage
The AI has a limit to how many AI tokens an individual can use per day. While trying the AI Chatbot feature, I got to understand that there are two types of limitations you get to hit. First, you hit a temporary limitation. Here, you still have the chance to try out the AI after a few minutes or a few hours.
Note: Claude 3.5 Sonnet hit a limitation before GPT-4. When the AI model hits a limit, GPT-4 takes less time to generate new AI tokens, but Claude 3.5 Sonnet takes more time. In the image references below, you can see that Claude 3.5 Sonnet says I should try again in 4 hours, while GPT-4 Omni says in 12 minutes.
The second type of limitation is not permanent, as you might think, but a temporary daily limitation. It comes up when you ask quite a lot of questions. In my case, I got this after asking about 13 questions to GPT-4 Omni.
Once you hit a limitation, you get an alert error message telling you how long it will take before you can try again. This can help you detect what type of limitation it is.
Note: Once you hit a limitation, the AI model doesn’t give any suitable response to your messages other than an error message telling you that you’ve hit a limitation.
