Suggest forum posts for unhandled technical questions

[erohmensing]

based on this idea here: https://forum.rasa.com/t/wanted-new-ideas-for-the-rasa-demo-bot/873/111?u=akelad

Things to consider:

• the posts we recommend should be relatively new, we don't want to link to outdated advice
• the posts should ideally have responses to increase the probability that there is actually a solution there

[akelad]

as discussed: it may make sense to look into the algolia api first to see if we can leverage search on the docs before suggesting the forum

[rgstephens]

From Ella in today's all hands deck:

the idea for handling technical questions is:
1. send the user to the relevant docs page
2. ask if that helped
3. if not, send them to some relevant and recent(ly answered) forum posts

[erohmensing]

Also to extend on that (it seemed too much to add to all-hands)

4. if not, ask the user why it didn't help
  - the bot led them to the wrong page
  - the page was correct but did not provide the answer to their question

which helps us determine if it was a sara issue or a lack of documentation.

[akelad]

@rgstephens let me know if you need info/credentials for the algolia API

[rgstephens]

Looks like there's a discourse-algolia plugin. It will index discourse (can optionally replace existing discourse search box - can see that feature here).

Installation involves:

• Restart of Discourse
• Run Rake task to initialize the index

@erohmensing @akelad Shall we try loading this?

[akelad]

hmmm but is there not some sort of search API on the forum? I mean how else does the search bar work 🤔

[erohmensing]

@rgstephens you should be able to use their search api directly, e.g. by watching the network requests when typing how to use featurized slot into the search bar, i saw it GET from

https://forum.rasa.com/search/query?term=how%20to%20use%20featurized%20slot&include_blurbs=true

and the response included 5 topics:

{posts: [{id: 9988, name: "Rui Conti", username: "ruiconti",…},…],…}
categories: []
grouped_search_result: {more_posts: true, more_users: null, more_categories: null, term: "how to use featurized slot",…}
groups: []
posts: [{id: 9988, name: "Rui Conti", username: "ruiconti",…},…]
tags: []
topics: [{id: 3849, title: "Nested Intent Classification", fancy_title: "Nested Intent Classification",…},…]
0: {id: 3849, title: "Nested Intent Classification", fancy_title: "Nested Intent Classification",…}
1: {id: 7086, title: "Rasa core evaluation metrics", fancy_title: "Rasa core evaluation metrics",…}
2: {id: 14636, title: "Activating new form from the current form",…}
3: {id: 8864, title: "Slot filling feature is not working",…}
4: {id: 5319, title: "Long conversations get misunderstood",…}
users: []

[rgstephens]

We aren’t using Algolia API for the forum search function. We are using Algolia for Analytics and the only API key we have is for the Analytics API.

I’ll try the Discourse search.

[akelad]

As Greg and I discussed in 1:1 yesterday:

• we'll be using the algolia search API for searching the docs, so that we can send a link to the user
• if they say that didn't help, we'll use the forum search to send them suggested forum topics

[rgstephens]

Yes, sorry for my confusion on this.

With the new API key Alan got us, I have a test function that extracts the top response from the Docs.

I'll create a branch on rasa-demo and setup the Docs & Forum search.

[rgstephens]

I have a basic implementation going. At this point, I'm adding two buttons in the ActionDefaultAskAffirmation call. One with a link to the top result from the docs (using Algolia) and another with the top link from the forum. Can spend more time fine tuning this. For the forum, I agree with @erohmensing's originally thinking - something recent, a solution has been flagged, good view count.

However, I'm running into an issue returning a button with a URL but there's no href in the button as I would expect and Mr. Bot seems to support it.

I'm going to look into the Mr. Bot code to see what's going on there.

I also now have a docker-compose.yml and associated Dockerfiles that will bring up Sara along with local Rasa X and Mr. Bot containers. I'll add this to the docker-local branch.

[rgstephens]

Mr. Bot currently allows only a single button with a link. It uses a separate link Snippet message type that is sent with our utter_elements. Buttons are sent via our utter_button_message, known as quick reply messages in Mr. Bot. It also looks like Mr Bot cannot handle us sending it the buttons followed immediately with the links. If I do that, the buttons flash up and then are replaced by the links.

My thinking was to add a link to the top Docs & Forum URL's to the end of the button list provided by ActionDefaultAskAffirmation. Something like this:

I think the next step is to look in more detail at enhancing Mr. Bot. It would seem preferable to me for Mr. Bot buttons to handle markdown style URL links.

[erohmensing]

I also now have a docker-compose.yml and associated Dockerfiles that will bring up Sara along with local Rasa X and Mr. Bot containers

what do you mean here? We shouldn't have to run Mr bot at all for this. you mean just for opening it up in the widget instead of using rasa shell? not sure if that is necessary

It would seem preferable to me for Mr. Bot buttons to handle markdown style URL links

@rgstephens that would be nice at some point, but lets not let it block the implementation of this functionality. I you can include markdown-styled links in the message itself, you can do that instead of adding buttons, in a list format, similar to how we give list in response to what can you help me with

[erohmensing]

My thinking was to add a link to the top Docs & Forum URL's to the end of the button list provided by ActionDefaultAskAffirmation.

Interesting addition, I think it's a good idea. I think maybe a little OOS of this PR though, let's try to get technical_questions handled first. The less things we try to implement at once, the quicker we can get them merged in -- Iterate Quickly 😄(oh and the PRs stay smaller which is also nice for reviewing 😋)

[rgstephens]

I setup technical_questions to respond with an action that currently returns a suggested reference from the docs and one from the forum (see below). Akela said this morning that she envisions the flow would first suggest docs, then ask if they want more suggestions and suggest forum links. Is that the flow we want?

Will improve this formatting if we want a combined docs/forum response:

[akelad]

yes i think that would be the best flow! sorry if that wasn't clear.
Super excited for this feature!

[akelad]

Basically we ask the user "Did that help?"
If they say no, then we send forum links

[btotharye]

ok I closed out my other duplicate issues and mentioned them here just so we have visibility.

[rgstephens]

I thought we were consolidating unmatched utterances into a single NLU related issue which would be resolved by writing more intent utterance examples. This issue is not about writing more NLU examples.

[rgstephens]

I need to look at the following issues:

• current dialog below suggests the top 2 search results, do we want just the top result, how many?
• need to clarify distinction between techincal_question and out_of_scope, too many questions going to OOS so we don't do search
• several technical_question intent examples are dropping below the 0.8 nlu_threshold?? Here are a couple of examples:
  • which language is rasa programmed in
  • can you help me with this problem
• improve Discourse search as mentioned in comments above
• Consider always adding a Search Docs button to action_default_ask_affirmation? It would invoke action_docs_search.
• Don't forget to store and use API keys from docker-compose.override.yml
• remove duckling references since we have duckling answer
• technical_questions can be invoked from a action_default_ask_affirmation button in which case we need look back one user event to get the original user utterance for the search

[rgstephens]

Here's a screenshot example of where this is at. Items from comment above still need to be done.

[akelad]

@rgstephens could you open a PR for what you already have? And the rest of those we can tackle separately.

[btotharye]

I thought we were consolidating unmatched utterances into a single NLU related issue which would be resolved by writing more intent utterance examples. This issue is not about writing more NLU examples.

Ok maybe I'm super confused I thought anything related to being eventually answered by doc page routing was going here or being duped to here?

[erohmensing]

@rgstephens another idea to add to your list of further enhancements:

currently we have inputs like these under technical_question:

- I am getting some error
- I have a problem
- I need help with a problem
- can you help me with this problem
- can you help with some documentation
- conda threw some weird error
- conda throws some unexpected error

I think we should make a separate intent for these vague "i have a problem" inputs, and then add something to the beginning of the technical question flow in this case, like "I can help you find documentation or other issues for that. Can you describe the problem you're having?" which would hopefully then lead into the technical question flow.

[rgstephens]

@erohmensing Good idea, will split these into at least a couple of new issues when this one is closed.

[erohmensing]

@rgstephens please go ahead and split them! nice work 👍 closed by #420

(p.s. if you put closes #355 in the PR description, it will auto-close the issue when the PR is merged)

[erohmensing]

Right now we have an utterance utter_search_bar that tells people if we can't explain the nlu part they are asking about, they should use the search bar. this should trigger a technical question for the question they asked instead.