26 November,2023 07:32 AM IST | Mumbai | Jaison Lewis
Representation Pic
A new project called Text Generation WebUI or OobaBooga as it is rather funnily known, thanks to the name of its creator on GitHub, is the latest LLM model in town. Whatever the name, it aims to simplify how LLMs are installed on the local computer, similar to what Automatic 1111 did for Stable Diffusion, truly democratising the technology. So, let's get right into it.
>> A computer with a Nvidia graphics card and more RAM on the card, which translates to a better-performing LLM. You can technically run it on your processor as well, but be prepared to wait a few hours for every response.
>> Lots of storage space. LLM models are at least a couple of GBs each, and if you want a choice of a few LLMs, be prepared to fill up your hard drive. >> Access to the Internet to download the models and the software. Once installed, however, the LLM will not need the Internet to function.
You will need to go to OobaBooga's Github (https://bit.ly/smdooba), click on the green colour Code button and then click on Download Zip. The entire file is around 20MB. Copy this zip file to the place where you plan to use it, and make sure there is a lot of space in the destination drive. Unzip the file. Go into the directory called text-generation-webui-main.
There should be one more directory with the same name. Go into that directory as well. If you are using Windows, double-click on the start_windows.bat file. This should start the installation process. The program will ask you what graphics card you are running. If you are running a Nvidia card, select âA'. AMD GPUs are only supported on Linux installations. If you don't have a GPU, select âN'.
If you have selected âA' for Nvidia, you will have another choice to make. It will ask you if you want to use CUDA 11.8 instead of 12.1. If you have an RTX or a GTX card, select âN'. If you have an older GPU, select âY'. If you are unsure select âN'. It will start installing everything you need to run the program. Brace yourself: this is going to take a while. Once done, though, you can immediately start up the web interface. It should say, "Running one local URL: https://127.0.0.1:7860.
Type the URL in your browser.
The interface is useless without a model. There are several models to choose from, but Mistral 7B is currently my favourite model. Let's see how to install it. First, go to TheBloke's Hugging Face (https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GPTQ). Scroll down to "How to download, including from branches."
Select the text "TheBloke/Mistral-7B-Instruct-v0.1-GPTQ". Open your Text-generation-webui window. On top select the âModel' tab. Go to the Download model or LoRA section on the right and paste the text. Click on Download.
This will take a while because the file is a couple of GBs. You can use the same process to download different models, including LLAMA, Qwen, Orca or Gorilla. Read about models that suit your specific use case. It's a bit of a rabbit hole, but if you want something that works well, stick to Mistral 7B Instruct.
Once downloaded. On the left top section of the Model tab, select the model you downloaded and press Load if the model you downloaded doesn't show up. Close the Command window running in the background, and start the LLM up again by running the "start_windows.bat" file. Make sure you try loading the model again, and once it is loaded, head to the Chat tab. There, you can ask all sorts of questions, except those that show illegal or dangerous results. Just like Chat GPT, you can't ask for instructions for making a bomb or about current affairs.
Text-generation-webui is pretty self-sufficient, and you can easily upgrade it to the newest version by running the "update_windows.bat" file in the same directory as the start file. The update should be run every few weeks, as there are new and exciting additions to the program. This is, however, an upgrade tool only for the main program. For a newer model, you must explore TheBloke's hugging face and download a newer version there. There are usually good descriptions for models but feel free to explore the parent page for each model to see if the model is a good fit for you.
