Lab Instructions

Lab Instructions for Local LLM

Prerequisites

1. Checked out Lab Repo from GitHub CS595 Lab Repo

2. Pull the code to get the latest updates for GitHub Repo

3. Python version 3.10 or later

4. A Python virtual environment to link and use for the project

5. Download Models Meta-Llama-3-8B-Instruct-Q4_K_S.gguf and Phi-3-mini-4k-instruct.Q6_K.ggufand place them in <Project Root>/labs/local_llm folder

Instructions

1. Open <Project Root>

2. Activate the python virtual environment

3. Go to /labs/local_llm

4. Install Requirements pip install -r requirements.txt

5. Implement the below code in file local_llm.ipynb:

   1. def chunk_text(text, chunk_size=500, chunk_overlap=50)

   2. def llama_embed_text(text)-> np.ndarray

   3. Logic to Process embeddings for all text chunks : for i, chunk in enumerate(text_chunks)

   4. def search_similar_chunks(query, k=3)

   5. def run_llm(prompt: str)

   6. def run_llm_with_pdf_knowledge(user_query, k=3)

6. Run the local_llm.ipynb using Cursor, VSCode or any other jupyter supported editors

Submission

1. Short Report (1–2 pages PDF)

   1. Summarise your local LLM setup steps.

   2. Document difficulties or errors you encountered, and how you resolved them.

   3. Highlight your key observations, especially comparing queries with and without PDF

      context:

      1. For instance, ask: “What is the patient's ID?” or “What is the patient's last recorded

         blood pressure?”

      2. Observe whether your LLM can correctly retrieve and interpret those fields.

2. Completed Notebook/Script

   1. Include code showing how you loaded the PDF, chunked the text, generated

      embeddings, and queried the model.

3. Evidence of Successful Queries

   1. Provide screenshots or copied console outputs that demonstrate your queries and the

      model’s responses.

   2. Clearly identify which responses used contextual knowledge and which did not.

Links

1. Local-LLM GitHub Repository

2. Meta-Llama-3-8B-Instruct-Q4_K_S.gguf

3. Phi-3-mini-4k-instruct-Q6_K.gguf

4. Llama.cpp GitHub Repository

5. FAISS GitHub Repository

Reference Tutorials

1. PDF Chatbot Tutorial by Helicone

2. “Build Your Own RAG App” by Nasser Maronie