Problem Statement¶
Business Context¶
The healthcare industry is rapidly evolving, with professionals facing increasing challenges in managing vast volumes of medical data while delivering accurate and timely diagnoses. The need for quick access to comprehensive, reliable, and up-to-date medical knowledge is critical for improving patient outcomes and ensuring informed decision-making in a fast-paced environment.
Healthcare professionals often encounter information overload, struggling to sift through extensive research and data to create accurate diagnoses and treatment plans. This challenge is amplified by the need for efficiency, particularly in emergencies, where time-sensitive decisions are vital. Furthermore, access to trusted, current medical information from renowned manuals and research papers is essential for maintaining high standards of care.
To address these challenges, healthcare centers can focus on integrating systems that streamline access to medical knowledge, provide tools to support quick decision-making, and enhance efficiency. Leveraging centralized knowledge platforms and ensuring healthcare providers have continuous access to reliable resources can significantly improve patient care and operational effectiveness.
Common Questions to Answer
1. Diagnostic Assistance: "What are the common symptoms and treatments for pulmonary embolism?"
2. Drug Information: "Can you provide the trade names of medications used for treating hypertension?"
3. Treatment Plans: "What are the first-line options and alternatives for managing rheumatoid arthritis?"
4. Specialty Knowledge: "What are the diagnostic steps for suspected endocrine disorders?"
5. Critical Care Protocols: "What is the protocol for managing sepsis in a critical care unit?"
Objective¶
As an AI specialist, your task is to develop a RAG-based AI solution using renowned medical manuals to address healthcare challenges. The objective is to understand issues like information overload, apply AI techniques to streamline decision-making, analyze its impact on diagnostics and patient outcomes, evaluate its potential to standardize care practices, and create a functional prototype demonstrating its feasibility and effectiveness.
Data Description¶
The Merck Manuals are medical references published by the American pharmaceutical company Merck & Co., that cover a wide range of medical topics, including disorders, tests, diagnoses, and drugs. The manuals have been published since 1899, when Merck & Co. was still a subsidiary of the German company Merck.
The manual is provided as a PDF with over 4,000 pages divided into 23 sections.
Installing and Importing Necessary Libraries and Dependencies¶
In [ ]:
# Installation for GPU llama-cpp-python
# uncomment and run the following code in case GPU is being used
# !CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.85 --force-reinstall --no-cache-dir -q
# Installation for CPU llama-cpp-python
# uncomment and run the following code in case GPU is not being used
!CMAKE_ARGS="-DLLAMA_CUBLAS=off" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.85 --force-reinstall --no-cache-dir -q
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 40.9 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 151.7 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 103.1 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 68.1 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 114.7 MB/s eta 0:00:00 Building wheel for llama-cpp-python (pyproject.toml) ... done ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. opencv-python-headless 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.5 which is incompatible. opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-exporter-otlp-proto-common==1.37.0, but you have opentelemetry-exporter-otlp-proto-common 1.38.0 which is incompatible. opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-proto==1.37.0, but you have opentelemetry-proto 1.38.0 which is incompatible. opentelemetry-exporter-otlp-proto-http 1.37.0 requires opentelemetry-sdk~=1.37.0, but you have opentelemetry-sdk 1.38.0 which is incompatible. google-adk 1.19.0 requires opentelemetry-api<=1.37.0,>=1.37.0, but you have opentelemetry-api 1.38.0 which is incompatible. google-adk 1.19.0 requires opentelemetry-sdk<=1.37.0,>=1.37.0, but you have opentelemetry-sdk 1.38.0 which is incompatible. numba 0.60.0 requires numpy<2.1,>=1.22, but you have numpy 2.3.5 which is incompatible. tensorflow 2.19.0 requires numpy<2.2.0,>=1.26.0, but you have numpy 2.3.5 which is incompatible. langgraph-prebuilt 1.0.5 requires langchain-core>=1.0.0, but you have langchain-core 0.3.80 which is incompatible. opencv-contrib-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.5 which is incompatible. opencv-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.5 which is incompatible.
Note:
- After running the above cell, kindly restart the runtime (for Google Colab) or notebook kernel (for Jupyter Notebook), and run all cells sequentially from the next cell.
- On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.
In [ ]:
# For installing the libraries & downloading models from HF Hub
!pip install huggingface_hub==0.35.3 pandas==2.2.2 tiktoken==0.12.0 pymupdf==1.26.5 langchain==0.3.27 langchain-community==0.3.31 chromadb==1.1.1 sentence-transformers==5.1.1 numpy==2.3.3 -q
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. opencv-python-headless 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.3 which is incompatible. numba 0.60.0 requires numpy<2.1,>=1.22, but you have numpy 2.3.3 which is incompatible. tensorflow 2.19.0 requires numpy<2.2.0,>=1.26.0, but you have numpy 2.3.3 which is incompatible. opencv-contrib-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.3 which is incompatible. opencv-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.3.3 which is incompatible.
Note:
- After running the above cell, kindly restart the runtime (for Google Colab) or notebook kernel (for Jupyter Notebook), and run all cells sequentially from the next cell.
- On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook.
In [ ]:
#Libraries for processing dataframes,text
import json,os
import tiktoken
import pandas as pd
#Libraries for Loading Data, Chunking, Embedding, and Vector Databases
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyMuPDFLoader
from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain_community.vectorstores import Chroma
#Libraries for downloading and loading the llm
from huggingface_hub import hf_hub_download
from llama_cpp import Llama
Question Answering using LLM¶
Downloading and Loading the model¶
In [ ]:
model_name_or_path = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF"
model_basename = "mistral-7b-instruct-v0.2.Q6_K.gguf"
In [ ]:
model_path = hf_hub_download(
repo_id= model_name_or_path,
filename=model_basename
)
/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
In [ ]:
llm = Llama(
model_path=model_path,
n_ctx=2300,
n_gpu_layers=38,
n_batch=512
)
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
In [ ]:
# llm = Llama(
# model_path=model_path,
# n_ctx=1024,
# n_cores=-2
# )
Response¶
In [ ]:
def response(query,max_tokens=128,temperature=0,top_p=0.95,top_k=50):
model_output = llm(
prompt=query,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k
)
return model_output['choices'][0]['text']
In [ ]:
response("What treatment options are available for managing hypertension?")
Out[ ]:
'\n\nHypertension, or high blood pressure, is a common condition that can increase the risk of various health problems such as heart disease, stroke, and kidney damage. The good news is that there are several effective treatment options available to help manage hypertension and reduce the risk of complications. Here are some of the most commonly used treatments:\n\n1. Lifestyle modifications: Making lifestyle changes is often the first line of defense against hypertension. This may include eating a healthy diet rich in fruits, vegetables, whole grains, and lean proteins; limiting sodium intake; getting regular physical activity'
Query 1: What is the protocol for managing sepsis in a critical care unit?¶
In [ ]:
user_input = "What is the protocol for managing sepsis in a critical care unit?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
'\n\nSepsis is a life-threatening condition that can arise from an infection, and it requires prompt recognition and aggressive management in a critical care unit. The following are general steps for managing sepsis in a critical care unit:\n\n1. Early recognition: Recognize the signs and symptoms of sepsis early and initiate treatment as soon as possible. Sepsis can present with various clinical features, including fever or hypothermia, tachycardia or bradycardia, altered mental status, respiratory distress, and lactic acidosis.\n2. Source control'
Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶
In [ ]:
user_input = "What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical prodecure should be followed to treat it?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
'\n\nAppendicitis is a medical condition characterized by inflammation of the appendix, a small tube-shaped organ located in the lower right side of the abdomen. The symptoms of appendicitis can vary from person to person, but some common signs and symptoms include:\n\n1. Abdominal pain: The pain is typically located in the lower right quadrant of the abdomen and may be sharp, crampy, or dull. It may start as a mild discomfort that comes and goes, but it usually worsens over time and becomes constant and severe.\n2. Loss of appetite'
Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶
In [ ]:
user_input = "What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nSudden patchy hair loss, also known as alopecia areata, is a common autoimmune disorder that affects the hair follicles. It can result in round or oval bald patches on the scalp, but it can also occur on other parts of the body such as the beard area, eyebrows, and eyelashes.\n\nThe exact cause of alopecia areata is not known, but it's believed to be related to a problem with the immune system. Some possible triggers for this condition include stress, genetics, viral infections, and certain medications."
Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶
In [ ]:
user_input = "What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nA person who has sustained a physical injury to brain tissue, also known as a traumatic brain injury (TBI), may require various treatments depending on the severity and location of the injury. Here are some common treatments recommended for TBIs:\n\n1. Emergency care: The first priority is to ensure the person's airway is clear, they are breathing, and their heart is beating normally. In severe cases, emergency surgery may be required to remove hematomas or other obstructions.\n2. Medications: Depending on the symptoms, medications may be prescribed to manage conditions such as"
Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶
In [ ]:
user_input = "What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nFirst and foremost, if you suspect that someone has fractured their leg while hiking, it's essential to ensure their safety and prevent further injury. Here are some necessary precautions:\n\n1. Keep the person calm and still: Encourage them to remain as still as possible to minimize pain and prevent worsening the injury.\n2. Assess the situation: Check for any signs of shock, such as pale skin, rapid heartbeat, or low blood pressure. If you notice these symptoms, seek medical help immediately.\n3. Immobilize the leg: Use a splint,"
Question Answering using LLM with Prompt Engineering¶
In [ ]:
system_prompt = "If not found, I don't know"
Query 1: What is the protocol for managing sepsis in a critical care unit?¶
In [ ]:
user_input = system_prompt+"\n"+"What is the protocol for managing sepsis in a critical care unit?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
'\nSepsis is a life-threatening condition that can arise from an infection. The management of sepsis in a critical care unit involves a multidisciplinary approach and prompt recognition and treatment. Here are some general steps that may be taken to manage sepsis in a critical care unit:\n1. Early recognition: Recognize the signs and symptoms of sepsis early, which may include fever or hypothermia, tachycardia or bradycardia, respiratory distress, altered mental status, and lactic acidosis.\n2. Source control: Identify'
Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶
In [ ]:
user_input = "What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
'\n\nAppendicitis is a medical condition characterized by inflammation of the appendix, a small tube-shaped organ located in the lower right side of the abdomen. The symptoms of appendicitis can vary from person to person, but some common signs include:\n\n1. Abdominal pain: The pain is typically located in the lower right quadrant of the abdomen and may start as a mild discomfort that gradually worsens over time. The pain may be constant or intermittent and may be aggravated by movement, deep breathing, or coughing.\n2. Loss of appetite'
Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶
In [ ]:
user_input = "What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nSudden patchy hair loss, also known as alopecia areata, is a common autoimmune disorder that affects the hair follicles. It can result in round or oval bald patches on the scalp, but it can also occur on other parts of the body such as the beard area, eyebrows, and eyelashes.\n\nThe exact cause of alopecia areata is not known, but it's believed to be related to a problem with the immune system. Some possible triggers for this condition include stress, genetics, viral infections, and certain medications."
Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶
In [ ]:
user_input = "What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nA person who has sustained a physical injury to brain tissue, also known as a traumatic brain injury (TBI), may require various treatments depending on the severity and location of the injury. Here are some common treatments recommended for TBIs:\n\n1. Emergency care: The first priority is to ensure the person's airway is clear, they are breathing, and their heart is beating normally. In severe cases, emergency surgery may be required to remove hematomas or other obstructions.\n2. Medications: Depending on the symptoms, medications may be prescribed to manage conditions such as"
Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶
In [ ]:
user_input ="What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?"
response(user_input)
Llama.generate: prefix-match hit
Out[ ]:
"\n\nFirst and foremost, if you suspect that someone has fractured their leg while hiking, it's essential to ensure their safety and prevent further injury. Here are some necessary precautions:\n\n1. Keep the person calm and still: Encourage them to remain as still as possible to minimize pain and prevent worsening the injury.\n2. Assess the situation: Check for any signs of shock, such as pale skin, rapid heartbeat, or low blood pressure. If you notice these symptoms, seek medical help immediately.\n3. Immobilize the leg: Use a splint,"
Data Preparation for RAG¶
In [ ]:
#Libraries for processing dataframes,text
import json,os
import tiktoken
import pandas as pd
#Libraries for Loading Data, Chunking, Embedding, and Vector Databases
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyMuPDFLoader
from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain_community.vectorstores import Chroma
Loading the Data¶
In [ ]:
from google.colab import drive
drive.mount('/content/drive/')
Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount("/content/drive/", force_remount=True).
In [ ]:
import os
files_and_directories = os.listdir('/content/drive/MyDrive/Colab Notebooks/')
for item in files_and_directories:
print(item)
BigDataPart2 BigDataPart1 BigDataPart3 big-data-challenge main1_level1_finalized.ipynb config.py main2_level1_finalized.ipynb medical_diagnosis_manual.pdf Full_Code_NLP_RAG_Project_Notebook (1).ipynb
In [ ]:
import os
# Define the base directory where the PDF is located
base_dir = '/content/drive/MyDrive/Colab Notebooks/'
# Define the specific PDF file name
pdf_file_name = 'medical_diagnosis_manual.pdf'
# Construct the full path using os.path.join
manual_pdf_path = os.path.join(base_dir, pdf_file_name)
print(f"Corrected manual PDF path: {manual_pdf_path}")
Corrected manual PDF path: /content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf
In [ ]:
# Now, let's try loading the PDF again with the corrected path
pdf_loader = PyMuPDFLoader(manual_pdf_path)
manual = pdf_loader.load()
print(f"Successfully loaded {len(manual)} pages from the PDF.")
Successfully loaded 4114 pages from the PDF.
Data Overview¶
Checking the first 5 pages¶
In [ ]:
for i in range(5):
print(f"Page Number : {i+1}",end="\n")
print(manual[i].page_content,end="\n")
Page Number : 1 yi.erika0220@gmail.com 9UQFXT5OPG ant for personal use by yi.erika0220@gm shing the contents in part or full is liable Page Number : 2 yi.erika0220@gmail.com 9UQFXT5OPG This file is meant for personal use by yi.erika0220@gmail.com only. Sharing or publishing the contents in part or full is liable for legal action. Page Number : 3 Table of Contents 1 Front ................................................................................................................................................................................................................ 1 Cover ....................................................................................................................................................................................................... 2 Front Matter ........................................................................................................................................................................................... 53 1 - Nutritional Disorders ............................................................................................................................................................... 53 Chapter 1. Nutrition: General Considerations ..................................................................................................................... 59 Chapter 2. Undernutrition ............................................................................................................................................................. 69 Chapter 3. Nutritional Support ................................................................................................................................................... 76 Chapter 4. Vitamin Deficiency, Dependency & Toxicity .................................................................................................. 99 Chapter 5. Mineral Deficiency & Toxicity .............................................................................................................................. 108 Chapter 6. Obesity & the Metabolic Syndrome ............................................................................................................... 120 2 - Gastrointestinal Disorders .............................................................................................................................................. 120 Chapter 7. Approach to the Patient With Upper GI Complaints ............................................................................... 132 Chapter 8. Approach to the Patient With Lower GI Complaints ............................................................................... 143 Chapter 9. Diagnostic & Therapeutic GI Procedures .................................................................................................... 150 Chapter 10. GI Bleeding ............................................................................................................................................................ 158 Chapter 11. Acute Abdomen & Surgical Gastroenterology ......................................................................................... 172 Chapter 12. Esophageal & Swallowing Disorders .......................................................................................................... 183 Chapter 13. Gastritis & Peptic Ulcer Disease .................................................................................................................. 196 Chapter 14. Bezoars & Foreign Bodies .............................................................................................................................. 199 Chapter 15. Pancreatitis ............................................................................................................................................................ 206 Chapter 16. Gastroenteritis ...................................................................................................................................................... 213 Chapter 17. Malabsorption Syndromes .............................................................................................................................. 225 Chapter 18. Irritable Bowel Syndrome ................................................................................................................................ 229 Chapter 19. Inflammatory Bowel Disease ......................................................................................................................... 241 Chapter 20. Diverticular Disease ........................................................................................................................................... 246 Chapter 21. Anorectal Disorders ............................................................................................................................................ 254 Chapter 22. Tumors of the GI Tract ...................................................................................................................................... 275 3 - Hepatic & Biliary Disorders ............................................................................................................................................ 275 Chapter 23. Approach to the Patient With Liver Disease ........................................................................................... 294 Chapter 24. Testing for Hepatic & Biliary Disorders ...................................................................................................... 305 Chapter 25. Drugs & the Liver ................................................................................................................................................ 308 Chapter 26. Alcoholic Liver Disease .................................................................................................................................... 314 Chapter 27. Fibrosis & Cirrhosis ............................................................................................................................................ 322 Chapter 28. Hepatitis .................................................................................................................................................................. 333 Chapter 29. Vascular Disorders of the Liver ..................................................................................................................... 341 Chapter 30. Liver Masses & Granulomas .......................................................................................................................... 348 Chapter 31. Gallbladder & Bile Duct Disorders ............................................................................................................... 362 4 - Musculoskeletal & Connective Tissue Disorders ......................................................................................... 362 Chapter 32. Approach to the Patient With Joint Disease ............................................................................................ 373 Chapter 33. Autoimmune Rheumatic Disorders .............................................................................................................. 391 Chapter 34. Vasculitis ................................................................................................................................................................. 416 Chapter 35. Joint Disorders ..................................................................................................................................................... 435 Chapter 36. Crystal-Induced Arthritides .............................................................................................................................. 443 Chapter 37. Osteoporosis ......................................................................................................................................................... 448 Chapter 38. Paget's Disease of Bone .................................................................................................................................. 451 Chapter 39. Osteonecrosis ....................................................................................................................................................... 455 Chapter 40. Infections of Joints & Bones ........................................................................................................................... 463 Chapter 41. Bursa, Muscle & Tendon Disorders ............................................................................................................. 470 Chapter 42. Neck & Back Pain ............................................................................................................................................... 481 Chapter 43. Hand Disorders .................................................................................................................................................... yi.erika0220@gmail.com 9UQFXT5OPG This file is meant for personal use by yi.erika0220@gmail.com only. Sharing or publishing the contents in part or full is liable for legal action. Page Number : 4 491 Chapter 44. Foot & Ankle Disorders ..................................................................................................................................... 502 Chapter 45. Tumors of Bones & Joints ............................................................................................................................... 510 5 - Ear, Nose, Throat & Dental Disorders .................................................................................................................. 510 Chapter 46. Approach to the Patient With Ear Problems ........................................................................................... 523 Chapter 47. Hearing Loss ......................................................................................................................................................... 535 Chapter 48. Inner Ear Disorders ............................................................................................................................................ 542 Chapter 49. Middle Ear & Tympanic Membrane Disorders ........................................................................................ 550 Chapter 50. External Ear Disorders ..................................................................................................................................... 554 Chapter 51. Approach to the Patient With Nasal & Pharyngeal Symptoms ....................................................... 567 Chapter 52. Oral & Pharyngeal Disorders ......................................................................................................................... 578 Chapter 53. Nose & Paranasal Sinus Disorders ............................................................................................................. 584 Chapter 54. Laryngeal Disorders ........................................................................................................................................... 590 Chapter 55. Tumors of the Head & Neck ........................................................................................................................... 600 Chapter 56. Approach to Dental & Oral Symptoms ....................................................................................................... 619 Chapter 57. Common Dental Disorders ............................................................................................................................. 629 Chapter 58. Dental Emergencies .......................................................................................................................................... 635 Chapter 59. Temporomandibular Disorders ...................................................................................................................... 641 6 - Eye Disorders ............................................................................................................................................................................ 641 Chapter 60. Approach to the Ophthalmologic Patient .................................................................................................. 669 Chapter 61. Refractive Error ................................................................................................................................................... 674 Chapter 62. Eyelid & Lacrimal Disorders ........................................................................................................................... 680 Chapter 63. Conjunctival & Scleral Disorders ................................................................................................................. 690 Chapter 64. Corneal Disorders ............................................................................................................................................... 703 Chapter 65. Glaucoma ............................................................................................................................................................... 710 Chapter 66. Cataract ................................................................................................................................................................... 713 Chapter 67. Uveitis ...................................................................................................................................................................... 719 Chapter 68. Retinal Disorders ................................................................................................................................................. 731 Chapter 69. Optic Nerve Disorders ...................................................................................................................................... 737 Chapter 70. Orbital Diseases .................................................................................................................................................. 742 7 - Dermatologic Disorders .................................................................................................................................................... 742 Chapter 71. Approach to the Dermatologic Patient ....................................................................................................... 755 Chapter 72. Principles of Topical Dermatologic Therapy ............................................................................................ 760 Chapter 73. Acne & Related Disorders ............................................................................................................................... 766 Chapter 74. Bullous Diseases ................................................................................................................................................. 771 Chapter 75. Cornification Disorders ..................................................................................................................................... 775 Chapter 76. Dermatitis ............................................................................................................................................................... 786 Chapter 77. Reactions to Sunlight ........................................................................................................................................ 791 Chapter 78. Psoriasis & Scaling Diseases ........................................................................................................................ 799 Chapter 79. Hypersensitivity & Inflammatory Disorders ............................................................................................. 808 Chapter 80. Sweating Disorders ............................................................................................................................................ 811 Chapter 81. Bacterial Skin Infections ................................................................................................................................... 822 Chapter 82. Fungal Skin Infections ...................................................................................................................................... 831 Chapter 83. Parasitic Skin Infections ................................................................................................................................... 836 Chapter 84. Viral Skin Diseases ............................................................................................................................................ 841 Chapter 85. Pigmentation Disorders .................................................................................................................................... 846 Chapter 86. Hair Disorders ....................................................................................................................................................... 855 Chapter 87. Nail Disorders ....................................................................................................................................................... 861 Chapter 88. Pressure Ulcers ................................................................................................................................................... 867 Chapter 89. Benign Tumors ..................................................................................................................................................... 874 Chapter 90. Cancers of the Skin ............................................................................................................................................ 882 8 - Endocrine & Metabolic Disorders ............................................................................................................................. 882 Chapter 91. Principles of Endocrinology ............................................................................................................................ 887 Chapter 92. Pituitary Disorders .............................................................................................................................................. 901 Chapter 93. Thyroid Disorders ................................................................................................................................................ yi.erika0220@gmail.com 9UQFXT5OPG This file is meant for personal use by yi.erika0220@gmail.com only. Sharing or publishing the contents in part or full is liable for legal action. Page Number : 5 921 Chapter 94. Adrenal Disorders ................................................................................................................................................ 936 Chapter 95. Polyglandular Deficiency Syndromes ........................................................................................................ 939 Chapter 96. Porphyrias .............................................................................................................................................................. 949 Chapter 97. Fluid & Electrolyte Metabolism ..................................................................................................................... 987 Chapter 98. Acid-Base Regulation & Disorders .............................................................................................................. 1001 Chapter 99. Diabetes Mellitus & Disorders of Carbohydrate Metabolism ........................................................ 1024 Chapter 100. Lipid Disorders ................................................................................................................................................ 1034 Chapter 101. Amyloidosis ...................................................................................................................................................... 1037 Chapter 102. Carcinoid Tumors .......................................................................................................................................... 1040 Chapter 103. Multiple Endocrine Neoplasia Syndromes ......................................................................................... 1046 9 - Hematology & Oncology ............................................................................................................................................... 1046 Chapter 104. Approach to the Patient With Anemia .................................................................................................. 1050 Chapter 105. Anemias Caused by Deficient Erythropoiesis ................................................................................... 1061 Chapter 106. Anemias Caused by Hemolysis ............................................................................................................... 1078 Chapter 107. Neutropenia & Lymphocytopenia ........................................................................................................... 1086 Chapter 108. Thrombocytopenia & Platelet Dysfunction ......................................................................................... 1097 Chapter 109. Hemostasis ...................................................................................................................................................... 1104 Chapter 110. Thrombotic Disorders ................................................................................................................................... 1107 Chapter 111. Coagulation Disorders .................................................................................................................................. 1113 Chapter 112. Bleeding Due to Abnormal Blood Vessels ........................................................................................... 1116 Chapter 113. Spleen Disorders ............................................................................................................................................ 1120 Chapter 114. Eosinophilic Disorders ................................................................................................................................. 1126 Chapter 115. Histiocytic Syndromes ................................................................................................................................. 1131 Chapter 116. Myeloproliferative Disorders ..................................................................................................................... 1141 Chapter 117. Leukemias ......................................................................................................................................................... 1154 Chapter 118. Lymphomas ...................................................................................................................................................... 1164 Chapter 119. Plasma Cell Disorders ................................................................................................................................. 1172 Chapter 120. Iron Overload ................................................................................................................................................... 1177 Chapter 121. Transfusion Medicine ................................................................................................................................... 1186 Chapter 122. Overview of Cancer ...................................................................................................................................... 1198 Chapter 123. Tumor Immunology ....................................................................................................................................... 1204 Chapter 124. Principles of Cancer Therapy ................................................................................................................... 1215 10 - Immunology; Allergic Disorders ........................................................................................................................... 1215 Chapter 125. Biology of the Immune System ............................................................................................................... 1227 Chapter 126. Immunodeficiency Disorders .................................................................................................................... 1243 Chapter 127. Allergic & Other Hypersensitivity Disorders ....................................................................................... 1263 Chapter 128. Transplantation ............................................................................................................................................... 1281 11 - Infectious Diseases ........................................................................................................................................................ 1281 Chapter 129. Biology of Infectious Disease ................................................................................................................... 1300 Chapter 130. Laboratory Diagnosis of Infectious Disease ...................................................................................... 1306 Chapter 131. Immunization ................................................................................................................................................... 1313 Chapter 132. Bacteria & Antibacterial Drugs ................................................................................................................. 1353 Chapter 133. Gram-Positive Cocci .................................................................................................................................... 1366 Chapter 134. Gram-Positive Bacilli ................................................................................................................................... 1376 Chapter 135. Gram-Negative Bacilli ................................................................................................................................. 1405 Chapter 136. Spirochetes ...................................................................................................................................................... 1413 Chapter 137. Neisseriaceae ................................................................................................................................................. 1419 Chapter 138. Chlamydia & Mycoplasmas ...................................................................................................................... 1421 Chapter 139. Rickettsiae & Related Organisms .......................................................................................................... 1431 Chapter 140. Anaerobic Bacteria ........................................................................................................................................ 1450 Chapter 141. Mycobacteria ................................................................................................................................................... 1470 Chapter 142. Fungi ................................................................................................................................................................... 1493 Chapter 143. Approach to Parasitic Infections ............................................................................................................. 1496 Chapter 144. Nematodes (Roundworms) ....................................................................................................................... yi.erika0220@gmail.com 9UQFXT5OPG This file is meant for personal use by yi.erika0220@gmail.com only. Sharing or publishing the contents in part or full is liable for legal action.
Checking the number of pages¶
In [ ]:
len(manual)
Out[ ]:
4114
Data Chunking¶
In [ ]:
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name='cl100k_base',
chunk_size=512,
chunk_overlap=128,
separators=['\n\n', '\n', ' ', '']
)
In [ ]:
document_chunks = pdf_loader.load_and_split(text_splitter)
In [ ]:
len(document_chunks)
Out[ ]:
9387
In [ ]:
document_chunks[0].page_content
Out[ ]:
'yi.erika0220@gmail.com\n9UQFXT5OPG\nant for personal use by yi.erika0220@gm\nshing the contents in part or full is liable'
In [ ]:
document_chunks[2].page_content
Out[ ]:
'Table of Contents\n1\nFront ................................................................................................................................................................................................................\n1\nCover .......................................................................................................................................................................................................\n2\nFront Matter ...........................................................................................................................................................................................\n53\n1 - Nutritional Disorders ...............................................................................................................................................................\n53\nChapter 1. Nutrition: General Considerations .....................................................................................................................\n59\nChapter 2. Undernutrition .............................................................................................................................................................\n69\nChapter 3. Nutritional Support ...................................................................................................................................................\n76\nChapter 4. Vitamin Deficiency, Dependency & Toxicity ..................................................................................................\n99\nChapter 5. Mineral Deficiency & Toxicity ..............................................................................................................................\n108\nChapter 6. Obesity & the Metabolic Syndrome ...............................................................................................................\n120\n2 - Gastrointestinal Disorders ..............................................................................................................................................\n120\nChapter 7. Approach to the Patient With Upper GI Complaints ...............................................................................\n132\nChapter 8. Approach to the Patient With Lower GI Complaints ...............................................................................\n143\nChapter 9. Diagnostic & Therapeutic GI Procedures ....................................................................................................\n150\nChapter 10. GI Bleeding ............................................................................................................................................................\n158\nChapter 11. Acute Abdomen & Surgical Gastroenterology .........................................................................................\n172\nChapter 12. Esophageal & Swallowing Disorders ..........................................................................................................\n183\nChapter 13. Gastritis & Peptic Ulcer Disease ..................................................................................................................\n196\nChapter 14. Bezoars & Foreign Bodies ..............................................................................................................................\n199\nChapter 15. Pancreatitis ............................................................................................................................................................\n206\nChapter 16. Gastroenteritis ......................................................................................................................................................\n213\nChapter 17. Malabsorption Syndromes ..............................................................................................................................\n225\nChapter 18. Irritable Bowel Syndrome ................................................................................................................................\n229\nChapter 19. Inflammatory Bowel Disease .........................................................................................................................\n241\nChapter 20. Diverticular Disease ...........................................................................................................................................\n246\nChapter 21. Anorectal Disorders ............................................................................................................................................\n254\nChapter 22. Tumors of the GI Tract ......................................................................................................................................\n275\n3 - Hepatic & Biliary Disorders ............................................................................................................................................\n275\nChapter 23. Approach to the Patient With Liver Disease ...........................................................................................\n294\nChapter 24. Testing for Hepatic & Biliary Disorders ......................................................................................................\n305'
In [ ]:
document_chunks[3].page_content
Out[ ]:
"229\nChapter 19. Inflammatory Bowel Disease .........................................................................................................................\n241\nChapter 20. Diverticular Disease ...........................................................................................................................................\n246\nChapter 21. Anorectal Disorders ............................................................................................................................................\n254\nChapter 22. Tumors of the GI Tract ......................................................................................................................................\n275\n3 - Hepatic & Biliary Disorders ............................................................................................................................................\n275\nChapter 23. Approach to the Patient With Liver Disease ...........................................................................................\n294\nChapter 24. Testing for Hepatic & Biliary Disorders ......................................................................................................\n305\nChapter 25. Drugs & the Liver ................................................................................................................................................\n308\nChapter 26. Alcoholic Liver Disease ....................................................................................................................................\n314\nChapter 27. Fibrosis & Cirrhosis ............................................................................................................................................\n322\nChapter 28. Hepatitis ..................................................................................................................................................................\n333\nChapter 29. Vascular Disorders of the Liver .....................................................................................................................\n341\nChapter 30. Liver Masses & Granulomas ..........................................................................................................................\n348\nChapter 31. Gallbladder & Bile Duct Disorders ...............................................................................................................\n362\n4 - Musculoskeletal & Connective Tissue Disorders .........................................................................................\n362\nChapter 32. Approach to the Patient With Joint Disease ............................................................................................\n373\nChapter 33. Autoimmune Rheumatic Disorders ..............................................................................................................\n391\nChapter 34. Vasculitis .................................................................................................................................................................\n416\nChapter 35. Joint Disorders .....................................................................................................................................................\n435\nChapter 36. Crystal-Induced Arthritides ..............................................................................................................................\n443\nChapter 37. Osteoporosis .........................................................................................................................................................\n448\nChapter 38. Paget's Disease of Bone ..................................................................................................................................\n451\nChapter 39. Osteonecrosis .......................................................................................................................................................\n455\nChapter 40. Infections of Joints & Bones ...........................................................................................................................\n463\nChapter 41. Bursa, Muscle & Tendon Disorders .............................................................................................................\n470\nChapter 42. Neck & Back Pain ...............................................................................................................................................\n481\nChapter 43. Hand Disorders ....................................................................................................................................................\nyi.erika0220@gmail.com\n9UQFXT5OPG\nThis file is meant for personal use by yi.erika0220@gmail.com only."
Embedding¶
In [ ]:
embedding_model = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
/tmp/ipython-input-3002436811.py:1: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-huggingface package and should be used instead. To use it run `pip install -U :class:`~langchain-huggingface` and import as `from :class:`~langchain_huggingface import HuggingFaceEmbeddings``. embedding_model = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
In [ ]:
embedding_1 = embedding_model.embed_query(document_chunks[0].page_content)
embedding_2 = embedding_model.embed_query(document_chunks[1].page_content)
In [ ]:
print("Dimension of the embedding vector ",len(embedding_1))
len(embedding_1)==len(embedding_2)
Dimension of the embedding vector 384
Out[ ]:
True
In [ ]:
embedding_1,embedding_2
Out[ ]:
([-0.07799344509840012, 0.05522589385509491, -0.004778371192514896, -0.025384459644556046, 0.07581344246864319, -0.018376892432570457, 0.12176954746246338, 0.03152600675821304, 0.005126860458403826, 0.009516126476228237, 0.08502651751041412, -0.0004012853023596108, 0.006589178927242756, -0.013660921715199947, -0.02281372994184494, -0.024491511285305023, -0.0637589693069458, -0.06164788827300072, -0.088883176445961, -0.027152109891176224, -0.017307845875620842, -0.035725172609090805, -0.00319072138518095, -0.07279198616743088, -0.04981670156121254, 0.03761101886630058, -0.02827776037156582, 0.06178930774331093, 0.008998526260256767, -0.06604358553886414, 0.023974651470780373, 0.038113612681627274, 0.058909591287374496, -0.013527078554034233, 0.03196187689900398, 0.058886315673589706, -0.018196461722254753, -0.09924376755952835, 0.009629708714783192, -0.04748279228806496, 1.0143363397219218e-05, 0.011993570253252983, -0.018023278564214706, 0.045897722244262695, -0.010122442618012428, 0.06969739496707916, 0.052573464810848236, -0.03474084287881851, -0.003008725121617317, 0.04634889215230942, -0.016467465087771416, -0.01366478856652975, 0.013073954731225967, 0.07692425698041916, -0.05777271091938019, -0.025702456012368202, 0.02641921117901802, -0.02180231362581253, -0.030391192063689232, 0.0006090631941333413, 0.0290234312415123, 0.05956174433231354, -0.06293687969446182, 0.008551263250410557, 0.01997961848974228, -0.05748305842280388, -0.004559448920190334, 0.011085432954132557, 0.0017882466781884432, -0.054996833205223083, 0.0016359431901946664, -0.01757078245282173, -0.06766089797019958, 0.08867987245321274, -0.04502802714705467, 0.06373682618141174, 0.04367844760417938, -0.03569093346595764, -0.0029642407316714525, -0.030280612409114838, -0.018055111169815063, 0.04141601175069809, 0.08341192454099655, -0.052741583436727524, -0.03174823895096779, 0.1352580040693283, 0.0011770665878430009, 0.05664398893713951, 0.017626186832785606, -0.0015976728172972798, 0.016594180837273598, 0.03162407875061035, 0.13741475343704224, 0.035236794501543045, -0.05513386055827141, 0.04382798448204994, -0.007372646126896143, -0.04547877982258797, -0.050707489252090454, 0.09665720909833908, 0.014541354030370712, 0.04517803341150284, 0.0224616602063179, -0.05181987211108208, -0.047172483056783676, -0.021654831245541573, -0.04658343642950058, 0.0667814090847969, -3.581121927709319e-05, 0.03372856229543686, -0.06838515400886536, 0.0010852015111595392, 0.00895210262387991, -0.02489190362393856, 0.1114787757396698, -0.043760672211647034, -0.05176934227347374, 0.050315842032432556, 0.06547891348600388, -0.11184223741292953, 0.05653967708349228, -0.04298459365963936, -0.024457450956106186, -0.015493425540626049, -0.09453970938920975, -0.09584657102823257, 0.04211468622088432, 9.690951752331562e-33, -0.01711421087384224, 0.09432298690080643, 0.029505377635359764, -0.020564178004860878, -0.04579759016633034, 0.005810732487589121, 0.038549501448869705, 0.06497058272361755, -0.0706433579325676, 0.041761476546525955, 0.007583636790513992, 0.0011110100895166397, -0.03681614622473717, -0.020095154643058777, -0.05817066878080368, 0.032585617154836655, 0.02479938417673111, 0.08161772042512894, 0.07852277904748917, -0.0019434395944699645, 0.024896467104554176, -0.05115876346826553, -0.04320646822452545, 0.025279467925429344, 0.02510092221200466, 0.07899020612239838, 0.022946862503886223, -0.058780208230018616, 0.007944757118821144, 0.010471729561686516, 0.0756361186504364, 0.0010711501818150282, -0.017034249380230904, 0.07135571539402008, 0.010191013105213642, 0.034462396055459976, -0.07615536451339722, 0.010495783761143684, -0.10029434412717819, -0.012362021021544933, 0.10301978886127472, -0.07198716700077057, 0.060137055814266205, 0.01887071318924427, -0.051872618496418, -0.03524452820420265, 0.016288693994283676, 0.007874599657952785, 0.09080414474010468, -0.001800375641323626, -0.060791559517383575, -0.00600431626662612, -0.041097741574048996, -0.05468755587935448, -0.03571109473705292, -0.01398807018995285, -0.014422198757529259, -0.014390389434993267, -0.009278951212763786, -0.043525271117687225, -0.00494921812787652, 0.02600853703916073, -0.07516483217477798, 0.024904469028115273, -0.030097542330622673, -0.10587877035140991, -0.020147409290075302, -0.1410156488418579, 0.011145084165036678, -0.06263301521539688, -0.05993124842643738, -0.04502643272280693, 0.07271213829517365, 0.049906618893146515, -0.10340378433465958, -0.05348283052444458, -0.0029318910092115402, 0.020267128944396973, 0.026540014892816544, 0.03465237841010094, -0.08725742250680923, -0.05643851310014725, 0.04565189406275749, 0.026310322806239128, -0.040821585804224014, 0.02825349010527134, 0.05031875893473625, 0.0018109489465132356, -0.010268324986100197, 0.06530790030956268, 0.004321488551795483, -0.003426461713388562, -0.04550614207983017, 0.003131438046693802, -0.00723457895219326, -1.0506575119805329e-32, -0.04597264528274536, -0.07331523299217224, -0.08969902247190475, -0.05506593734025955, 0.010539975017309189, 0.054750218987464905, 0.04775642231106758, 0.0927966982126236, -0.0025483649224042892, -0.009276228956878185, -0.08103451132774353, 0.04020683467388153, 0.037611354142427444, -0.0719861090183258, 0.007103877142071724, 0.004628312308341265, 0.05183733254671097, -0.03194565325975418, -0.001109649776481092, -0.035837322473526, -0.02384464256465435, 0.10917625576257706, 0.0409637950360775, 0.02170283906161785, -0.003625427605584264, -0.007656692992895842, 0.08922954648733139, 0.04586415737867355, -0.031415291130542755, 0.05779963731765747, 0.10654225200414658, -0.045620180666446686, -0.16526955366134644, 0.0038973034825176, -0.023191828280687332, -0.16409651935100555, 0.055137552320957184, 0.04961225762963295, -0.03701968118548393, -0.018777567893266678, 0.010590110905468464, 0.047233421355485916, -0.021626200526952744, 0.07601870596408844, -0.020594079047441483, -0.08275911957025528, 0.03362013027071953, -0.08954649418592453, 0.03000064566731453, 0.0977926254272461, 0.01874249428510666, -0.09121422469615936, 0.0639452263712883, -0.044201042503118515, 0.02509048953652382, 0.059633467346429825, 0.07065235823392868, -0.025196634232997894, 0.028321174904704094, -0.10032539069652557, 0.009321082383394241, 0.05622158944606781, 0.008766942657530308, 0.009466535411775112, 0.03433304652571678, -0.03143689036369324, 0.08128770440816879, -0.019885681569576263, 0.039996348321437836, -0.08017179369926453, 0.030400333926081657, 0.01026246789842844, 0.005303863435983658, -0.0664575845003128, 0.09367409348487854, -0.009167322888970375, 0.006956912577152252, 0.06007919833064079, -0.012036540545523167, -0.004528976045548916, 0.06997043639421463, 0.0025399387814104557, -0.03146142512559891, 0.006411934737116098, 0.07227359712123871, -0.12768901884555817, 0.02659556083381176, 0.04512520879507065, -0.0010502374498173594, -0.00754051236435771, -0.05678047984838486, 0.04696478694677353, 0.0005130782374180853, 0.029329149052500725, -0.03795348107814789, -4.262678743316428e-08, 0.019506102427840233, -0.027851631864905357, 0.033198099583387375, 0.0022505579981952906, 0.008473094552755356, -0.00016927780234254897, 0.013293628580868244, 0.03532782569527626, -0.0017152364598587155, 0.05091886594891548, 0.0026884078979492188, -0.01825197972357273, -0.02942717634141445, 0.04126835614442825, -0.0830923542380333, -0.0938916727900505, -0.016213275492191315, 0.018494462594389915, -0.0846405103802681, -0.07198385149240494, -0.047872643917798996, -0.0031700823456048965, -0.0012762784026563168, -0.06625032424926758, 0.04219160974025726, 0.01688411459326744, 0.022230980917811394, 0.016161207109689713, -0.061635036021471024, 0.02733246050775051, -0.08614981174468994, 0.004087566398084164, 0.028512155637145042, -0.040652889758348465, -0.044425852596759796, -0.02065901830792427, 0.02302771806716919, -0.05620397999882698, -0.04845743253827095, -0.04679189994931221, 0.039343561977148056, -0.02398556098341942, 0.08969110995531082, 0.025555014610290527, 0.0596451610326767, -0.047334201633930206, -0.021294251084327698, -0.1133623868227005, 0.07705453038215637, -0.02342440001666546, -0.05785667151212692, -0.07046658545732498, -0.005159209482371807, 0.014897755347192287, -0.05004395917057991, 0.0036724836099892855, 0.06698185205459595, 0.05606968700885773, 0.06470507383346558, 0.06538337469100952, -0.0041669378988444805, 0.014502054080367088, 0.00882007647305727, -0.0637826919555664], [-0.08602702617645264, 0.029867498204112053, -0.020749853923916817, -0.008226017467677593, 0.09007387608289719, -0.008148028515279293, 0.061080269515514374, 0.011456898413598537, 0.05112479254603386, 0.034672100096940994, 0.09290582686662674, 0.08110833168029785, 0.038261089473962784, -0.005085325334221125, -0.019110267981886864, 0.01915760710835457, -0.11440371721982956, -0.06650522351264954, -0.08808334916830063, -0.01942417211830616, -0.00639416603371501, -0.013028784655034542, 0.021509896963834763, -0.039506204426288605, -0.013365023769438267, -0.016502907499670982, -0.028059542179107666, 0.03230514004826546, -0.03397683799266815, -0.050378959625959396, -0.015214449726045132, 0.06344591826200485, 0.028507137671113014, 0.029810592532157898, 0.031089914962649345, 0.049414847046136856, 0.06524182111024857, -0.06860775500535965, -0.02288649044930935, -0.0491897352039814, -0.02276521921157837, -0.009881996549665928, -0.03031756356358528, -0.008138063363730907, 0.006783930119127035, 0.09825560450553894, 0.0296431053429842, -0.014638517051935196, -0.052253205329179764, -0.0020529013127088547, -0.09035531431436539, 0.004872235935181379, 0.03415990620851517, 0.013554487377405167, 0.025640415027737617, -0.03024674579501152, -0.018457641825079918, -0.0021648104302585125, -0.012776029296219349, 0.03859233111143112, -0.013121112249791622, 0.057626549154520035, -0.060742441564798355, -0.028086304664611816, 0.03205687180161476, -0.02251749485731125, 0.018508657813072205, 0.040164437144994736, -0.054471004754304886, -0.09700798243284225, -0.03584308922290802, -0.0016000025207176805, -0.05173887684941292, 0.07029198110103607, -0.00836747046560049, -0.018552055582404137, -0.0245006512850523, -0.016186976805329323, 0.0057934909127652645, -0.030538050457835197, 0.009101912379264832, 0.030802372843027115, 0.14231723546981812, -0.07399868965148926, -0.034365519881248474, 0.08540693670511246, 0.033780794590711594, 0.024665111675858498, 0.04775289446115494, -0.02629334107041359, -0.014397247694432735, 0.03490667790174484, 0.11763566732406616, 0.04885711893439293, -0.03680841997265816, 0.014353417791426182, -0.0019091902067884803, -0.014973247423768044, -0.047522883862257004, 0.07278866320848465, 0.061487697064876556, 0.02714129537343979, 0.03947063162922859, -0.00845800619572401, 0.01437515951693058, -0.048710159957408905, 0.0585431270301342, 0.038263577967882156, -0.04091876372694969, 0.0015620304038748145, -0.036642178893089294, -0.0396859310567379, -0.04832775145769119, -0.1420377641916275, 0.10874105989933014, -0.05960126966238022, -0.05747108906507492, 0.1078236848115921, 0.027695680037140846, -0.07032407075166702, -0.003519103629514575, -0.015911174938082695, -0.06103448569774628, 0.030513977631926537, -0.08094324171543121, -0.12337282299995422, 0.05343589931726456, 7.035914824691079e-33, 0.008898397907614708, 0.08431436121463776, 0.021252140402793884, 0.0036736733745783567, -0.011374428868293762, -0.05780654400587082, 0.004852666519582272, 0.02519388310611248, -0.1339765042066574, -0.012249738909304142, 0.012768907472491264, -0.029345005750656128, -0.01362630259245634, 0.05419161170721054, -0.031946081668138504, 0.06594587117433548, -0.02577482908964157, 0.0630769357085228, 0.06749959290027618, -0.000542941561434418, 0.05143889784812927, -0.025036564096808434, -0.01648523658514023, 0.020763104781508446, -0.020349550992250443, 0.08283638954162598, 0.05441979691386223, -0.08479689806699753, 0.03549591079354286, -0.004146593622863293, 0.08596356958150864, -0.06095409765839577, 0.020715728402137756, -0.01065471675246954, -0.0025192012544721365, 0.04679117351770401, -0.015738435089588165, 0.02203710563480854, -0.05496057868003845, 0.047484733164310455, 0.08872818201780319, -0.06338512152433395, 0.006085049360990524, -0.001583557459525764, -0.051431406289339066, 0.07207042723894119, 0.04870209842920303, -0.021023346111178398, 0.060259200632572174, -0.07917037606239319, -0.03885265439748764, 0.007442100904881954, -0.02942483499646187, -0.006979147903621197, 0.020295914262533188, 0.027536222711205482, 0.0009386857273057103, -0.07256980240345001, -0.01307851541787386, 0.0039095571264624596, 0.0017075407085940242, 0.0006879349239170551, -0.06897392123937607, 0.028991907835006714, -0.03225613012909889, -0.1166658103466034, -0.04599030315876007, -0.06276346743106842, 0.055282920598983765, -0.04218905419111252, -0.07848899066448212, -0.08340456336736679, 0.05647895112633705, 0.015730716288089752, -0.03595792129635811, -0.08820775896310806, -0.046038299798965454, 0.022072548046708107, 0.07015782594680786, -0.0016740842256695032, -0.07734899967908859, -0.01546491775661707, 0.0423596017062664, -0.002392671536654234, -0.06671255826950073, 0.032744746655225754, 0.005666288081556559, 0.025393668562173843, -0.020553309470415115, 0.044773150235414505, -0.038880355656147, 0.01730881817638874, -0.027920681983232498, 0.020900452509522438, -0.0029940467793494463, -8.52540208371723e-33, -0.034290291368961334, -0.03579491376876831, -0.09972018748521805, -0.039677124470472336, 0.005597619339823723, 0.07993708550930023, 0.05637586489319801, 0.05147875100374222, 0.012270063161849976, -0.014464092440903187, -0.05468771234154701, 0.019734753295779228, 0.03839893639087677, -0.05650150403380394, 0.004969869740307331, -0.019263381138443947, -0.004303904250264168, -0.04780391603708267, -0.07058726996183395, -0.009735600091516972, -0.09337923675775528, 0.010346205905079842, 0.09243876487016678, 0.053863756358623505, 0.030514201149344444, -0.0056558288633823395, 0.11533892154693604, 0.02824818529188633, 0.0157020203769207, 0.046847086399793625, 0.0422554537653923, -0.05660981312394142, -0.16625265777111053, -0.0447855107486248, 0.0027341924142092466, -0.15607470273971558, 0.009221833199262619, 0.07635917514562607, -0.03981437534093857, 0.01429676078259945, 0.000515979714691639, 0.05998709797859192, -0.051586881279945374, 0.004757375922054052, -0.01211564987897873, 0.003857759293168783, 0.01756151393055916, -0.0727151483297348, 0.024134019389748573, 0.025384947657585144, 0.0601188950240612, -0.04966703802347183, 0.08160322904586792, -0.08261127769947052, 0.05498190224170685, 0.017449934035539627, 0.013978255912661552, 0.012890592217445374, -0.003741893684491515, -0.030790336430072784, -0.03413187712430954, 0.013940542936325073, -0.030916767194867134, 0.0018490966176614165, 0.0389653705060482, -0.026467829942703247, 0.05024445429444313, 0.003248578170314431, 0.07286828756332397, -0.09482891857624054, -0.009366917423903942, -0.02817879058420658, -0.025564510375261307, -0.08033261448144913, 0.038328684866428375, -0.0012922182213515043, 0.007639268878847361, 0.08560154587030411, -0.01435128040611744, 0.058727484196424484, 0.10128962993621826, 0.004039836581796408, -0.043372783809900284, 0.03813030198216438, 0.050849657505750656, -0.12211715430021286, 0.025161966681480408, 0.0520441047847271, 0.01881946064531803, -0.0064063360914587975, -0.002725618425756693, 0.07932684570550919, -0.010938196443021297, -0.0037348943296819925, -0.04382694885134697, -4.388295948842824e-08, -0.011222186498343945, -0.055277835577726364, 0.01345805637538433, -0.0023103284183889627, -0.05145220458507538, 0.028235187754034996, 0.015277898870408535, -0.04066374525427818, 0.04327819123864174, 0.04059145227074623, 0.03962811827659607, -0.05550524964928627, -0.041964903473854065, 0.05101592466235161, -0.11437670886516571, -0.12553833425045013, 0.05410507693886757, -0.00808828417211771, 0.016402339562773705, -0.025798238813877106, -0.03406644985079765, 0.00853660050779581, 0.014382860623300076, -0.11264351010322571, 0.05524611473083496, 0.043483197689056396, 0.013851548545062542, -0.03292303532361984, -0.08545871824026108, 0.004624413792043924, -0.04524052515625954, 0.01345084048807621, 0.011512456461787224, -0.01366790384054184, -0.01822025701403618, 0.02956468239426613, 0.01838797703385353, -0.05141778662800789, -0.07197370380163193, -0.042452141642570496, 0.0804150179028511, 0.017001992091536522, 0.07816670835018158, -0.009507676586508751, 0.048738762736320496, -0.06659192591905594, 0.04065236821770668, -0.1421905905008316, 0.11230488121509552, 0.04373255744576454, -0.05352064594626427, -0.03299080207943916, 0.07312233746051788, 0.06716413050889969, 0.006768306717276573, 0.028173698112368584, 0.06024523451924324, 0.0020245176274329424, 0.048486556857824326, 0.06498394906520844, 0.053615033626556396, 0.022443387657403946, -0.012659025378525257, -0.01449032872915268])
Vector Database¶
In [ ]:
out_dir = 'medical_db'
if not os.path.exists(out_dir):
os.makedirs(out_dir)
In [ ]:
vectorstore = Chroma.from_documents(
documents=document_chunks,
embedding=embedding_model,
persist_directory=out_dir
)
# vectorstore.persist()
In [ ]:
vectorstore = Chroma(
persist_directory=out_dir,
embedding_function=embedding_model
)
In [ ]:
vectorstore.embeddings
Out[ ]:
HuggingFaceEmbeddings(client=SentenceTransformer(
(0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
), model_name='all-MiniLM-L6-v2', cache_folder=None, model_kwargs={}, encode_kwargs={}, multi_process=False, show_progress=False)
In [ ]:
vectorstore.similarity_search("Disorder",k=3)
Out[ ]:
[Document(metadata={'creationDate': 'D:20120615054440Z', 'keywords': '', 'total_pages': 4114, 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'format': 'PDF 1.7', 'moddate': '2025-11-13T01:12:47+00:00', 'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'author': '', 'creationdate': '2012-06-15T05:44:40+00:00', 'creator': 'Atop CHM to PDF Converter', 'subject': '', 'trapped': '', 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'modDate': 'D:20251113011247Z', 'page': 3261}, page_content='• Conduct disorder, characterized by poor control of aggressive impulses against others, possibly\nredirected against self (see p. 3060)\nMore immediate precipitating factors can include\n• Loss of self-esteem (eg, resulting from family arguments, a humiliating disciplinary episode, pregnancy,\nor school failure)\n• Loss of a boyfriend or girlfriend\n• Loss of familiar surroundings (eg, school, neighborhood, friends) due to a geographic move\nOther contributing factors may include a lack of structure and boundaries, leading to an overwhelming\nfeeling of lack of direction, and intense parental pressure to succeed accompanied by the feeling of\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nChapter 305. Mental Disorders in Children & Adolescents\n3252\nyi.erika0220@gmail.com\n9UQFXT5OPG\nThis file is meant for personal use by yi.erika0220@gmail.com only.\nSharing or publishing the contents in part or full is liable for legal action.'),
Document(metadata={'page': 1674, 'author': '', 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'keywords': '', 'total_pages': 4114, 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'modDate': 'D:20251113011247Z', 'moddate': '2025-11-13T01:12:47+00:00', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'creationDate': 'D:20120615054440Z', 'creationdate': '2012-06-15T05:44:40+00:00', 'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'subject': '', 'creator': 'Atop CHM to PDF Converter', 'format': 'PDF 1.7', 'trapped': ''}, page_content='disorders (ictal or postictal). When depersonalization occurs independently of other mental or physical\ndisorders and is persistent or recurrent, depersonalization disorder is present. It is estimated to occur in\nabout 2% of the general population.\nSymptoms and Signs\nPatients feel detached from their body, mind, feelings, or sensations. Most patients also say they feel\nunreal (derealization), like an automaton, or as if they were in a dream or in some other way detached\nfrom the world. Some patients cannot recognize or describe their emotions (alexithymia). Patients may\ndescribe themselves as the "walking dead." Symptoms are almost always distressing and, when severe,\nprofoundly intolerable. Anxiety and depression are common.\nSymptoms are often chronic; about one third of patients have recurrent episodes, and two thirds have\ncontinuous symptoms. Episodic symptoms sometimes become continuous.\nPatients often have great difficulty describing their symptoms and may fear or believe they are going\ncrazy. They always retain the knowledge that their unreal experiences are not real but rather are just the\nway that they feel. This awareness differentiates depersonalization disorder from a psychotic disorder, in\nwhich such insight is always lacking.\nDiagnosis\n• Medical and psychiatric evaluation\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nChapter 159. Dissociative Disorders\n1665\nyi.erika0220@gmail.com\n9UQFXT5OPG\nThis file is meant for personal use by yi.erika0220@gmail.com only.\nSharing or publishing the contents in part or full is liable for legal action.'),
Document(metadata={'format': 'PDF 1.7', 'page': 1757, 'moddate': '2025-11-13T01:12:47+00:00', 'creationDate': 'D:20120615054440Z', 'creationdate': '2012-06-15T05:44:40+00:00', 'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'modDate': 'D:20251113011247Z', 'trapped': '', 'creator': 'Atop CHM to PDF Converter', 'subject': '', 'keywords': '', 'total_pages': 4114, 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'author': ''}, page_content='Chapter 166. Somatoform and Factitious Disorders\nIntroduction\nSomatization is the expression of mental phenomena as physical (somatic) symptoms. Typically, the\nsymptoms cannot be explained by a physical disorder. Disorders characterized by somatization extend in\na continuum from those in which symptoms develop unconsciously and nonvolitionally to those in which\nsymptoms develop consciously and volitionally. This continuum includes somatoform disorders, factitious\ndisorders, and malingering. Somatization typically leads to seeking medical evaluation and treatment.\nSomatoform disorders are characterized by physical symptoms that are not fully explained by another\ndisorder—physical or mental. Symptoms of somatoform disorders are not volitional. Somatoform disorders\nare distressing and often impair social, occupational, academic, or other aspects of functioning. These\ndisorders include body dysmorphic disorder, conversion disorder, hypochondriasis, pain disorder,\nsomatization disorder, undifferentiated somatoform disorder, and somatoform disorder not otherwise\nspecified. Body dysmorphic disorder differs somewhat from other somatoform disorders in that it is\ncharacterized by preoccupation with perceived defects in physical appearance.\nFactitious disorders involve the conscious and volitional feigning of symptoms without any external\nincentive (eg, time off from work) and is thus distinguished from malingering. Patients gain gratification\nfrom assuming the sick role through the simulation, exaggeration, or aggravation of symptoms and signs.\nSymptoms and signs may be mental, physical, or both. The most severe and chronic form is Munchausen\nsyndrome.\nMalingering is intentional feigning of physical or mental symptoms motivated by an external incentive\n(eg, feigning illness to avoid work or military duty, to evade criminal prosecution, or to obtain financial\ncompensation or drugs for abuse). Malingering is suspected in the following cases:\n• Patients report symptoms, yet little is detected through unannounced observation, physical examination,\nor laboratory testing.\n• The claimed disability and objective findings are markedly discrepant.\n• Patients do not cooperate with efforts to diagnose or treat potential causes of symptoms.\nBody Dysmorphic Disorder\nBody dysmorphic disorder is preoccupation with an imagined or a slight defect in appearance\nthat causes significant distress or impairment of social, occupational, academic, or other')]
Retriever¶
In [ ]:
retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k":3}
)
In [ ]:
rel_docs = retriever.get_relevant_documents(query="What is the protocol for managing sepsis in a critical care unit?")
rel_docs
Out[ ]:
[Document(metadata={'modDate': 'D:20251113011247Z', 'creationdate': '2012-06-15T05:44:40+00:00', 'subject': '', 'trapped': '', 'keywords': '', 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'total_pages': 4114, 'format': 'PDF 1.7', 'page': 2400, 'creationDate': 'D:20120615054440Z', 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'creator': 'Atop CHM to PDF Converter', 'author': '', 'moddate': '2025-11-13T01:12:47+00:00'}, page_content="16 - Critical Care Medicine\nChapter 222. Approach to the Critically Ill Patient\nIntroduction\nCritical care medicine specializes in caring for the most seriously ill patients. These patients are best\ntreated in an ICU staffed by experienced personnel. Some hospitals maintain separate units for special\npopulations (eg, cardiac, surgical, neurologic, pediatric, or neonatal patients). ICUs have a high\nnurse:patient ratio to provide the necessary high intensity of service, including treatment and monitoring\nof physiologic parameters.\nSupportive care for the ICU patient includes provision of adequate nutrition (see p. 21) and prevention of\ninfection, stress ulcers and gastritis (see p. 131), and pulmonary embolism (see p. 1920). Because 15 to\n25% of patients admitted to ICUs die there, physicians should know how to minimize suffering and help\ndying patients maintain dignity (see p. 3480).\nPatient Monitoring and Testing\nSome monitoring is manual (ie, by direct observation and physical examination) and intermittent, with the\nfrequency depending on the patient's illness. This monitoring usually includes measurement of vital signs\n(temperature, BP, pulse, and respiration rate), quantification of all fluid intake and output, and often daily\nweight. BP may be recorded by an automated sphygmomanometer; a transcutaneous sensor for pulse\noximetry is used as well.\nOther monitoring is ongoing and continuous, provided by complex devices that require special training\nand experience to operate. Most such devices generate an alarm if certain physiologic parameters are\nexceeded. Every ICU should strictly follow protocols for investigating alarms.\nBlood Tests\nAlthough frequent blood draws can destroy veins, cause pain, and lead to anemia, ICU patients typically\nhave routine daily blood tests to help detect problems early. Generally, patients need a daily set of\nelectrolytes and a CBC. Patients with arrhythmias should also have Mg, phosphate, and Ca levels\nmeasured. Patients receiving TPN need weekly liver enzymes and coagulation profiles. Other tests (eg,\nblood culture for fever, CBC after a bleeding episode) are done as needed.\nPoint-of-care testing uses miniaturized, highly automated devices to do certain blood tests at the patient's\nbedside or unit (particularly ICU, emergency department, and operating room). Commonly available tests"),
Document(metadata={'trapped': '', 'creationDate': 'D:20120615054440Z', 'page': 2456, 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'creator': 'Atop CHM to PDF Converter', 'keywords': '', 'moddate': '2025-11-13T01:12:47+00:00', 'creationdate': '2012-06-15T05:44:40+00:00', 'format': 'PDF 1.7', 'modDate': 'D:20251113011247Z', 'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'subject': '', 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'author': '', 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'total_pages': 4114}, page_content='thrombocytopenia or recent GI bleeding, receiving concurrent heparin, or with recent aspirin or other\nanticoagulant use).\nOther emerging therapies for severe sepsis include cooling for hyperthermia and early treatment of renal\nfailure (eg, with continuous venovenous hemofiltration).\nTrials of monoclonal antibodies to the lipid A fraction of endotoxin, antileukotrienes, and antibodies to\ntumor necrosis factor have been unsuccessful.\nThe Merck Manual of Diagnosis & Therapy, 19th Edition\nChapter 227. Sepsis & Septic Shock\n2447\nyi.erika0220@gmail.com\n9UQFXT5OPG\nThis file is meant for personal use by yi.erika0220@gmail.com only.\nSharing or publishing the contents in part or full is liable for legal action.'),
Document(metadata={'file_path': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'moddate': '2025-11-13T01:12:47+00:00', 'format': 'PDF 1.7', 'modDate': 'D:20251113011247Z', 'creator': 'Atop CHM to PDF Converter', 'creationDate': 'D:20120615054440Z', 'trapped': '', 'subject': '', 'author': '', 'source': '/content/drive/MyDrive/Colab Notebooks/medical_diagnosis_manual.pdf', 'keywords': '', 'total_pages': 4114, 'title': 'The Merck Manual of Diagnosis & Therapy, 19th Edition', 'producer': 'pdf-lib (https://github.com/Hopding/pdf-lib)', 'page': 1307, 'creationdate': '2012-06-15T05:44:40+00:00'}, page_content='shaking chills, persistent fever, altered sensorium, hypotension, and GI symptoms (abdominal pain,\nnausea, vomiting, diarrhea) suggests sepsis or septic shock. Septic shock develops in 25 to 40% of\npatients with significant bacteremia.\nDiagnosis\nIf bacteremia, sepsis, or septic shock is suspected, cultures are obtained of blood and any other\nappropriate specimens (see p. 1166).\nTreatment\n• Antibiotics\nIn patients with suspected bacteremia, empiric antibiotics are given after appropriate cultures are\nobtained. Early treatment of bacteremia with an appropriate antimicrobial regimen appears to improve\nsurvival. Continuing therapy involves adjusting antibiotics according to the results of culture and\nsusceptibility testing, surgically draining any abscesses, and usually removing any internal devices that\nare the suspected source of bacteria.\nBiological Warfare and Terrorism\nBiological warfare is the use of microbiological agents for hostile purposes. Such use is contrary to\ninternational law and has rarely taken place during formal warfare in modern history, despite the extensive\npreparations and stockpiling of biological agents carried out during the 20th century by most major\npowers. For a variety of reasons (including uncertain military efficacy and the threat of massive\nretaliation), experts consider the use of biological agents in formal warfare unlikely. The area of most\nconcern is the use of such agents by terrorist groups. Biological agents are thought by some people to be\nan ideal weapon for terrorists. These agents may be delivered clandestinely, and they have delayed\neffects, allowing the user to remain undetected.\nPotential biological agents include anthrax, botulinum toxin, brucellosis, encephalitis, viruses,\nhemorrhagic fever viruses (Ebola and Marburg), plague, tularemia, and smallpox. Each of these agents is\npotentially fatal and, except for anthrax and botulinum toxin, can be passed from person to person.\nAnthrax is of most concern; anthrax spores are relatively easy to prepare and spread through the air,\ncreating the potential for distribution by airplane. Theoretically, 1 kg of anthrax could kill 10,000 people,\nalthough technical difficulties with preparing the spores in a sufficiently fine powder would probably limit')]
In [ ]:
model_output = llm(
"What is the protocol for managing sepsis in a critical care unit?",
max_tokens=128,
temperature=0
)
Llama.generate: prefix-match hit
In [ ]:
model_output['choices'][0]['text']
Out[ ]:
'\n\nSepsis is a life-threatening condition that can arise from an infection, and it requires prompt recognition and aggressive management in a critical care unit. The following are general steps for managing sepsis in a critical care unit:\n\n1. Early recognition: Recognize the signs and symptoms of sepsis early and initiate treatment as soon as possible. Sepsis can present with various clinical features, including fever or hypothermia, tachycardia or bradycardia, altered mental status, respiratory distress, and lactic acidosis.\n2. Source control'
System and User Prompt Template¶
In [ ]:
qna_system_message = "Describe the assistant's role"
In [ ]:
qna_user_message_template = """
{context}
{question}
"""
Response Function¶
In [ ]:
def generate_rag_response(user_input,k=3,max_tokens=128,temperature=0,top_p=0.95,top_k=50):
global qna_system_message,qna_user_message_template
# Retrieve relevant document chunks
relevant_document_chunks = retriever.get_relevant_documents(query=user_input,k=k)
context_list = [d.page_content for d in relevant_document_chunks]
# Combine document chunks into a single context
context_for_query = ". ".join(context_list)
user_message = qna_user_message_template.replace('{context}', context_for_query)
user_message = user_message.replace('{question}', user_input)
prompt = qna_system_message + '\n' + user_message
# Generate the response
try:
response = llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k
)
# Extract and print the model's response
response = response['choices'][0]['text'].strip()
except Exception as e:
response = f'Sorry, I encountered the following error: \n {e}'
return response
Question Answering using RAG¶
Query 1: What is the protocol for managing sepsis in a critical care unit?¶
In [ ]:
user_input = "What is the protocol for managing sepsis in a critical care unit?"
generate_rag_response(user_input,top_k=20)
Llama.generate: prefix-match hit
Out[ ]:
'The protocol for managing sepsis in a critical care unit includes the following steps:\n1. Suspect sepsis or septic shock based on clinical signs and symptoms such as shaking chills, persistent fever, altered sensorium, hypotension, and GI symptoms (abdominal pain, nausea, vomiting, diarrhea).\n2. Obtain cultures of blood and any other appropriate specimens for laboratory analysis.\n3. Administer empiric antibiotics based on the suspected organism or microbiological culture results.\n4. Monitor vital signs closely and provide supportive care'
Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶
In [ ]:
user_input = "What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?"
generate_rag_response(user_input,top_k=20)
Llama.generate: prefix-match hit
Out[ ]:
"The common symptoms for appendicitis include epigastric or periumbilical pain followed by brief nausea, vomiting, and anorexia; after a few hours, the pain shifts to the right lower quadrant. Pain increases with cough and motion. Classic signs are right lower quadrant direct and rebound tenderness located at McBurney's point (junction of the middle and outer thirds of the line joining the umbilicus to the anterior superior spine). Additional signs include pain felt in the right lower quadrant with palpation of the left lower quadrant (Rovsing"
Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶
In [ ]:
user_input = "What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?"
generate_rag_response(user_input,top_k=20)
--------------------------------------------------------------------------- NameError Traceback (most recent call last) /tmp/ipython-input-2142197100.py in <cell line: 0>() 1 user_input = "What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?" ----> 2 generate_rag_response(user_input,top_k=20) /tmp/ipython-input-141734326.py in generate_rag_response(user_input, k, max_tokens, temperature, top_p, top_k) 2 global qna_system_message,qna_user_message_template 3 # Retrieve relevant document chunks ----> 4 relevant_document_chunks = retriever.get_relevant_documents(query=user_input,k=k) 5 context_list = [d.page_content for d in relevant_document_chunks] 6 NameError: name 'retriever' is not defined
Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶
In [ ]:
user_input = "What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?"
generate_rag_response(user_input,top_k=20)
Llama.generate: prefix-match hit
Out[ ]:
"The assistant's role in this scenario would be to provide support and assistance to the person with a brain injury and their family members. This may include:\n1. Helping with daily living activities such as bathing, dressing, feeding, and mobility.\n2. Providing emotional support and counseling to help cope with the trauma and adjust to the new situation.\n3. Assisting with communication needs, such as using alternative methods of communication if speech is impaired.\n4. Coordinating care with healthcare professionals, including neurologists, rehabilitation therapists, and other specialists.\n5. Providing information"
Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶
In [ ]:
user_input = "What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?"
generate_rag_response(user_input,top_k=20)
Llama.generate: prefix-match hit
Out[ ]:
'A person who has fractured their leg during a hiking trip should first be assessed for any life-threatening injuries or conditions. If there are no immediate threats to their safety, the following precautions and treatment steps can be taken:\n\n1. Immobilize the affected leg: Use a splint or sling to prevent further injury and decrease pain. This will also help to prevent fat embolism in long-bone fractures.\n2. Control bleeding: Apply direct pressure to any visible wounds with a clean cloth to stop the bleeding. If the bleeding does not stop, seek medical attention immediately'
Fine-tuning¶
Query 1: What is the protocol for managing sepsis in a critical care unit?¶
In [ ]:
user_input = "What is the protocol for managing sepsis in a critical care unit?"
generate_rag_response(user_input,temperature=0.5)
Llama.generate: prefix-match hit
Out[ ]:
'The protocol for managing sepsis in a critical care unit includes the following steps:\n1. Suspect sepsis or septic shock based on clinical signs and symptoms, such as shaking chills, persistent fever, altered sensorium, hypotension, and GI symptoms (abdominal pain, nausea, vomiting, diarrhea).\n2. Obtain cultures of blood and any other appropriate specimens for laboratory analysis.\n3. Administer empiric antibiotics as soon as possible after appropriate cultures are obtained. The choice of antibiotics depends on the suspected source of infection'
Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶
In [ ]:
user_input_2 = "What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?"
generate_rag_response(user_input_2,temperature=0.5)
Llama.generate: prefix-match hit
Out[ ]:
'The common symptoms for appendicitis include epigastric or periumbilical pain that shifts to the right lower quadrant, anorexia, nausea, vomiting, abdominal tenderness, and a low-grade fever. However, these classic findings appear in less than 50% of patients, and many variations of symptoms and signs occur.\n\nAppendicitis cannot be cured via medicine alone as it requires surgical removal of the appendix to prevent complications such as perforation, gangrene, and peritonitis. The treatment for acute appendicitis'
Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶
In [ ]:
user_input_2 = "What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?"
generate_rag_response(user_input_2,temperature=0.5)
Llama.generate: prefix-match hit
Out[ ]:
'Alopecia Areata\nSudden patchy hair loss, also known as alopecia areata, is a common condition characterized by round or oval areas of complete hair loss on the scalp, beard, or other hair-bearing areas. The underlying cause of alopecia areata is not well understood but is believed to be an autoimmune disorder affecting genetically susceptible individuals exposed to unclear environmental triggers.\nThe diagnosis of alopecia areata is typically made based on the clinical presentation and exclusion of other causes of hair loss. Microscopic hair examination or'
Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶
In [ ]:
user_input_2 = "What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?"
generate_rag_response(user_input_2,temperature=0.5)
Llama.generate: prefix-match hit
Out[ ]:
"The assistant's role in this situation would be to provide support and assistance to the patient as they undergo medical treatment and rehabilitation. This could include helping with activities of daily living, such as bathing, dressing, and feeding, as well as providing emotional support and encouragement during the recovery process. The assistant may also help coordinate appointments with healthcare providers, manage medications, and assist with physical therapy exercises as prescribed by a healthcare professional. Additionally, the assistant could help the patient's family members understand the condition and treatment plan, as well as provide them with resources for support and education. It is important for the"
Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶
In [ ]:
user_input_2 = "What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?"
generate_rag_response(user_input_2,temperature=0.5)
Llama.generate: prefix-match hit
Out[ ]:
'A person who has fractured their leg during a hiking trip requires prompt medical attention. Here are some necessary precautions and treatment steps:\n\n1. Immobilize the affected limb: Use a splint or sling to immobilize the leg and prevent further injury. This will also help reduce pain.\n2. Control bleeding: Apply direct pressure to the wound with a clean cloth to control any bleeding. If bleeding does not stop, seek medical attention immediately.\n3. Pain relief: Administer pain medication as prescribed by a healthcare professional. Over-the-counter pain relievers like acet'
Output Evaluation¶
Let us now use the LLM-as-a-judge method to check the quality of the RAG system on two parameters - retrieval and generation. We illustrate this evaluation based on the answeres generated to the question from the previous section.
- We are using the same Mistral model for evaluation, so basically here the llm is rating itself on how well he has performed in the task.
In [ ]:
groundedness_rater_system_message = ""
In [ ]:
relevance_rater_system_message = ""
In [ ]:
user_message_template = """
###Question
{question}
###Context
{context}
###Answer
{answer}
"""
In [ ]:
def generate_ground_relevance_response(user_input,k=3,max_tokens=128,temperature=0,top_p=0.95,top_k=50):
global qna_system_message,qna_user_message_template
# Retrieve relevant document chunks
relevant_document_chunks = retriever.get_relevant_documents(query=user_input,k=3)
context_list = [d.page_content for d in relevant_document_chunks]
context_for_query = ". ".join(context_list)
# Combine user_prompt and system_message to create the prompt
prompt = f"""[INST]{qna_system_message}\n
{'user'}: {qna_user_message_template.format(context=context_for_query, question=user_input)}
[/INST]"""
response = llm(
prompt=prompt,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
stop=['INST'],
)
answer = response["choices"][0]["text"]
# Combine user_prompt and system_message to create the prompt
groundedness_prompt = f"""[INST]{groundedness_rater_system_message}\n
{'user'}: {user_message_template.format(context=context_for_query, question=user_input, answer=answer)}
[/INST]"""
# Combine user_prompt and system_message to create the prompt
relevance_prompt = f"""[INST]{relevance_rater_system_message}\n
{'user'}: {user_message_template.format(context=context_for_query, question=user_input, answer=answer)}
[/INST]"""
response_1 = llm(
prompt=groundedness_prompt,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
stop=['INST'],
)
response_2 = llm(
prompt=relevance_prompt,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
stop=['INST'],
)
return response_1['choices'][0]['text'],response_2['choices'][0]['text']
Query 1: What is the protocol for managing sepsis in a critical care unit?¶
In [ ]:
ground,rel = generate_ground_relevance_response(user_input="What is the protocol for managing sepsis in a critical care unit?",max_tokens=370)
print(ground,end="\n\n")
print(rel)
Llama.generate: prefix-match hit Llama.generate: prefix-match hit Llama.generate: prefix-match hit
of a biological attack in a critical care unit, strict infection control measures should be implemented to prevent the spread of the agent. This includes wearing appropriate personal protective equipment (PPE), isolating affected patients, and decontaminating the environment. In addition to standard sepsis management, specific treatments for the particular biological agent may be required. It is important for critical care units to have a well-developed response plan in place for dealing with such situations. of a biological attack in a critical care unit, strict infection control measures should be implemented to prevent the spread of the agent. This includes wearing appropriate personal protective equipment (PPE), isolating affected patients, and implementing environmental decontamination procedures. In addition, specific treatments for the particular biological agent may be required, such as antibiotics for anthrax or supportive care for viral hemorrhagic fevers. It is important to note that the use of biological agents in warfare or terrorism is contrary to international law and poses significant risks to public health and safety.
Query 2: What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?¶
In [ ]:
ground,rel = generate_ground_relevance_response(user_input="What are the common symptoms for appendicitis, and can it be cured via medicine? If not, what surgical procedure should be followed to treat it?",max_tokens=370)
print(ground,end="\n\n")
print(rel)
Llama.generate: prefix-match hit Llama.generate: prefix-match hit Llama.generate: prefix-match hit
The common symptoms for appendicitis include epigastric or periumbilical pain that shifts to the right lower quadrant, brief nausea, vomiting, anorexia, and abdominal tenderness, particularly at McBurney's point. Fever is also common. However, these classic findings appear in less than 50% of patients, and symptoms can vary greatly, especially in infants, children, elderly patients, and pregnant women. Appendicitis cannot be cured via medicine alone as it requires surgical removal of the appendix to prevent complications such as perforation, gangrene, and peritonitis. Treatment involves open or laparoscopic appendectomy, which should be preceded by IV antibiotics. If surgery is impossible, antibiotics can improve survival rates but are not curative. In cases where a large inflammatory mass is found involving the appendix, terminal ileum, and cecum, resection of the entire mass and ileocolostomy may be necessary. Hernias of the abdominal wall are a different condition that requires elective surgical repair when they become incarcerated or strangulated. Abdominal hernias are extremely common and necessitate about 700,000 operations each year in the US. The common symptoms for appendicitis include epigastric or periumbilical pain that shifts to the right lower quadrant, brief nausea, vomiting, anorexia, and abdominal tenderness, particularly at McBurney's point. Fever is also common. However, these classic findings appear in less than 50% of patients, and symptoms can vary greatly, especially in infants, children, elderly patients, and pregnant women. Appendicitis cannot be cured via medicine alone as it requires surgical removal of the appendix to prevent complications such as perforation, gangrene, and peritonitis. Treatment involves open or laparoscopic appendectomy, which should be preceded by IV antibiotics. If surgery is impossible, antibiotics can improve survival rates but are not curative. In cases where a large inflammatory mass is found involving the appendix, terminal ileum, and cecum, resection of the entire mass and ileocolostomy may be necessary. Hernias of the abdominal wall are a different condition that requires elective surgical repair when they become incarcerated or strangulated. Abdominal hernias are extremely common and necessitate about 700,000 operations each year in the US.
Query 3: What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?¶
In [ ]:
ground,rel = generate_ground_relevance_response(user_input="What are the effective treatments or solutions for addressing sudden patchy hair loss, commonly seen as localized bald spots on the scalp, and what could be the possible causes behind it?",max_tokens=370)
print(ground,end="\n\n")
print(rel)
Llama.generate: prefix-match hit Llama.generate: prefix-match hit Llama.generate: prefix-match hit
also occur in people with no known family history of the condition, suggesting that other factors may play a role. Stress, illness, or trauma can trigger the onset of alopecia areata in genetically susceptible individuals. In some cases, vitamin deficiencies or certain medications may be contributing factors. A thorough evaluation by a healthcare professional is necessary to determine the underlying cause and develop an appropriate treatment plan. also occur in people with no known family history of the condition, suggesting that other factors may play a role. Stress, illness, or trauma can trigger the onset of alopecia areata in genetically susceptible individuals. In some cases, vitamin deficiencies or certain medications may be contributing factors. A thorough evaluation by a healthcare professional is necessary to determine the underlying cause and develop an appropriate treatment plan.
Query 4: What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?¶
In [ ]:
ground,rel = generate_ground_relevance_response(user_input="What treatments are recommended for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function?",max_tokens=370)
print(ground,end="\n\n")
print(rel)
Llama.generate: prefix-match hit Llama.generate: prefix-match hit Llama.generate: prefix-match hit
You have provided an accurate summary of the recommended treatments for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function. The treatments include supportive care, specific treatments based on the type and cause of the brain injury, rehabilitation, medications, and long-term care. Supportive care focuses on maintaining the patient's overall health and preventing further damage to the brain. Specific treatments may involve surgical intervention for certain types of brain injuries. Rehabilitation is necessary when neurologic deficits persist, and it should be provided through a team approach that combines physical, occupational, and speech therapy, skill-building activities, and counseling. Medications may be prescribed to manage specific symptoms and conditions. Long-term care is often required for patients with persistent neurologic deficits. It's important to note that each case is unique, and treatment plans may vary depending on the individual's specific condition. You have provided an accurate summary of the recommended treatments for a person who has sustained a physical injury to brain tissue, resulting in temporary or permanent impairment of brain function. The treatments include supportive care, specific treatments based on the type and cause of the brain injury, rehabilitation, medications, and long-term care. Supportive care focuses on maintaining the patient's overall health and preventing further damage to the brain. Specific treatments may involve surgical intervention for certain types of brain injuries. Rehabilitation is necessary when neurologic deficits persist, and it should be provided through a team approach that combines physical, occupational, and speech therapy, skill-building activities, and counseling. Medications may be prescribed to manage specific symptoms and conditions. Long-term care is often required for patients with persistent neurologic deficits. It's important to note that each case is unique, and treatment plans may vary depending on the individual's specific condition.
Query 5: What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?¶
In [ ]:
ground,rel = generate_ground_relevance_response(user_input="What are the necessary precautions and treatment steps for a person who has fractured their leg during a hiking trip, and what should be considered for their care and recovery?",max_tokens=370)
print(ground,end="\n\n")
print(rel)
Llama.generate: prefix-match hit Llama.generate: prefix-match hit Llama.generate: prefix-match hit
Additionally, it's important to consider the risk of Deep Vein Thrombosis (DVT) during the recovery process. While walking or otherwise moving the legs periodically can help reduce this risk, those at higher risk may require additional preventive treatment such as thrombosis prophylaxis with medications like low-dose unfractionated heparin (UFH), low molecular weight heparin (LMWH), warfarin, or newer anticoagulants. Consult a healthcare professional for specific recommendations based on the individual's risk factors and injury severity. Here are some additional steps to prevent DVT: 1. Elevate the legs: Keep the legs elevated above heart level as much as possible when resting. 2. Avoid prolonged sitting or standing: Try to avoid long periods of inactivity, especially during travel or extended recovery periods. 3. Exercise regularly: Engage in regular lower extremity exercises, such as ankle pumps and leg extensions, to promote blood flow and prevent stagnation. 4. Wear compression stockings: Consider wearing compression stockings to help improve circulation and reduce the risk of DVT. 5. Maintain a healthy lifestyle: Eat a balanced diet, stay hydrated, and avoid smoking or excessive alcohol consumption to support overall health and reduce the risk of complications during recovery. Additionally, it's important to consider the risk of Deep Vein Thrombosis (DVT) during the recovery process. While walking or otherwise moving the legs periodically can help reduce this risk, those at higher risk may require additional preventive treatment such as thrombosis prophylaxis with medications like low-dose unfractionated heparin (UFH), low molecular weight heparin (LMWH), warfarin, or newer anticoagulants. Consult a healthcare professional for specific recommendations based on the individual's risk factors and injury severity. Here is a summary of the necessary precautions and treatment steps: 1. Assess the severity of the injury: Determine if it is a closed or open fracture, and assess for any signs of nerve or artery damage. Seek emergency medical help if there are signs of severe bleeding, shock, or nerve damage. 2. Immobilize the leg: Use a splint or sling to immobilize the leg and prevent further injury. 3. Rest and elevate the leg: Encourage the person to rest and elevate their leg as much as possible to reduce swelling and promote healing. 4. Apply ice: Apply ice to the injured area for 15-20 minutes at a time, several times a day, to help reduce swelling and pain. 5. Pain management: Prescribe NSAIDs or analgesics as necessary to manage pain. Opioids may be required for severe pain. 6. Consider surgical intervention: Depending on the severity and location of the fracture, surgery may be necessary to realign the bone and ensure proper healing. 7. Follow-up care: Schedule regular follow
Actionable Insights and Business Recommendations¶
Power Ahead
Understand disorders