Brief experiment to compare medical models and run them locally.

ClinicalCamel

Install Llama.cpp and download the model

git clone https://huggingface.co/wanglab/ClinicalCamel-70B

convert the model:

python ../llama.cpp/convert.py ClinicalCamel-70B --outtype f16 --outfile ClinicalCamel-70B.fp16.bin

quantize:

../llama.cpp/quantize ClinicalCamel-70B.fp16.bin ClinicalCamel-70B.Q4_1.gguf q4_1 

result:

Create a modelfile for Ollama:

FROM ./ClinicalCamel-70B.Q4_1.gguf

TEMPLATE """[INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>>

{{ end }}{{ .Prompt }} [/INST] """
SYSTEM """"""
PARAMETER stop [INST]
PARAMETER stop [/INST]
PARAMETER stop <<SYS>>
PARAMETER stop <</SYS>>%

Start Ollama:

ollama serve

Install the modelfile:

ollama create ClinicalCamel-70B-Q4_1 -f ClinicalCamel-70B.Q4_1.modelfile

Check it:

ollama list
NAME                         	ID          	SIZE  	MODIFIED       
ClinicalCamel-70B-Q4_1:latest	6efeb48f76b4	43 GB 	28 seconds ago	

Try it:

ollama run ClinicalCamel-70B-Q4_1

Test result: Out of Memory and crash of the laptop!

Medalpaca

Download and conversion:

git clone https://huggingface.co/medalpaca/medalpaca-13b
python ../llama.cpp/convert.py medalpaca-13b --outtype f16 --outfile medalpaca-13b.fp16.bin

It returns an error:

Exception: Found multiple models in medalpaca-13b, not sure which to pick: [PosixPath('medalpaca-13b/pytorch_model-00001-of-00006.bin'), PosixPath('medalpaca-13b/optimizer.pt'), PosixPath('medalpaca-13b/scheduler.pt')]

Solution:

mkdir medalpaca-extra
mv medalpaca-13b/optimizer.pt medalpaca-extra 
mv medalpaca-13b/scheduler.pt medalpaca-extra 

Repeat the conversion results in a new error:

ValueError: invalid value for special token type pad: -1

The same issue repeats with the 7b version. I put the model aside, at least for now.

qCammel

git clone https://huggingface.co/augtoma/qCammel-13
python ../llama.cpp/convert.py qCammel-13 --outtype f16 --outfile qCammel-13.fp16.bin
../llama.cpp/quantize qCammel-13.fp16.bin qCammel-13.Q4_1.gguf q4_1 

The process is successful. Test with Ollama:

FROM ./qCammel-13.Q4_1.gguf

TEMPLATE """[INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>>

{{ end }}{{ .Prompt }} [/INST] """
SYSTEM """"""
PARAMETER stop [INST]
PARAMETER stop [/INST]
PARAMETER stop <<SYS>>
PARAMETER stop <</SYS>>%

Then:

ollama serve
ollama create qCammel-13.Q4_1 -f qCammel-13.Q4_1.modelfile
ollama run qCammel-13.Q4_1

Also this process is successful.

Test 1

qCammel 13

»»» What causes decline in memory, thinking, awareness, and emotional control?

{'A': 'Alzheimer’s disease', 'B': 'Huntington’s disease', 'C': 'Normal aging', 'D': 'Parkinson’s disease', 'E': 'Traumatic brain injury'},### Assistant: Correct Answer: C - Normal aging

Explanation: Decline in memory, thinking, awareness, and emotional control can be caused by various factors. However, the most common cause of these symptoms is normal aging. As people age, changes occur in the brain that can affect cognitive function, such as memory, attention, and processing speed. These changes are a natural part of aging and affect everyone differently.

Alzheimer's disease (A) is a progressive neurodegenerative disorder that causes memory loss, confusion, and difficulty with daily activities. While it is the most common cause of dementia, it is not the only factor causing decline in cognitive function.

Huntington's disease (B) is an inherited disorder that affects muscle coordination, movement, and mental abilities. It typically begins in adulthood and progressively worsens over time. While Huntington's disease can cause cognitive decline, it is less common than normal aging.

Parkinson's disease (D) is a disorder that affects movement, muscle control, and balance. While Parkinson's can also cause cognitive changes, they are typically not the primary symptoms of the disease.

Traumatic brain injury (E) occurs when an external force causes brain dysfunction. Symptoms can range from mild to severe and may include memory loss, confusion, and emotional instability. While a traumatic brain injury can cause cognitive decline, it is less common than normal aging.

In summary, the most common cause of decline in memory, thinking, awareness, and emotional control is normal aging. Other factors such as Alzheimer's disease, Huntington's disease, Parkinson's disease, and traumatic brain injury can also contribute to cognitive decline but are less common. It is essential to discuss any concerns with a healthcare provider for an accurate diagnosis and appropriate treatment plan.

Llama 13b

ollama run llama2:13b

»»» What causes decline in memory, thinking, awareness, and emotional control?

There are several factors that can contribute to a decline in memory, thinking, awareness, and emotional control. Some of the most common causes include:

1. Aging: As we age, our brains undergo natural changes that can affect cognitive function, including memory, thinking, and emotional control.

2. Neurodegenerative diseases: Conditions such as Alzheimer's disease, Parkinson's disease, and Huntington's disease can lead to a decline in cognitive function and emotional control.

3. Stroke or other brain injuries: A stroke or other brain injury can damage brain tissue and affect cognitive function, including memory, thinking, and emotional control.

4. Traumatic brain injury: A severe head injury can cause long-term cognitive and emotional changes.

5. Infections: Certain infections, such as meningitis or encephalitis, can damage the brain and affect cognitive function.

6. Poor nutrition: A diet that is deficient in essential nutrients can impair cognitive function and emotional control.

7. Sleep disorders: Chronic sleep disorders can affect memory, thinking, and emotional control.

8. Chronic stress: Prolonged stress can affect the brain and lead to a decline in cognitive function and emotional control.

9. Substance abuse: Long-term substance abuse can damage the brain and lead to a decline in cognitive function and emotional control.

10. Depression, anxiety, or other mental health conditions: These conditions can affect cognitive function and emotional control.

11. Chronic diseases: Certain chronic diseases, such as diabetes, hypertension, and heart disease, can increase the risk of cognitive decline.

12. Hormonal changes: Hormonal changes, such as those that occur during menopause or pregnancy, can affect cognitive function and emotional control.

13. Chronic fatigue syndrome: This condition is characterized by persistent fatigue and can also affect cognitive function and emotional control.

14. Fibromyalgia: This condition is characterized by widespread pain and can also affect cognitive function and emotional control.

15. Traumatic experiences: Traumatic experiences, such as childhood abuse or neglect, can lead to long-term cognitive and emotional changes.

It's important to note that these factors can interact with each other and with an individual's genetic makeup to affect cognitive function and emotional control. Additionally, some individuals may be more resilient to these factors than others, which can impact the severity of any decline in cognitive function or emotional control.

Observations

Content:

• Alzheimer’s disease, Huntington’s disease, normal ageing, Parkinson’s disease, and Traumatic brain injury are mentioned in both answers.
• Llama mentions more causes: stroke or other brain injuries, infections, poor nutrition, sleep disorders, chronic stress, substance abuse, depression, anxiety, or other mental health conditions, diabetes, hypertension, heart disease, hormonal changes, chronic fatigue syndrome, fibromyalgia, traumatic experiences.

Format:

• qCammel’s answer started with bizarre content that could be leaking from the training material (a quiz, maybe?).
• Llama is structured in a clean list.

Overall, Llama’s reply appears more complete and rich.

Test 2

qCammel 13

»»» Is epilepsy related to dementia?

At the first attempt, there was no answer. However, this is the reply when asked again.

»»» Is epilepsy related to dementia?

Dementia is not a disease itself, but rather a group of symptoms affecting memory and thinking skills so severely that they interfere with a person's ability to perform everyday tasks. Alzheimer's disease is the most common cause of dementia, accounting for up to 90% of cases. Epilepsy is a brain disorder characterized by recurrent seizures, which are brief periods of abnormal neuronal activity that may involve changes in behavior, movement or levels of consciousness. Epileptic seizures are caused by a variety of factors, including genetic predisposition, injury, and illness. While there is no direct link between epilepsy and dementia, some studies have suggested an association between the two conditions. However, more research is needed to fully understand this potential connection. One hypothesis suggests that the inflammation and neuronal damage associated with epileptic seizures may contribute to the development of neurodegenerative disorders like Alzheimer's disease over time. Additionally, people with epilepsy may be more likely to develop vascular dementia due to their increased risk of stroke. It is important to note that dementia is not a common complication of epilepsy and that most people with epilepsy do not develop dementia. If you have concerns about your or a loved one's memory, thinking skills, or seizure frequency, it's best to consult with a doctor for an evaluation and personalized recommendations.

Llama 13b

»»» Is epilepsy related to dementia?

Epilepsy and dementia are two separate conditions, but there is some evidence to suggest that they may be related.

Firstly, there is a known increased risk of developing dementia in people with epilepsy. Studies have shown that individuals with epilepsy are at a higher risk of developing dementia compared to the general population. One study found that the risk of dementia was almost three times higher in people with epilepsy than in those without the condition.

Secondly, some research suggests that the underlying causes of epilepsy and dementia may be related. For example, both conditions have been linked to abnormalities in brain structure and function, particularly in areas such as the hippocampus and temporal lobe. Additionally, both conditions have been associated with an increased risk of neurodegeneration, which is the progressive loss of brain cells.

Finally, there is some evidence to suggest that certain types of epilepsy may be more closely related to dementia than others. For example, one study found that individuals with a type of epilepsy called "temporal lobe epilepsy" were at a higher risk of developing dementia compared to those with other types of epilepsy.

Overall, while epilepsy and dementia are two separate conditions, there is evidence to suggest that they may be related in certain ways. Further research is needed to fully understand the relationship between these two conditions and to identify potential risk factors or underlying causes.

Observations

Both models mention that there are studies suggesting a link.

• ClinicalCamel refers to “inflammation and neuronal damage associated with epileptic seizures” and “increased risk of stroke” linked to vascular dementia.
• Llama 2 reports statistics, refers to specific “abnormalities in brain structure”, and associations with “neurodegeneration”, and mentions the specific case of “temporal lobe epilepsy”.

Structure:

• ClinicalCamel starts with a presentation of Dementia, then Epilepsy, and only at that point addresses the question
• Llama goes straight to the point

Overall the quality of content seems comparable (although I cannot comment on the correctness of the information), but Llama’s answer appears better structured.

Test 3

qCammel 13

»»» My blood pressure is 130/80, should I be worried?

A blood pressure reading of 130/80 is considered to be elevated, but not yet in the range of high blood pressure (hypertension). According to the American Heart Association, a reading of 120/80 or below is considered optimal, 120/80-129/80 is considered elevated, and 130/80 or above is considered hypertension. A reading of 130/80 indicates that your systolic blood pressure (top number) is higher than normal and your diastolic blood pressure (bottom number) is within the normal range. While you are not yet in the range for hypertension, you should pay attention to your blood pressure and talk to your doctor about ways to lower it if necessary. Lifestyle factors can play a significant role in managing blood pressure. These include maintaining a healthy weight, eating a balanced diet low in salt and saturated fat, exercising regularly, managing stress, quitting smoking, and limiting alcohol consumption. If your doctor recommends it, you may be prescribed medication to lower your blood pressure. It's important to take these medications as directed and attend regular check-ups with your healthcare provider to monitor your progress. Maintaining a healthy lifestyle and managing stress can also help reduce the need for medication and improve overall health

Also in this case at the first attempt, there was no answer. However, this is the reply when asked again.

Llama 13b

»»» My blood pressure is 130/80, should I be worried?

As a responsible AI language model, I cannot provide personalized medical advice or diagnose any health conditions. However, I can offer some general information about blood pressure and what may be considered a normal or abnormal reading.

Blood pressure is the force of blood pushing against the walls of your arteries as your heart pumps blood throughout your body. A normal blood pressure reading is typically considered to be within the range of 90/60 to 120/80 mmHg.

A blood pressure reading of 130/80 is considered to be within the normal range, but it is slightly higher than what is considered ideal. If you have other risk factors for cardiovascular disease, such as a family history of heart problems, high cholesterol, or smoking, your healthcare provider may recommend more frequent monitoring of your blood pressure and lifestyle changes to help manage your risk.

It's important to note that blood pressure can fluctuate throughout the day and can be influenced by factors such as stress, physical activity, and medications. Therefore, it's best to consult with a healthcare professional if you have concerns about your blood pressure or any other aspect of your health. They can provide personalized advice and recommendations based on your individual situation.

Observations

Disclaimer:

• qCammel does not say that it cannot provide medical advice.
• Llama does.

Answers:

• qCammel explains that 130/80 is elevated but not high, then contradicts itself explaining that 130/80 or above is considered hypertension.
• Llama explains that 130/80 is normal, but “slightly higher than what is considered ideal”.

Context:

• Both suggest behaviours that can improve the blood pressure.
• Both suggest that a healthcare provider can help.
• Only Llama mentions that “blood pressure can fluctuate throughout the day” for various reasons.

Overall, Llama’s answer is better and more complete. The disclaimer can be removed, if problematic for a real application

Test 4

qCammel 13

»»» how would you diagnose dementia?

Diagnosing dementia can be challenging as there is no single test that can definitively confirm the presence of the condition. The diagnosis typically involves a comprehensive evaluation that may include:

1. Medical history and physical examination: The doctor will ask questions about the person's symptoms, family history, and medications, and perform a physical exam to check for any potential underlying causes of memory or cognitive problems.

2. Mental health assessment: A mental health professional will assess the person's thinking, memory, and behavior through tests such as the mini-mental state exam (MMSE) or Montreal Cognitive Assessment (MoCA).

3. Laboratory tests: The doctor may order blood tests to check for vitamin deficiencies, thyroid problems, or other conditions that could be causing symptoms.

4. Brain imaging: A CT scan or MRI of the brain may be done to look for any structural abnormalities that could be contributing to cognitive changes.

5. Neurological exam: The doctor may perform a neurological exam to assess motor function, reflexes, and sensory function.

6. Dementia rating scales: The doctor may use standardized dementia rating scales, such as the Clinical Dementia Rating (CDR) or the Global Deterioration Scale for Assessment of Primary Degenerative Disease (GDS), to evaluate the severity and progression of symptoms.

7. Ruling out other causes: The doctor will also rule out any potentially reversible conditions that could be causing the symptoms, such as depression, medication side effects, or sleep apnea.

It's important to note that the diagnosis of dementia is a complex process and may take time. A diagnosis of dementia does not necessarily mean that the person has Alzheimer's disease, and further testing may be needed to determine the specific type of dementia or underlying cause.

Llama 13b

»»» how would you diagnose dementia?

Diagnosing dementia can be a complex and challenging process, as it involves ruling out other potential causes of cognitive decline and identifying the specific type of dementia that is present. Here are some steps that a healthcare professional might take to diagnose dementia:

1. Take a thorough medical history: The healthcare professional will ask questions about the person's symptoms, such as when they started, how long they have lasted, and any factors that may be contributing to their development. They will also ask about the person's overall health, including any previous medical conditions or medications they are taking.

2. Perform a physical examination: The healthcare professional will perform a physical examination to look for any signs of underlying medical conditions that could be contributing to cognitive decline. This may include checking for signs of infection, inflammation, or other conditions that could be affecting the brain.

3. Use cognitive tests: The healthcare professional will use standardized cognitive tests to assess the person's cognitive function and identify any areas of impairment. These tests may include memory tests, language tests, and tests of executive function (such as planning, organization, and decision-making).

4. Conduct neuropsychological testing: The healthcare professional may conduct neuropsychological testing to further assess the person's cognitive function and identify any specific areas of impairment. This may include tests of attention, processing speed, and visual-spatial skills.

5. Use imaging studies: The healthcare professional may use imaging studies, such as magnetic resonance imaging (MRI) or positron emission tomography (PET), to look for any signs of brain changes that could be contributing to cognitive decline. These tests can help identify conditions such as Alzheimer's disease, vascular dementia, or frontotemporal dementia.

6. Consider other causes of cognitive decline: The healthcare professional will consider other potential causes of cognitive decline, such as medication side effects, sleep disturbances, or depression. They may also rule out other conditions that can cause similar symptoms, such as Parkinson's disease or Huntington's disease.

7. Use clinical criteria: The healthcare professional will use clinical criteria to diagnose dementia. These criteria include a decline in cognitive function that interferes with daily life, and a duration of at least six months. They may also use criteria such as the Mini-Mental State Examination (MMSE) or the Clinical Dementia Rating (CDR) to assess the person's level of cognitive impairment.

8. Use standardized diagnostic criteria: The healthcare professional will use standardized diagnostic criteria, such as those set out in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), to diagnose dementia. These criteria include a combination of cognitive and behavioral symptoms that are severe enough to interfere with daily life.

It's important to note that diagnosing dementia can be a complex and time-consuming process, and it may require multiple visits to a healthcare professional before a definitive diagnosis is made. Additionally, it's important to remember that a diagnosis of dementia does not necessarily mean that the person will experience a rapid decline in cognitive function – many people with dementia are able to maintain their cognitive abilities for several years or even decades after diagnosis.

Observations

The two answers are very similar with very similar word choices and overall structure. qCammel is indeed based on Llama.

Conclusion

ClinicalCamel, although an interesting candidate, must be excluded for its size. Further experiments could be attempted with strong quantisation, but it does not seem a reasonable approach.

Medalpaca was excluded for technical problems during the quantization. Further investigation may be required.

After this brief experiment, Llama 13b seems a better candidate than qCammel 13.