Create README.md

README.md

@@ -0,0 +1,304 @@
+---
+license: cc
+datasets:
+- vector-institute/newsmediabias-plus
+---
+
+```markdown
+---
+language: en
+tags:
+  - transformers
+  - llama
+  - multimodal
+  - vision-language
+  - newsmedia
+  - disinformation-detection
+  - text-classification
+  - image-classification
+license: apache-2.0
+datasets:
+  - https://huggingface.co/datasets/vector-institute/newsmediabias-plus
+library_name: transformers
+pipeline_tag: text-classification
+---
+
+# LLaMA 3.2 Multimodal News Media Bias Detector
+
+**Developed by the Vector Institute**
+
+## Model Overview
+
+The LLaMA 3.2 Multimodal News Media Bias Detector is a fine-tuned version of the LLaMA 2 Vision-Instruct model, designed to assess potential disinformation in news media articles by analyzing both text and image content. The model identifies the presence of rhetorical techniques commonly associated with disinformation and classifies articles as 'Likely' or 'Unlikely' to be disinformation.
+
+## Model Details
+
+- **Base Model**: [meta-llama/Llama-3.2-11B-Vision-Instruct](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct)
+- **Architecture**: Multimodal (Text and Image)
+- **Model Size**: 11B parameters
+- **Quantization**: 4-bit quantization (`nf4`) using `bitsandbytes` for efficient inference
+- **Framework**: [Hugging Face Transformers](https://github.com/huggingface/transformers)
+- **Fine-tuning**: The model was fine-tuned using the [PEFT](https://github.com/huggingface/peft) library with LoRA adapters
+
+## Intended Use
+
+### Primary Use Cases
+
+- **Disinformation Detection**: Analyze news articles and accompanying images to detect potential disinformation.
+- **Media Analysis**: Assist researchers and analysts in studying bias and rhetorical techniques in media content.
+- **Educational Tool**: Serve as a resource for learning about multimodal models and disinformation detection.
+
+### Limitations
+
+- **Not a Fact-Checker**: The model does not verify factual accuracy but assesses rhetorical techniques.
+- **Bias in Training Data**: The model's judgments may reflect biases present in the training data.
+- **Language and Domain Limitations**: Primarily trained on English-language news articles; performance may degrade on other languages or domains.
+
+## How to Use
+
+### Installation
+
+Ensure you have the required libraries installed:
+
+```bash
+pip install transformers accelerate bitsandbytes
+```
+
+### Loading the Model
+
+import torch
+from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig
+
+merged_model_save_path= "vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector"
+# Load quantization config if used during saving
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+
+# Load the merged model
+model = AutoModelForVision2Seq.from_pretrained(
+    merged_model_save_path,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    quantization_config=bnb_config  
+)
+model.eval()
+
+# Load the processor
+processor = AutoProcessor.from_pretrained(merged_model_save_path)
+
+import pandas as pd
+import os
+from PIL import Image
+
+# Path to your dataset CSV and image folder
+dataset_csv_path = 'sample_dataset.csv'
+image_folder_path = 'sampled_images'
+
+# Load the DataFrame
+df = pd.read_csv(dataset_csv_path)
+
+# Function to prepare samples from the DataFrame
+def prepare_samples_from_dataframe(df, image_folder_path):
+    samples = []
+    for index, row in df.iterrows():
+        unique_id = row['unique_id']  # Replace with the column that contains the image identifier
+        text_content = row['first_paragraph']  # Replace with your text column name
+
+        # Find the image file
+        possible_extensions = ['jpg', 'jpeg', 'png']
+        image_path = None
+        for ext in possible_extensions:
+            img_path = os.path.join(image_folder_path, f"{unique_id}.{ext}")
+            if os.path.exists(img_path):
+                image_path = img_path
+                break
+
+        if image_path is None:
+            print(f"No image found for ID {unique_id}")
+            continue
+
+        image = Image.open(image_path).convert("RGB")
+        max_size = (224, 224)
+        image.thumbnail(max_size, Image.Resampling.LANCZOS)
+
+        # Prepare the prompt text
+        sample_text = (
+            "Assess the text and image below for potential disinformation (try finding deliberately misleading or biased information) by identifying the presence of rhetorical techniques listed.\n"
+            "If you find any of the listed rhetorical techniques, then the article is likely disinformation; if not, it is likely not disinformation.\n\n"
+            "Rhetorical Techniques Checklist:\n"
+            "- Emotional Appeal: Uses language that intentionally invokes extreme emotions like fear or anger, aiming to distract from lack of factual backing.\n"
+            "- Exaggeration and Hyperbole: Makes claims that are unsupported by evidence, or presents normal situations as extraordinary to manipulate perceptions.\n"
+            "- Bias and Subjectivity: Presents information in a way that unreasonably favors one perspective, omitting key facts that might provide balance.\n"
+            "- Repetition: Uses repeated messaging of specific points or misleading statements to embed a biased viewpoint in the reader's mind.\n"
+            "- Specific Word Choices: Employs emotionally charged or misleading terms to sway opinions subtly, often in a manipulative manner.\n"
+            "- Appeals to Authority: References authorities who lack relevant expertise or cites sources that do not have the credentials to be considered authoritative in the context.\n"
+            "- Lack of Verifiable Sources: Relies on sources that either cannot be verified or do not exist, suggesting a fabrication of information.\n"
+            "- Logical Fallacies: Engages in flawed reasoning such as circular reasoning, strawman arguments, or ad hominem attacks that undermine logical debate.\n"
+            "- Conspiracy Theories: Propagates theories that lack proof and often contain elements of paranoia or implausible scenarios as facts.\n"
+            "- Inconsistencies and Factual Errors: Contains multiple contradictions or factual inaccuracies that are easily disprovable, indicating a lack of concern for truth.\n"
+            "- Selective Omission: Deliberately leaves out crucial information that is essential for a fair understanding of the topic, skewing perception.\n"
+            "- Manipulative Framing: Frames issues in a way that leaves out alternative perspectives or possible explanations, focusing only on aspects that support a biased narrative.\n\n"
+            f"{text_content}\n\n"
+            "Please **only** provide your answer in the format: 'Classification: Likely' or 'Classification: Unlikely'. Do not include any additional text or explanation."
+        )
+
+        sample = {
+            "unique_id": unique_id,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": [
+                        {
+                            "type": "text",
+                            "text": sample_text,
+                        },
+                        {
+                            "type": "image",
+                            "image": image,
+                        }
+                    ],
+                }
+            ]
+        }
+
+        samples.append(sample)
+
+    return samples
+
+import re
+
+def extract_assistant_reply(generated_text):
+    lines = generated_text.strip().split('\n')
+    for idx, line in enumerate(lines):
+        if line.strip().lower() == 'assistant':
+            assistant_reply = '\n'.join(lines[idx+1:]).strip()
+            return assistant_reply
+    return lines[-1].strip()
+
+def extract_classification(assistant_reply):
+    match = re.search(r'Classification:\s*(Likely|Unlikely)', assistant_reply, re.IGNORECASE)
+    if match:
+        return match.group(1).capitalize()
+    else:
+        # Attempt to find 'Likely' or 'Unlikely' anywhere in the reply
+        match = re.search(r'\b(Likely|Unlikely)\b', assistant_reply, re.IGNORECASE)
+        if match:
+            return match.group(1).capitalize()
+        else:
+            return 'Unknown'
+
+def generate_prediction(sample):
+    # Remove the model.to(device) line
+    # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    # model.to(device)
+
+    # Prepare the input texts and images
+    texts = processor.apply_chat_template(sample["messages"], tokenize=False)
+    image_input = sample["messages"][0]['content'][1]['image']
+
+    # Prepare the inputs without moving them to a device
+    inputs = processor(text=texts, images=image_input, return_tensors="pt", padding=True)
+
+    # Ensure that the inputs are on the same device as the model's embeddings
+    # This is handled internally by the model when using device_map="auto"
+
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=20,
+            do_sample=False,
+            num_beams=1,
+        )
+
+    generated_texts = processor.batch_decode(outputs, skip_special_tokens=True)
+    assistant_reply = extract_assistant_reply(generated_texts[0])
+
+    return assistant_reply
+
+# Prepare the samples from the DataFrame
+samples = prepare_samples_from_dataframe(df, image_folder_path)
+
+# Limit to a subset if desired
+samples_to_infer = samples[:5]  # For example, take the first 5 samples
+
+# Run inference and collect results
+results = []
+
+for sample in samples_to_infer:
+    assistant_reply = generate_prediction(sample)
+    predicted_label = extract_classification(assistant_reply)
+
+    # Collect results
+    result = {
+        'unique_id': sample['unique_id'],
+        'assistant_reply': assistant_reply,
+        'predicted_label': predicted_label,
+        # Add any other fields you need from the sample or DataFrame
+    }
+    results.append(result)
+
+    # Display the results
+    print(f"Sample ID: {sample['unique_id']}")
+    print("Assistant's Reply:")
+    print(assistant_reply)
+    print(f"Predicted Label: {predicted_label}")
+    print("-" * 50)
+
+results_df = pd.DataFrame(results)
+# Save to CSV if desired
+results_df.to_csv('inference_results.csv', index=False)
+``
+### Example Output
+
+```
+Assistant's Reply:
+Classification: Likely
+```
+
+## Training Data
+
+The model was fine-tuned on a custom dataset of news articles and images labeled for potential disinformation based on the presence of specific rhetorical techniques. The dataset includes balanced samples of articles classified as 'Likely' or 'Unlikely' to be disinformation.
+
+## Training Procedure
+
+- **Fine-Tuning**: The model was fine-tuned using the PEFT library with LoRA adapters to efficiently adapt the base model to the disinformation detection task.
+- **Quantization**: 4-bit quantization was applied using `bitsandbytes` to optimize inference performance.
+- **Hyperparameters**: Standard hyperparameters were used, with adjustments made for optimal performance on the custom dataset.
+
+## Evaluation
+
+The model was evaluated on a balanced test set with the following metrics:
+
+- **Accuracy**: *88%*
+- **Precision**: *78%*
+- **Recall**: *75%*
+
+*Note: Replace the above metrics with actual results from your evaluations.*
+
+## Limitations and Biases
+
+- **Biases in Data**: The model's performance and judgments are influenced by the biases present in the training data.
+- **False Positives/Negatives**: The model may incorrectly classify some articles due to subtle rhetorical techniques or lack thereof.
+- **Ethical Considerations**: Users should be cautious when interpreting the model's outputs and consider them as part of a broader analysis.
+
+## Ethical Considerations
+
+- **User Responsibility**: The model should not be used as the sole basis for making critical decisions.
+- **Data Privacy**: Ensure compliance with data privacy laws when using the model with user data.
+- **Misuse Prevention**: The model should not be used to unjustly label or discriminate against individuals or groups.
+
+
+
+## License
+
+This model is licensed under the Apache 2.0 License.
+
+---
+
+**Disclaimer**: The model is provided "as is" without any warranty. The developers are not responsible for any consequences arising from the use of this model.
+
+```