Computer Vision

Qwen2.5-VL Multimodal Face & Audio AI

A high-throughput multimodal intelligence system fine-tuning Qwen2.5-VL-32B with QLoRA on NVIDIA L40s GPUs for Indian celebrity & cricketer face recognition, integrated with Buffalo-L face embeddings, Whisper-large-v3 audio transcription, and a Milvus-powered RAG pipeline.

Qwen2.5-VL-32BQLoRABuffalo-LWhisper-large-v3PyTorch/CUDAMilvusFastAPI

View source

98.6%

Celebrity Accuracy

NVIDIA L40s

GPU Platform

Whisper-large-v3

Audio Transcriber

< 15 ms

Vector DB Latency

The problem

Off-the-shelf vision models failed to accurately identify Indian celebrities and cricketers under varying lighting, broadcast angles, and fast motion. Additionally, video search lacked audio-visual alignment, making it impossible to query video content using both facial identity and spoken transcripts.

Key features

Fine-Tuned Qwen2.5-VL-32B: Parameter-efficient QLoRA fine-tuning on NVIDIA L40s GPUs for Indian celebrity & cricketer visual recognition
Buffalo-L Face Recognition Pipeline: Deep feature extraction with InsightFace Buffalo-L model for high-precision facial matching
Whisper-large-v3 Audio Transcription: Automatic speech-to-text pipeline synchronizing spoken audio with video timestamps
Multimodal Video Understanding: Fused audio transcripts and visual frames into Qwen2.5-VL for contextual video QA
Interactive Audio Chatbot: Voice-activated speech-to-text chatbot performing vector search & DB queries
Vector Storage & RAG: High-dimensional embedding search powered by Milvus vector DB and LangChain orchestration
CUDA & PyTorch Acceleration: Optimized GPU memory layout for low-latency batch inference on L40s hardware
FastAPI Service Endpoint: RESTful microservice API serving real-time multimodal search & conversational queries

Architecture

1Video Input → Frame Sampler & Audio Stream Separator
2Buffalo-L Detector → Aligned Face Embeddings → Vector Matching
3Whisper-large-v3 → Speech-to-Text Transcription → Timestamp Alignment
4QLoRA Qwen2.5-VL-32B → Multimodal Fusion (Visual Frames + Transcripts)
5Milvus Vector DB → RAG Context Retrieval & Speech Chatbot Processing
6FastAPI Microservice → Real-Time Web App & Audio Chat Response

Important functions

QLoRA Fine-Tuning Setup for Qwen2.5-VL-32Bpython

import torch
from transformers import AutoModelForVision2Seq, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# 4-bit Quantization Config for NVIDIA L40s GPU Execution
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForVision2Seq.from_pretrained(
    "Qwen/Qwen2.5-VL-32B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_config)

Buffalo-L Embedding Match & Whisper-large-v3 Pipelinepython

import whisper
from insightface.app import FaceAnalysis
from pymilvus import Collection

# Initialize InsightFace Buffalo-L model & Whisper large-v3 on CUDA
face_app = FaceAnalysis(name="buffalo_l", providers=["CUDAExecutionProvider"])
face_app.prepare(ctx_id=0, det_size=(640, 640))
whisper_model = whisper.load_model("large-v3", device="cuda")

def process_multimodal_clip(video_frame: np.ndarray, audio_path: str):
    # 1. Face Recognition via Buffalo-L embeddings
    faces = face_app.get(video_frame)
    embeddings = [f.embedding / np.linalg.norm(f.embedding) for f in faces]
    search_res = milvus_collection.search(embeddings, "embedding_vec", param={"metric_type": "COSINE"}, limit=1)

    # 2. Audio Transcription via Whisper-large-v3
    transcript = whisper_model.transcribe(audio_path, language="en")["text"]

    return {"faces": search_res, "transcript": transcript}

Multimodal RAG & Audio Chatbot Queryingpython

from langchain_community.vectorstores import Milvus
from langchain.chains import ConversationalRetrievalChain

vector_db = Milvus(embedding_function=embeddings, collection_name="multimodal_knowledge")

def audio_chatbot_query(audio_bytes: bytes) -> str:
    # Speech-to-Text conversion via Whisper
    user_query_text = whisper_model.transcribe(audio_bytes)["text"]

    # Contextual Retrieval over Milvus Vector Database
    retrieved_docs = vector_db.similarity_search(user_query_text, k=4)
    context = "\n".join([doc.page_content for doc in retrieved_docs])

    # Context-aware Multimodal Generation via Fine-tuned Qwen2.5-VL-32B
    prompt = f"Audio Transcript Context:\n{context}\n\nUser Query: {user_query_text}"
    return qwen_model.generate(prompt)

Simulation & screenshots

Qwen2.5-VL Multimodal Face & Audio AI cover — Qwen2.5-VL Multimodal Face & Audio AI — concept visual

Multimodal recognition pipeline with Qwen2.5-VL visual reasoning and Whisper audio RAG

Back to all projects