Available · open to roles@ Publicis SapientServing LLMs on H100/H200

Shivang Singh.

Machine Learning Engineer · LLM Inference & Serving

I make open-source LLMs fast and cheap to run in production.

Read the case study

Resume (PDF)

Bengaluru, India

LLM Inference·vLLM · TensorRT-LLM·Speculative decoding·K8s · H100

Inference results

H100 / H200

Decode tok/s · 27B FP8 LLM on a single H100: 79 → 148
Median time-to-first-token at 32 concurrent users: −86%
Schedulable GPUs per H100 node with MIG (2.5×): 8 → 20
Deploy time for a 35B model (−70%): 550s → 163s

Read the case study

Scroll

01 · Featured case study

Speculative decoding on a 27B FP8 LLM (single H100)

A 27B FP8 LLM served on a single H100 needed faster decode and a lower time-to-first-token under concurrent load — without giving up tail latency.

79 → 148

decode tok/s — nearly doubled

-86%

median time-to-first-token at 32 concurrent users

Setup

GPU: 1× NVIDIA H100 (80 GB)
Model: 27B-parameter LLM, FP8 weights
Engine: TODOserving engine + version (vLLM?)
Speculative method: TODOdraft model / method and speculation depth
Concurrency: 32 concurrent users (TTFT measurement)
Input / output lengths: TODOprompt and output token lengths

What I tried

Enabled speculative decoding on the existing FP8 deployment, then tested further tuning options one at a time against the same benchmark.

Results · baseline vs tuned

Decode throughput, 27B FP8 LLM on a single H100higher is better · tok/s

Baseline

79 tok/s

Speculative decoding

148 tok/s

Baseline vs speculative decoding
Metric	Baseline	Tuned
Decode speed	79 tok/s	148 tok/s
Median TTFT @ 32 users	TODOms	-86% TODOms
p99 latency	TODOms	TODOms

TODOchart: TTFT vs concurrency, baseline vs speculative decoding

What I rejected and why

FP8 KV cache
Regressed in measurement — dropped. TODOsize of regression
Dynamic speculation depth
Regressed in measurement — dropped. TODOsize of regression
Chunked-prefill tuning
Regressed in measurement — dropped. TODOsize of regression

TensorRT-LLM vs vLLM

Benchmarked TensorRT-LLM against vLLM on identical weights. TensorRT-LLM showed 32% worse p99 latency with no throughput gain, so vLLM stayed.

TensorRT-LLM vs vLLM on identical weights
Metric	TensorRT-LLM	vLLM
p99 latency	32% worse	baseline
Throughput	no gain	baseline
Decision	rejected	kept

TODOmodel + workload used for this comparison

02 · Platform work

Serving open-source LLMs on Kubernetes.

Core engineer on a self-hosted platform to fine-tune and serve open-source LLMs on Kubernetes, with autoscaling, scale-to-zero and GPU monitoring across AWS EKS and on-prem H100/H200 clusters.

Publicis Sapient · Jul 2026 — Present

Serving 11 models

224M – 2.8T

parameters

Deployed and benchmarked 11 models from 224M to 2.8T parameters, including a 2.8T MoE on a single 8-GPU node.

Weight caching & deploy time

550s → 163s

35B deploy (−70%)

Co-built cached model-weight delivery and image reuse, cutting deploy time by 70% for a 35B model and bringing cached cold starts under 0.5 s.

MIG GPU slicing

8 → 20

schedulable GPUs per H100 node

Co-built GPU slicing (MIG) in the scheduler so small models no longer hold a whole 80 GB GPU — 2.5× more schedulable GPUs per H100 node.

Multi-LoRA serving

1 slice

instead of 4 GPUs

Served a base model + 4 LoRA adapters on 1 GPU slice instead of 4 GPUs.

03 · Experience

Earlier work.

Machine Learning Engineer at Publicis Sapient since Jun 2025. Before the inference platform: production GenAI, and computer vision at Lincode Vision Labs.

Publicis Sapient · Machine Learning Engineer

Multimodal GenAI Platform for Marketing Asset Analysis

Jun 2025 — Jun 2026Bengaluru, India

Architected a production multimodal GenAI platform for a Fortune 500 pharma client, cutting marketing asset analysis from hours to ~2 min per asset (95% faster) across 10,000+ assets.
Built multi-stage LLM pipelines combining Gemini with YOLO and OCR to extract 50+ structured signals per asset, with rate-limit handling and LLM-as-judge evaluation to track output quality.
Shipped the backend as FastAPI microservices on Kubernetes with autoscaling, handling 1,000+ concurrent requests under burst traffic.

GeminiYOLOOCRLLM-as-judgeFastAPIKubernetes

Lincode Vision Labs · Data Science Intern

Defect detection

Oct 2024 — May 2025Bengaluru, India

Replaced YOLOv8 with RF-DETR for defect detection: 1.8× faster inference and +7% mAP50.
Built data pipelines over 30,000+ images, improving model accuracy by 10%.

RF-DETRYOLOv8PyTorchOpenCV

04 · Side projects

Things I've shipped.

/ project 01

Dossier

Autonomous agentic job search system

$0.06

per application

A 7-agent system that finds, scores and researches jobs and writes tailored resumes, cutting prep per application from hours to under 2 minutes at $0.06. Cheap rule-based filters before LLM scoring remove 65% of LLM calls across 550+ jobs per run.

PythonOpenAIClaudeTavilySQLiteLaTeX

View source

Case study

/ project 02

FedFV-CV

Federated learning for biometric auth (B.Tech thesis)

1.21%

error rate

Designed a new federated aggregation method for finger-vein authentication across 5 clients (122K images), reaching a 1.21% error rate and beating FedAvg.

PyTorchMobileNetV2Federated learning

View source

Case study

05 · Skills

The stack I reach for.

LLM Inference

8 items

vLLMTensorRT-LLMSGLangllama.cppspeculative decodingFP8multi-LoRANVIDIA MIG

Infra & MLOps

10 items

KubernetesHelmAWS EKSKEDARook-CephBuildKitDockerPrometheusGrafanaDCGM

Training & ML

7 items

PyTorchtorchrun/NCCLLoRA fine-tuningscikit-learnYOLORF-DETROpenCV

GenAI & Backend

11 items

PythonSQLFastAPIPostgresRedisCeleryLangGraphRAGDeepEvalAWSGCP

06 · Writing

Notes and write-ups.

All posts

May 29, 2026 5 min read

Building Dossier in Public: The M4 Milestone

Four milestones in, Dossier is no longer a side project — it's an agentic job-search system with real users. Here's what shipped, what broke, and why I'm building it on LinkedIn.

#dossier#build-in-public

May 22, 2026 9 min read

Dossier: An 8-Agent Job-Search Pipeline That Survives Production at ~$0.04 per Run

Why I stopped building a job board and started building a quality-first agentic pipeline. The architecture, the cost discipline, and the SaaS layer I wrapped around it.

#agents#llm

May 15, 2026 4 min read

What I Learned Shipping a Multimodal GenAI Platform to Production

A multimodal GenAI platform that processed 10,000+ marketing assets for a Fortune 500 pharma client. Here's what actually broke, what scaled, and what I'd build differently next time.

#genai#production

Coming next · LLM inference

TODOfirst inference post — title, date and link

AI Concierge

Live · grounded on resume

Don't read the resume. Talk to it.

Skip the scrolling. Ask my AI anything about my LLM inference work, benchmarks, side projects, hiring fit, or technical decisions. Streams answers grounded only on my real experience.

Free · instant · no signup

Try one of these