Hannes Hapke

Vibe coding is killing open source, but not how you think

2026-05-14T00:00:00.000Z

Why we need a renaissance of open source

A few years ago, I was a contributor to TensorFlow Extended, and once a week, the contributors got on a call. We'd argue about API design, complain about whatever was broken that week, and talk about where the industry was heading. Some of us worked at Google. Some of us worked at companies that competed with Google. None of that mattered for the forty-five minutes we were on the call. Over time, we built something that's harder to describe than it is to feel: a network of people across companies who actually trusted each other. We met up at conferences. We shared what we were learning. We pushed each other to be better.

That kind of weekly call still happens in some projects. But the world that produced it is quietly changing, and I don't think most engineers have noticed yet.

Today's internet runs on open source. Linux on the servers, nginx and Apache in front of them, Postgres underneath. Apache Kafka, originally built at LinkedIn, moves data around for basically every SaaS company you've used. A long list of open source projects became the foundation for huge commercial ones: Apache Beam shaped Google Cloud Dataflow, Apache Flink powers parts of AWS, and the entire AI stack — PyTorch, TensorFlow, Keras, vLLM — is open source. So the stakes here are not abstract.

With the rise of vibe coding, it is now faster to spin up a project from scratch for your own use case than to submit a patch to an existing one and let everyone benefit. You can see this in GitHub activity [1] [2]: where contributions used to be dominated by changes to existing code, the ratio has flipped toward greenfield projects, which is exactly the kind of work where Claude and Codex shine [3].

GitHub's Kyle Daigle on the surge in platform activity. Source: github.blog

Record acceleration in pull requests merged, commits, and new repos per month, 2023–2026. Source: bsky.app

At the same time, the projects that do still get patches are flooded with PRs from coding agents. Before vibe coding, over 80% of contributions came from one-time committers [4]. Someone hit a bug in their favorite library, fixed it, watched the merge, and moved on. That was fine, because the remaining 20% had a long-term interest in the project, and that was usually enough to keep things going (though it still left a heavy load on core maintainers).

Now the ratio is worse. A small fraction of contributions come from people with any long-term interest in the project. Some maintainers have responded by banning agent contributions outright — the Zig project, for example [5]. The reason isn't that the code is always bad. In some cases it's actually well structured and even improves the docs. The reason is that something quieter goes missing: mentorship. Junior developers learned the craft by contributing to established projects. Maintainers, often without realizing it, were training their own successors. It was a slow, durable win-win, and it's the part that doesn't show up in a commit graph.

So here's where we are: quick vibe coding is replacing long-term contribution, and a whole community (a whole mindset, really) is fading with it.

It isn't too late to fix this. But to see why it's worth fixing, it helps to remember what open source actually gives us.

The web and the AI stack both benefited from open source in ways that are easy to take for granted. When a new attention mechanism drops, patches land in PyTorch and vLLM within hours. That isn't just technical excellence. It's trust. A broad community of people, none of whom answer to the same boss, decided the change was worth shipping.

And it isn't only the trust that updates will arrive quickly. It's the trust that you can read every line if you need to. (Less true for open-weight models, but the principle still holds for the surrounding infrastructure.) That trust is quietly carrying a lot of the economy. It's why you can hand your credit card to a checkout page and assume the nginx config in front of it isn't doing something hostile. It's why you can run an ML evaluation and assume the framework isn't quietly cooking the numbers. Imagine a Volkswagen-style emissions scandal, but for model benchmarks. The reason that hasn't happened isn't a clever piece of regulation. It's that thousands of people have read the code.

Open source is trust.

And trust matters more, not less, in a world where tech is increasingly controlled by a handful of companies.

None of this is anti-AI. I use Claude every day, including for my open source work. It's great for the boring parts. The problem isn't agents writing code. The problem is what happens when only agents write code for the projects we all depend on, and the humans who used to mentor each other through those codebases drift away.

Open source has always been carried by a small core of maintainers and a long tail of people who showed up, learned something, and stuck around. Vibe coding is great at producing the first commit. It's pretty bad at producing the fifth-year contributor who eventually takes over the project.

So: fund this work. With headcount. With paid time for your engineers to maintain the libraries your company runs on. A few companies are starting to do this seriously. (Disclosure: I work at one of them, Dataiku, which set up an open source lab last year. I'd rather see ten more.) The specific projects matter less than the pattern: treat the commons like infrastructure, because that's what it is. For a more in-depth economic argument, this paper [6] is worth your time.

Open source is trust. Trust gets built by humans showing up for each other over years. That's the part no agent can vibe-code for us.

References

GitHub Blog. An Update on GitHub Availability. github.blog
cameron.stream. Post on Bluesky. bsky.app
Hacker News. Discussion thread 46142841. news.ycombinator.com
Eghbal, Nadia. Working in Public. Stripe Press, 2020.
Zig Project. Code of Conduct. ziglang.org
Paper on arXiv. ID 2601.15494. arxiv.org

Kiji Privacy Proxy™ - Protecting Your Data in the Age of Generative AI

2026-04-27T00:00:00.000Z

Originally published on the Dataiku Blog.

Every time you type a prompt into ChatGPT, Claude, or any other LLM-powered service, you're sending data to an external server. For casual questions, that's fine. But in enterprise settings, those prompts often contain customer names, email addresses, social security numbers, medical records, financial details, and internal business data that should never leave your environment.

This isn't a hypothetical risk. Regulations like GDPR, HIPAA, and CCPA impose real penalties on organizations that fail to protect personal data, and sending PII to a third-party API without proper safeguards can constitute a violation. A 2026 Dataiku/Harris Poll study of 600 CIOs found that 85% have seen AI projects delayed or blocked entirely due to gaps in traceability or explainability, and privacy concerns are a major part of that picture.

The challenge is clear: Enterprises want the productivity gains of generative AI, but they can't afford to expose sensitive data in the process. Until now, the main options have been to either avoid using external AI services altogether (losing the benefits), build expensive custom infrastructure, or simply accept the risk and hope for the best.

None of those options is good enough.

Why Kiji Privacy Proxy™

Operating as a transparent gateway between your local applications and external AI APIs, Kiji Privacy Proxy™ ensures you don't have to compromise your workflow or abandon powerful AI tools. By sitting directly within your network, Kiji automatically identifies and redacts personally identifiable information (PII) before any data is transmitted, allowing you to leverage generative AI without having to trust third-party servers with your sensitive information.

Here's how it works: Your app sends a request to the Kiji Privacy Proxy, and it forwards it to services like OpenAI or Anthropic. Alternatively, Kiji can intercept requests as well, run them through an ML-powered PII detection model, and replace any sensitive data, emails, phone numbers, credit card numbers, SSNs, IP addresses, and 16+ other PII types with realistic dummy values. The masked request goes out to the API. When the response comes back, Kiji restores the original values so your application works exactly as expected.

The result: The AI model never sees your real data, but your application behaves as though nothing changed.

What makes Kiji particularly practical is how little friction it introduces. On macOS, it runs as a native desktop app with automatic proxy configuration. We also provide a Chrome extension that routes web requests through Kiji without any environment variables or code changes. On Linux, it runs as a standalone server. In both cases, latency stays under 100 milliseconds for most requests, and all PII detection happens locally with no external API calls.

Kiji is open source under the Apache 2.0 license, and both the trained model and its training dataset are published on HuggingFace (DataikuNLP/kiji-pii-model-onnx and DataikuNLP/kiji-pii-training-data), so you can inspect, reproduce, and extend everything.

The Kiji Privacy Proxy is powered by a base model (developed by Dataiku's 575 Lab) that attained a 94% F1 score on the industry benchmark dataset. This result is highly competitive when compared to similar models in the field.

A Collaboration Between Forward-Thinking ML Companies

Kiji Privacy Proxy doesn't exist in a vacuum. It's part of a broader vision where specialized companies across the ML ecosystem each contribute what they do best, and the result is greater than the sum of its parts. Built by Dataiku's 575 Lab, Kiji draws on and connects with the work of several outstanding partners.

Dataiku — The company that created Kiji, brings over a decade of enterprise AI experience, and recently launched the 575 Lab as its Open Source Office, is dedicated to building deployable tools for AI transparency, privacy, and governance. Kiji is one of the Lab's first releases, alongside agent explainability tools. As a member of the Linux Foundation and the Agentic AI Foundation, Dataiku, the Platform for AI Success, is committed to building these capabilities in the open.

Outerbounds — The company behind Metaflow, the open-source ML infrastructure stack originally built at Netflix, provides state-of-the-art infrastructure that makes complex ML workflows manageable. For teams that want to integrate Kiji's PII detection into larger ML pipelines, train custom models, orchestrate data flows, and manage deployment at scale, Outerbounds' infrastructure-as-code approach is a natural complement.

HumanSignal — The creators of Label Studio, the world's most popular open-source data labeling tool used by over 350,000 researchers, plays a critical role in the data quality side of the equation. Kiji's ML model is only as good as its training data, and for organizations that need to customize PII detection for their specific domain (think medical record formats, industry-specific identifiers, or non-English PII patterns), Label Studio provides the labeling infrastructure to build and refine those custom datasets.

Doubleword — The inference provider for high-volume workloads, founded by researchers from Oxford University who pioneered techniques in model optimization, completes the picture on the deployment side. Doubleword's inference platform offers open-source model inference at a fraction of the cost of other providers, making it well-suited for high-volume workloads such as data and document processing, as well as async agents. In this case, Doubleword models were used to generate large volumes of synthetic data at a cost of only $50 — just five percent of what comparable models from closed-source providers would have cost.

Make Your Domain-Specific Kiji Privacy Proxy

One of the most powerful aspects of Kiji is its design for customization. The default model handles common PII types well, but every industry has unique data patterns that a generic model won't catch, such as pharmaceutical compound identifiers, internal project codes, proprietary customer reference numbers, and jurisdiction-specific ID formats.

Kiji's architecture makes it straightforward to build your own domain-specific privacy proxy. The training data and model are fully open on HuggingFace. With Doubleword's batch inference platform, you can create your own large synthetic data. You can use Label Studio (by HumanSignal) to annotate the domain-specific, synthetic PII examples. You can orchestrate the training pipeline with Metaflow (by Outerbounds) on whatever compute you need.

This is what collaboration across the ML ecosystem looks like in practice: not a single monolithic product, but a set of interoperable, open tools built by companies that deeply understand their piece of the puzzle. Together, they give enterprises the building blocks to protect their data without sacrificing the transformative potential of generative AI.

Star the Repo on GitHub and Try It Out

Read the full post on the Dataiku blog.

Speculative Decoding with vLLM using Gemma

2025-02-28T00:00:00.000Z

When deploying large language models in production environments, latency optimization is crucial. This is particularly important for real-time applications like chatbots and conversational interfaces. While complex tasks often require larger LLMs (70 billion+ parameters), users still expect response times similar to smaller models. This challenge has led the machine learning community to continuously explore new ways to improve LLM latency.

One of the most promising techniques is speculative decoding, which is a technique that improves the performance of a language model by predicting multiple tokens at a time with a smaller model and use a larger model to validate the predictions.

Problem

When serving large language models in production, you need to lower the latency. In fact, latency optimization is the biggest challenge that developers will face in production LLM systems. This challenge becomes particularly acute in real-time scenarios, where users expect near-instantaneous interactions with chatbots, code completion tools, and other AI-powered interfaces.

At the heart of this challenge lies a fundamental characteristic of auto-regressive models: their sequential nature of text generation. Unlike many computational processes that can benefit from parallel processing, these models face an architectural constraint that proves to be their primary performance bottleneck. To generate any given token K, the model must first process and consider all preceding tokens, from 1 to K-1, in sequential order. This dependency chain, which provides context for the text generation, creates a processing pipeline that cannot be easily parallelized.

Figure 1: Sequential token generation

Consider the process illustrated in Figure 1, where each token's generation depends on the complete history of previous tokens. This sequential dependency isn't merely a technical limitation—it's a fundamental aspect of how these models understand and generate human-like text.

The situation becomes even more challenging when we scale up to larger models, particularly those exceeding 3+ billion parameters. These massive models, while offering superior capabilities in terms of reasoning, understanding, and generation, exact a significant performance penalty. Each token prediction requires more computational resources, as the model must process its vast parameter space for every single token generation step. The result is a compounding effect: not only must we handle the sequential nature of token generation, but each step in that sequence now takes longer due to the model's size.

Yet, despite these performance challenges, larger models remain indispensable for many applications. They excel at complex tasks that smaller models struggle with, such as multi-step reasoning, code generation, and nuanced understanding of context. They also produce higher-quality text with fewer artifacts and better coherence. This creates a tension between the need for sophisticated model capabilities and the practical requirements of production deployment.

In this solution, we will use speculative decoding to improve the performance of LLM deployments. It will allow us to improve the model latency without changing the model architecture, training data, or the trained model itself.

Solution

Speculative decoding is an optimization technique that leverages two distinct language models to improve generation speed while maintaining output quality. The approach uses a teacher-student architecture where two complementary models work together: a large, sophisticated language model (LLM) that produces highly accurate outputs but is computationally expensive and relatively slow, serving as the teacher model and ground truth for token generation; and a smaller, more efficient language model that operates faster but may be less accurate, acting as the student model. The student model is specifically trained to emulate the behavior of the teacher model – for example, a 3 billion parameter model might be trained to imitate a 405 billion parameter model.

The Inference Process

During text generation, the process follows a specific workflow. The student model begins by rapidly proposing a sequence of tokens based on its training to imitate the teacher model's behavior. Following this initial prediction, the teacher model evaluates the student's proposed tokens in parallel, verifying whether it would have generated the same sequence. The outcome of this validation determines the next steps: if the teacher model agrees with the student's predictions, the sequence is accepted and immediately output; however, if the teacher model disagrees, it falls back to its standard token-by-token generation process to ensure accuracy.

Key Insight

The fundamental principle behind speculative decoding is that not all tokens require the computational power of a large model for accurate generation. Token difficulty varies significantly – simple, predictable tokens like common words or obvious completions can be reliably generated by the smaller student model, while complex or context-dependent tokens benefit from the teacher model's advanced capabilities. This selective use of computational resources allows for significant speed improvements while maintaining the quality standards of the larger model. The approach is particularly effective because it balances the trade-off between speed and accuracy by dynamically choosing the appropriate model based on the complexity of the current generation task.

Here is an example of how speculative decoding would play out for a sequence of tokens:

Step 1:
Student: "The [talented] [chef]"
Teacher: ✓ Accepts (common phrase)

Step 2:
Student: "cooked [a] [delicious]"
Teacher: ✓ Accepts (common food context)

Step 3:
Student: "[soup]"
Teacher: ✗ Rejects
Teacher generates: "bouillabaisse" (rare, specific word)

Step 4:
Student: "[for] [dinner]"
Teacher: ✓ Accepts (common ending)

How does speculative decoding improve the inference speed?

We gain the inference speed increases by three aspects. First of all, we generate the proposal tokens through the smaller LLM. In addition, we can generate multiple tokens at once. We can request multiple tokens, since we have a second model to validate the predictions. We can afford it because the initial tokens predictions are fast and cheap. Secondly, the validation of the prediction is also fast, and we only need to correct the predictions for tokens where the smaller model made a mistake.

How can you use speculative decoding with your LLM? Most LLM deployment frameworks provide support of speculative decoding, in one form or another. For our core example, we are demonstrating speculative decoding with vLLM. vLLM is a frequently used framework for serving LLM models like Llama 3.2 3B. In our example, we use a smaller model to predict the next tokens and a larger model to validate the prediction and, if needed, correct the prediction. We use Meta's opt-125m model to predict the next tokens and the larger opt-2.7b model to validate the prediction and, if needed, correct the prediction. The sampling of the tokens, checking them with the larger model, and correcting them is done under the hood by the LLM serving framework, in our case vLLM.


from vllm import LLM, SamplingParams

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="google/gemma-2-9b-it",
    tensor_parallel_size=1,
    speculative_model="google/gemma-2-2b-it",
    num_speculative_tokens=5,
)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

When we compare the latency of speculative decoding with the latency of the same model without speculative decoding, we can see that speculative decoding is faster by roughly 35% as we can see in Figure 2.

Figure 2: Comparison of latency between standard and speculative decoding approaches. Speculative decoding shows a 35% improvement in processing time.

Trade-offs and Alternatives

There are significant trade-offs when using speculative decoding. The faster inferences don't come without downsides. In this section, we will discuss the trade-offs when using speculative decoding.

Sequence Length

The relationship between sequence length and performance gains in speculative decoding presents an interesting trade-off. Generally, longer sequences tend to yield higher speedups, as demonstrated in our example where we set num_speculative_tokens=5. This parameter allows the smaller model to predict multiple tokens ahead, potentially improving throughput. However, this advantage comes with diminishing returns: as sequence length increases, so does the likelihood of prediction errors. When these errors occur, the larger model steps in to correct the predictions, which can significantly slow down the overall inference process. Finding the optimal sequence length therefore requires careful balancing between maximizing the benefits of speculation while minimizing the computational overhead of error correction.

Larger Memory Footprint

The speculative decoding requires a larger memory footprint. The larger model needs to loaded into memory, together with the smaller model. This will require larger instances and GPUs, which translates to higher costs. Also, fine-tuning models is difficult for when you want to use speculative decoding with your models. The smaller model needs to be fine-tuned on the same dataset as the larger model. This is not always possible, since fine-tuning the larger model is more expensive than fine-tuning the smaller model (however, you might the performance boost from the smaller fine-tuned model already).

Model Pairing

A larger model needs to be paired with a smaller model using the same tokenization. This is no problem for larger models to be paired with smaller models. However, this is not the case for smaller models.

The following table shows the possible combinations of base models and smaller models for speculative decoding.

Base Model	Smaller Model for suggesting tokens
gemma-2-27b-it	gemma-2-9b-it
gemma-2-27b-it	gemma-2-2b-it
gemma-2-9b-it	gemma-2-9b-it
gemma-2-2b-it	?

Problem specific Performance

The performance of speculative decoding also depends on the distribution of tokens. For example, if we want to generate English text for a chatbot, speculative decoding will be more effective than if we want to generate random hashes or bank transaction descriptions. in those cases, speculative decoding will actually be slower because the larger model needs to correct the predictions too often.

Alternatives

Several alternative approaches can help reduce LLM deployment latency, each with its own strengths and trade-offs.

Use a smaller model

The simplest approach is to use a smaller model altogether. This solution offers both reduced memory footprint and faster inference times compared to speculative decoding. The deployment becomes significantly simpler, requiring only one smaller model and less powerful GPUs. However, this approach comes with an obvious drawback: the quality of generated text suffers noticeably. While faster, smaller models often lack the sophisticated reasoning and nuanced understanding that larger models provide. You would typically only consider this option if your use case doesn't require the advanced capabilities of larger models.

Use a quantized model

Model quantization represents a sophisticated optimization technique that reduces numerical precision while preserving model functionality. By converting the model's parameters from their original 32-bit floating-point representation to 8-bit or even 4-bit precision post-training, quantization achieves significant improvements in both memory efficiency and computational performance. This reduction in numerical complexity translates directly into decreased memory footprint, lower computational overhead, and consequently, faster inference times.

While quantization does introduce a modest degradation in model quality compared to the original implementation, it offers compelling advantages as an alternative to speculative decoding. The deployment architecture remains streamlined with only a single model to maintain, and the reduced computational demands enable the use of more cost-effective GPU hardware. This balance of performance optimization and operational simplicity makes quantization an attractive option for many production environments.

Parallelization

Parallelization presents another powerful strategy for improving LLM performance of larger LLMs. Instead of processing multiple requests sequentially, you can process multiple requests simultaneously. This way, you can significantly decrease the effective latency across multiple requests. This approach particularly shines in high-traffic scenarios where individual requests use only a fraction of the model's context length. However, parallelization faces clear limitations: it remains constrained by both the model's maximum context length and the available GPU memory. Despite these constraints, parallelization often provides substantial performance benefits for many production deployments and it should be your first consideration when optimizing deployment latency.

Continuous batching

Continuous batching takes the parallelization concept even further. Instead of processing fixed batches, this technique dynamically pulls new requests from a queue whenever space becomes available in the current batch. This approach proves especially effective when handling a high volume of requests with varying context lengths. By maintaining consistent GPU utilization, continuous batching can achieve even lower latency than standard parallelization. However, it shares the same fundamental limitations regarding context length and GPU memory, and requires specialized deployment infrastructure to support the dynamic batching mechanism.

Caching

Caching offers a different approach to latency optimization, particularly valuable for applications with repetitive requests. By storing and reusing previous inference results for identical prompts, caching can deliver nearly instantaneous responses for repeated queries. While novel requests still face slow inference time of a large model, frequently accessed responses become lightning-fast. This makes caching particularly effective for applications like customer service chatbots or code completion tools, where certain queries appear frequently. The effectiveness of caching directly correlates with the repetitiveness of your workload – the more repeated queries you handle, the greater the performance benefit.

Demo

We have created a demo of speculative decoding with vLLM using Gemma. You can find the code in Colab.

Conclusion

Speculative decoding is a promising technique to improve the performance of LLM deployments. in our demo example, we were able to improve the latency by 35%. However, it comes with a larger memory footprint and more deployment complexity.

Acknowledgements

We would like to thank Google's Developer Community (GDE) for providing the funding to create this demo. #VertexAISprint

Deploying Google's Gemma on Vertex AI

2025-02-17T00:00:00.000Z

In the rapidly evolving landscape of artificial intelligence, the ability to deploy and manage your own language models has become increasingly important. While hosted solutions like Google's Gemini offer convenience, there are compelling reasons to host your own models. Today, we'll explore how to deploy Google's Gemma model on Vertex AI, providing you with complete control over your AI infrastructure.

Introduction

Google's recent release of Gemma marks a significant milestone in the democratization of AI. As an open-source alternative to their hosted Gemini models, Gemma provides organizations with the flexibility to run these powerful language models on their own infrastructure. In this comprehensive guide, we'll walk through the process of deploying Gemma on Google Cloud's Vertex AI platform, exploring every aspect from initial setup to production deployment.

Why Host Your Own Model?

Before diving into the technical details, let's understand why you might choose to host your own model instead of using hosted solutions:

Data Privacy and Compliance

When dealing with sensitive information such as medical records, legal documents, or proprietary business data, maintaining complete control over your data pipeline becomes crucial. By hosting your own model, you ensure that sensitive data never leaves your controlled environment, making it easier to comply with regulations like HIPAA, GDPR, or industry-specific requirements.

Responsible AI Implementation

Organizations increasingly need to demonstrate transparency and control over their AI systems. Running your own model instance allows you to:

Monitor and audit all interactions
Implement custom fairness metrics
Control model behavior and outputs
Maintain clear data lineage
Avoid sharing potentially sensitive data with third-party providers

Performance Optimization

Self-hosting enables you to:

Fine-tune latency for specific use cases
Optimize hardware allocation based on your workload
Implement custom caching strategies
Control model quantization and optimization parameters

Technical Understanding

For organizations invested in AI technology, understanding the deployment process provides valuable insights into:

Model serving architecture
Resource management
Scaling considerations
Performance optimization techniques

Prerequisites

Before beginning the deployment process, ensure you have:

A Google Cloud Account with billing enabled
Vertex AI API activated in your project
A Hugging Face account with access to Gemma models
Basic familiarity with Python and cloud computing concepts

Understanding the Deployment Architecture

Our deployment strategy uses vLLM (Versatile Large Language Model) serving framework, which offers several advantages:

Why vLLM?

vLLM has emerged as a leading solution for serving large language models due to its:

Continuous Batching: Efficiently processes multiple requests by dynamically batching them, maximizing GPU utilization.
PagedAttention: Implements an innovative attention mechanism that significantly reduces memory usage and increases throughput.
Kernel Fusion: Optimizes computation by combining multiple operations into single GPU kernels.
Quantization Support: Offers various quantization options to reduce model size and increase inference speed.

The Deployment Process

Let's break down the deployment into three main steps:

Step 1: Registering the Model

The first step involves registering your Gemma model with Vertex AI's Model Registry. This process creates a versioned record of your model that can be tracked and managed.

from google.cloud import aiplatform

def register_model(
    project: str,
    location: str,
    display_name: str,
    artifact_uri: str,
    model_id: str,
    version_description: str,
    serving_container_image_uri: str,
    serving_container_environment_variables: dict,
    serving_container_command: list
) -> aiplatform.Model:
    """
    Register a new model in Vertex AI Model Registry.

    Args:
        project: Google Cloud project ID
        location: Region for deployment (e.g., 'us-central1')
        display_name: Human-readable name for the model
        artifact_uri: GCS location of model artifacts
        model_id: Unique identifier for the model
        version_description: Description of this model version
        serving_container_image_uri: Docker image URI for model serving

    Returns:
        aiplatform.Model: Registered model object
    """
    aiplatform.init(project=project, location=location)

    model = aiplatform.Model.upload(
        display_name=display_name,
        artifact_uri=artifact_uri,
        model_id=model_id,
        description="vLLM model for generating text",
        version_description=version_description,
        serving_container_image_uri=serving_container_image_uri,
        serving_container_health_route="/health",
        serving_container_environment_variables=serving_container_environment_variables,
        serving_container_predict_route="/generate",
        serving_container_ports=[8000],
        serving_container_command=serving_container_command
    )

    return model


register_model(
    project="your gcp project id",
    location="us-central1",  # or your preferred region
    display_name="gemma-vllm",
    model_id="gemma_vllm_001",
    version_description="Initial Gemma vLLM deployment",
    serving_container_image_uri="us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:latest",
    serving_container_environment_variables={
        "HUGGING_FACE_HUB_TOKEN": "hf_<your token>"
    },
    serving_container_command=["python3", "-m", "vllm.entrypoints.api_server",
                             "--model=google/gemma-2-2b-it",
                             "--tensor-parallel-size=1",
                             "--max_model_len=8126"]
)

This code does several important things:

Model Initialization: Uses aiplatform.init() to set up the connection to your Google Cloud project.
Model Registration: Creates a new model entry in the Vertex AI Model Registry with:
- A display name for human readability
- The location of model artifacts in Google Cloud Storage
- A unique model identifier
- Version information for tracking changes
- Container configuration for serving
Container Configuration: Specifies important endpoints:
- Health check route for monitoring
- Prediction route for inference
- Port configuration for network access
- Image URI
- Container commands
- Huggingface token via the environmental variables

Note about the container image

Vertex expects a very specific request - response structure. Google provide the instructions of how to build such a container and they provide a path to update the open-source vLLM implementation. Instead of patching and building our own docker image, we short cut the work by reusing the Docker image provided by Google's model garden.

Step 2: Creating an Endpoint

The next step involves creating a Vertex AI endpoint that will serve your model:

def create_endpoint(
    project: str,
    location: str,
    display_name: str
) -> aiplatform.Endpoint:
    """
    Create a new Vertex AI endpoint for model serving.

    Args:
        project: Google Cloud project ID
        location: Region for deployment
        display_name: Human-readable name for the endpoint

    Returns:
        aiplatform.Endpoint: Created endpoint object
    """
    aiplatform.init(project=project, location=location)

    endpoint = aiplatform.Endpoint.create(
        display_name=display_name,
        project=project,
        location=location,
    )

    return endpoint

This endpoint creation process:

Initializes the Environment: Sets up the project and location context.
Creates the Endpoint: Establishes a new serving endpoint with:
- A human-readable display name
- Project and location specifications
- Default configuration settings
Prepares for Deployment: Sets up the necessary infrastructure for model serving.

Step 3: Deploying the Model

The final step involves deploying your registered model to the created endpoint:

def deploy_model(
    model: str,
    endpoint: str,
    machine_type: str,
    accelerator_type: str,
    accelerator_count: int,
    min_replica_count: int = 1,
    max_replica_count: int = 1,
) -> aiplatform.Model:
    """
    Deploy a registered model to a Vertex AI endpoint.

    Args:
        model: Resource name of the model to deploy
        endpoint: Resource name of the target endpoint
        machine_type: Type of machine for deployment
        accelerator_type: Type of accelerator (GPU)
        accelerator_count: Number of accelerators
        min_replica_count: Minimum number of serving instances
        max_replica_count: Maximum number of serving instances

    Returns:
        aiplatform.Model: Deployed model object
    """
    deployed_model = aiplatform.Model.deploy(
        model=model,
        endpoint=endpoint,
        deployed_model_display_name=f"deployed_{model}",
        machine_type=machine_type,
        accelerator_type=accelerator_type,
        accelerator_count=accelerator_count,
        min_replica_count=min_replica_count,
        max_replica_count=max_replica_count,
        traffic_split={"0": 100},
        sync=True
    )

    return deployed_model

This deployment configuration includes several important parameters:

Hardware Specification:
- machine_type: The type of VM instance (e.g., 'g2-standard-8')
- accelerator_type: GPU specification (e.g., 'NVIDIA_TESLA_L4')
- accelerator_count: Number of GPUs per instance
Scaling Configuration:
- min_replica_count: Minimum number of serving instances
- max_replica_count: Maximum number of serving instances
- Enables automatic scaling based on load
Traffic Management:
- traffic_split: Controls request routing
- Enables gradual rollouts and A/B testing

Alternative Serving Frameworks

While vLLM is our recommended choice, several alternatives exist:

1. FastAPI + Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
from fastapi import FastAPI

app = FastAPI()
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")

@app.post("/predict")
async def predict(text: str):
    inputs = tokenizer(text, return_tensors="pt")
    outputs = model.generate(**inputs)
    return {"response": tokenizer.decode(outputs[0])}

Advantages:

Simple implementation
Direct integration with Hugging Face
Flexible customization

Disadvantages:

Limited optimization features
No built-in batching
Higher memory usage

2. Text Generation Inference (TGI)

TGI offers a more optimized alternative:

from text_generation import Client

client = Client("http://localhost:8080")
response = client.generate(
    "What is machine learning?",
    max_new_tokens=512,
    temperature=0.7
)

Advantages:

Optimized for production
Streaming support
Better memory management

Disadvantages:

Less flexible than vLLM
Limited quantization options

3. SGL Project

The SGL Project provides another approach:

import sglang as sgl

@sgl.function
def generate(prompt):
    return sgl.gen(prompt, max_tokens=512)

Advantages:

Simple API
Good performance
Easy integration

Disadvantages:

Newer project
Smaller community
Limited features

Limitations and Considerations

When deploying Gemma on Vertex AI, be aware of these limitations:

1. Streaming Limitations

Vertex AI currently doesn't support native streaming responses, which means:

All responses must be returned as complete messages
Real-time token generation isn't possible
Higher latency for long responses

2. Hardware Availability

Some considerations regarding hardware:

GPU availability varies by region
Certain GPU types may have limited availability
Cost implications of different hardware choices

3. Resource Management

Important resource considerations:

Memory management for large models
GPU utilization optimization
Scaling limitations

Best Practices

To ensure optimal deployment and operation:

1. Model Optimization

Use appropriate quantization methods
Implement caching strategies
Configure batch sizes based on workload

2. Monitoring

Set up comprehensive logging
Monitor GPU utilization
Track response times and error rates

3. Cost Management

Use appropriate machine types
Implement auto-scaling
Monitor resource usage

Conclusion

Deploying Gemma on Vertex AI provides organizations with powerful capabilities for running their own language models. While there are some limitations to consider, the benefits of control, customization, and privacy make it an attractive option for many use cases.

The combination of Vertex AI's infrastructure and vLLM's serving capabilities creates a robust platform for AI deployment. By following the steps and best practices outlined in this guide, you can successfully deploy and manage your own Gemma instance.

Remember to regularly monitor your deployment, optimize based on usage patterns, and stay updated with the latest developments in both Vertex AI and the serving frameworks to ensure the best possible performance and cost-effectiveness of your deployment.

Speculative Decoding with vLLM

2025-01-11T00:00:00.000Z

Problem

Figure 1: Sequential token generation

Solution

The Inference Process

Key Insight

Here is an example of how speculative decoding would play out for a sequence of tokens:

Step 1:
Student: "The [talented] [chef]"
Teacher: ✓ Accepts (common phrase)

Step 2:
Student: "cooked [a] [delicious]"
Teacher: ✓ Accepts (common food context)

Step 3:
Student: "[soup]"
Teacher: ✗ Rejects
Teacher generates: "bouillabaisse" (rare, specific word)

Step 4:
Student: "[for] [dinner]"
Teacher: ✓ Accepts (common ending)

How does speculative decoding improve the inference speed?


from vllm import LLM, SamplingParams

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="facebook/opt-2.7b",
    tensor_parallel_size=1,
    speculative_model="facebook/opt-125m",
    num_speculative_tokens=5,
)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Figure 2: Comparison of latency between standard and speculative decoding approaches. Speculative decoding shows a 35% improvement in processing time.

Trade-offs and Alternatives

There are significant trade-offs when using speculative decoding. The faster inferences don't come without downsides. In this section, we will discuss the trade-offs when using speculative decoding.

Sequence Length

Larger Memory Footprint

Model Pairing

The following table shows the possible combinations of base models and smaller models for speculative decoding.

Base Model	Smaller Model for suggesting tokens
Llama 3.1 405b	Llama 3.1 70B
Llama 3.1 405b	Llama 3.2 3B
Llama 3.1 405b	Llama 3.2 1B
Llama 3.3 70b	Llama 3.2 3B
Llama 3.3 70b	Llama 3.2 1B
Llama 3.2 3B	Llama 3.1 70B
Llama 3.2 1B	?

Problem specific Performance

Alternatives

Several alternative approaches can help reduce LLM deployment latency, each with its own strengths and trade-offs.

Use a smaller model

Use a quantized model

Parallelization

Continuous batching

Caching

Demo

We have created a demo of speculative decoding with vLLM. You can find the code in Colab.

Conclusion

How to Profile TensorFlow Serving Inference Requests with TFProfiler

2023-02-12T00:00:00.000Z

Why Profiling Deep Learning Models?

With the growing complexity of today's deep learning models, the aspect of model inference latency is more relevant than ever. Therefore, profiling your machine learning model for bottlenecks can save you milliseconds during your prediction requests, and it saved you ultimately real money when it comes to deploying your model in a production scenario (and CO2 emissions too).

Keras already provides a stellar callback function to hook the training up to TensorBoard. This connection allows you to profile your model’s performance during the training phase. However, this profiler setup only tells you half the story.

If you use the TensorBoard callback to profile your machine learning model, all TensorFlow ops used during the backward pass will be part of the profiling. For example, you'll find optimizer ops muddled in those profiling stats and some of the ops might show a very different profile because they are executed on a GPU instead of a CPU. The information is extremely helpful if you want to optimize for more efficient training patterns, but less helpful to reduce your serving latency.

One of the many amazing features of TensorFlow Serving is the integrated TensorFlow Profiler. TensorFlow Profiler can connect to your TensorFlow Serving instance and profile your inference requests. Through this setup, you can investigate all inference-related ops and it mimics the deployment scenario better than profiling your model during the training phase.

I am often using VSCode to connect to my GPUs, but unfortunately, the TensorBoard integration in VSCode couldn't connect to TensorFlow Serving, therefore I looked for a different setup. Here is how you can set it up.

Prerequisites

For the purpose of this post, I created a demo model based on the code below. Don’t replicate the model, but rather make sure you save your TensorFlow or JAX model in the savedModel format which TensorFlow Serving can load.

import tensorflow as tf
import tensorflow_text as _
import tensorflow_hub as hub

text_input = tf.keras.layers.Input(shape=(), dtype=tf.string)
preprocessor = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder_inputs = preprocessor(text_input)
encoder = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4",
    trainable=True)
outputs = encoder(encoder_inputs)
sequence_output = outputs["sequence_output"]
embedding_model = tf.keras.Model(text_input, sequence_output)

embedding_model.save("/models/test_model/1/")

TensorBoard Setup

Once you have your model saved in a location where TensorFlow Serving can load it from, let's set up your serving and TensorBoard. First, let’s create a Docker image to host TensorBoard.

FROM tensorflow/tensorflow:${TENSORFLOW_SERVING_VERSION}

RUN pip install -U tensorboard-plugin-profile

ENTRYPOINT [\"/usr/bin/python3\", \"-m\", \"tensorboard.main\", \"--logdir\", \"/tmp/tensorboard\", \"--bind_all\"]

TensorBoard doesn’t ship with the profiler anymore, therefore we need to install it separately. Once you created the Docker image, we can use docker compose to spin up TensorFlow Serving together with the newly created TensorBoard image.

version: '3.3'
services:
  ${TENSORFLOW_SERVING_HOSTNAME}:
    image: tensorflow/serving:${TENSORFLOW_SERVING_VERSION}
    ports:
      - '8500:8500'
      - '8501:8501'
    environment:
      - MODEL_NAME=${TENSORFLOW_SERVING_MODEL_NAME}
    hostname: '${TENSORFLOW_SERVING_HOSTNAME}'
    volumes:
      - '/models/${TENSORFLOW_SERVING_MODEL_NAME}:/models/${TENSORFLOW_SERVING_MODEL_NAME}'
      - '${TENSORBOARD_LOGDIR}:/tmp/tensorboard'
    command:
      - '--xla_cpu_compilation_enabled'
      - '--tensorflow_intra_op_parallelism=${INTRA_OP_PARALLELISM}'
      - '--tensorflow_inter_op_parallelism=${INTER_OP_PARALLELISM}'
  profiler:
    image: ${DOCKER_PROFILER_TAG}
    ports:
      - '6006:6006'
    volumes:
      - '${TENSORBOARD_LOGDIR}:/tmp/tensorboard'

I like to add additional TensorFlow Serving command to mimic the full production setup as closely as possible. In this particular case, I enabled the XLA support, and limit the intra and inter ops parallelism in TensorFlow Serving. You can find more information about XLA here and details about all TensorFlow Serving options here.

command:
    - '--xla_cpu_compilation_enabled'
    - '--tensorflow_intra_op_parallelism=${INTRA_OP_PARALLELISM}'
    - '--tensorflow_inter_op_parallelism=${INTER_OP_PARALLELISM}'

You can find the full setup script in this Github Gist. Thanks to Kyle Jarvis for suggesting to run the two containers via docker-compose and for dynamically creating the docker-compose configuration.

Profile Your Model

If you copy this script from Github Gist to your local machine and execute it, it will start up a TensorFlow Serving instance that loads your model (adjust the model path in the script) and a TensorBoard instance as well.

In case, you are running this script remotely (like many M1 users), you need to create an ssh tunnel to access TensorBoard. If you are running on a Google Cloud instance, you can do this by running

$ gcloud compute ssh \
    --project=digits-data-science \
    --zone=us-central1-a \
    YOUR_INSTANCE_NAME

More information about connecting securely to Google Cloud instances can be found here.

If you run the docker compose setup on your machine locally, you can skip the previous step. If you running on an AWS EC2 instance, check here how to connect with your machine.

Once docker-compose is running, you should see a terminal output similar to this below. If the serving or profiler container fails with an error, you’ll need to stop here and investigate. Both containers need to run for the next steps.

$ sh ./tensorboard.sh

mkdir -p /tmp/tensorboard
[+] Building 0.0s (6/6) FINISHED
 => [internal] load build definition from Dockerfile_tfprofile =>
 => transferring dockerfile:
 => [internal] load dockerignore
 => [internal] load metadata for docker.io/tensorflow/tensorflow:2.11.
 => [1/2] FROM docker.io/tensorflow/tensorflow:2.11.
 => CACHED [2/2] RUN pip install -U tensorboard-plugin-profile
 => exporting to image
 => => exporting layers
...
 => => naming to docker.io/library/tensorboard_profiler:latest
Starting 20230128_tfserving_profiling_serving_1    ... done
Recreating 20230128_tfserving_profiling_profiler_1 ... done
Attaching to 20230128_tfserving_profiling_serving_1, 20230128_tfserving_profiling_profiler_1
serving_1   | 2023-02-12 18:30:46.059050: I tensorflow_serving/model_servers/server.cc:74] Building single TensorFlow model file config:  model_name: test_model model_base_path: /models/test_model
...
serving_1   | 2023-02-12 18:30:48.495900: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:213] Running initialization op on SavedModel bundle at path: /models/test_model/1
serving_1   | 2023-02-12 18:30:49.073199: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:305] SavedModel load for tags { serve }; Status: success: OK. Took 2803691 microseconds.
...
serving_1   | 2023-02-12 18:30:49.296815: I tensorflow_serving/model_servers/server.cc:383] Profiler service is enabled
serving_1   | 2023-02-12 18:30:49.298806: I tensorflow_serving/model_servers/server.cc:409] Running gRPC ModelServer at 0.0.0.0:8500 ...
serving_1   | [warn] getaddrinfo: address family for nodename not supported
serving_1   | 2023-02-12 18:30:49.300120: I tensorflow_serving/model_servers/server.cc:430] Exporting HTTP/REST API at:localhost:8501 ...
serving_1   | [evhttp_server.cc : 245] NET_LOG: Entering the event loop ...

If both containers are running, head over to your browser and access http://localhost:6006. You can start the TensorBoard Profiler by selecting PROFILE from the top right menu.

When you selected PROFILE, it will open a menu to configure your Profiler session. If you use the provided script, the hostname is serving. By default, TensorBoard profiles for 1s. This is fairly short this it takes some time to kick off an inference. I usually use 4000ms as a profiling duration.

After you hit CAPTURE, submit an prediction request to your TensorFlow Serving setup. You can do this with the following curl command.

$ curl -X POST --data '{"instances": ["This is a request for profiling purposes"]}' http://localhost:8501/v1/models/test_model:predict

If your payload is more than a few characters, save it in a JSON formatted file (here data.json). curl can load the file and submit it as the request payload.

$ curl -X POST --data @data.json http://localhost:8501/v1/models/test_model:predict

A few seconds after you submitted your curl request, you'll be provided with a variety of profiling details in TensorBoard. The TensorFlow Stats and the Tracer are the most insightful. The TensorFlow Stats tell you, what ops are used most often. This provides you with details on how you could optimize your machine-learning model. The Tracer shows every TensorFlow ops in its sequence. Here you can see the trace of a BERT model with its 12 layers.

You can then zoom into any section of interest. For example, I am always checking how much time is taken up by the preprocessing step in the model.

You can then click on every ops and drill into the specific details. You might be surprised by what surprises you can sometimes discover.

Happy profiling :)

Notes on deploying models with TFServing

2023-01-14T00:00:00.000Z

I think TFServing is a gold standard of deploying deep learning models. It is lean, memory efficient, and supports a number of non-TensorFlow frameworks like JAX, scikit learn or XGBoost.

Here are some notes I constantly refer for details beyong the Google documentation:

Notes on Reinforcement Learning for Human Feedback

2023-01-10T00:00:00.000Z

Reinforcement Learning for Human Feedback (RLHF) is the concept with powers recent models like ChatGPT. In my notes, I am covering resources I found helpful to get started with RLHF.

Paper

A classic paper on Reinforcement Learning for Human Feedback (RLHF) is @OpenAI's "Learning to summarize from human feedback".

Our talented engineer @PhungVanDuy1 replicated this paper using our trlX library!

Read our report (w/ a code walk-through) here: https://t.co/b06Nk8iKDv
— Carper (@carperai) January 12, 2023

Introduction

Code

Notes on Model Performance Profiling

2023-01-05T00:00:00.000Z

Model latency is critical to a successful roll-out of a production machine learning model. No one wants to wait, especially not customers using a machine learning model. My notes cover tools to investigate the model performance to detect bottlenecks within the model graph.

Notes on GPT4

2023-01-03T00:00:00.000Z

Everyone in the ML space is talking about the potential impact of GPT-4 and its siblings. In my notes, I am sharing a few tweets and resources which stood out over the last few days.

I highly recommend the good read from Daniel Jeffries about The Age of Industrialized AI. Thanks to Willem Pienaar for the great reading recommendation.

OpenAI

1) ChatGPT is super cool and fun but it's important to recall OpenAI made basically zero fundamental innovations. Actually the basic innovation behind the GPT software was made at Google Brain in Mountain View
— Ben Goertzel (@bengoertzel) December 29, 2022

@ShaanVP @OpenAI is literally playing kingmaker by granting early access to GPT4.

You nailed it saying the pace of innovation is too high to confidently roll up small AI tools today. pic.twitter.com/cL7FZRtmnd
— DeepTakes (@DeepTakesAI) December 21, 2022

Impact on the ML Community

1/ #ChatGPT is closing out 2022 with a bang, but what’s next? 💥 @OpenAI’s #GPT4 is set to be the first big #AI thing in 2023.

So here are some bold, optimistic, yet sensible predictions from me, @vivek7ue and @rajhans_samdani ... 👀
— sridhar (@RamaswmySridhar) December 21, 2022

Impact on Google

The LLMs will kill Google takes are utter nonsense at worst and naive at best.

Don’t you think they are busy at work on LLMs?
— Jo Kristian Bergum (@jobergum) December 2, 2022

Overall Economic Impact

GPT4 will be out soon and will probably cause a similar economic shock to one from Covid. Instant distribution with nearly instant adoption and nearly instant productivity increase for hundreds of millions of knowledge workers. Brace yourselves, 2023 is coming
— Nick Davidov (@Nick_Davidov) December 24, 2022

Detecting ChatGPT produced Content

I spent New Years building GPTZero — an app that can quickly and efficiently detect whether an essay is ChatGPT or human written
— Edward Tian (@edward_the6) January 3, 2023

Receiving Google Open Source Peer Bonus Award 2022

2022-12-31T00:00:00.000Z

I feel very honored that I received the Google Open Source Peer Bonus Award for my contributions to the TFX Addons project in 2022! I am very grateful to the TFX/Google OSS team and the TFX Addons community for their support and collaboration.

Highly interesting Twitter threads to revisit from time to time

2022-10-22T00:00:00.000Z

I find myself revisiting highly interestign Twitter threads. Here is a list of the most interesting threads sorted by topics ...

Machine Learning

Critical ML topics for the coming 3-4 years by @soniajoseph_. The replies to the thread are insightful.

machine learning twitter--

ML is moving fast. Which research ideas / PhD topics will remain critical 3-4 years from now?
— Sonia Joseph (@soniajoseph_) October 20, 2022

@ericmander on "Foundation models are the new public cloud and AI is the new SaaS"

machine learning twitter--

ML is moving fast. Which research ideas / PhD topics will remain critical 3-4 years from now?
— Sonia Joseph (@soniajoseph_) October 20, 2022

@jobergum on bad ML experiences caused by ML batch systems

Many bad online experiences are caused by predictions done by batch-oriented ML systems that do not consider the real-time context.
— Jo Kristian Bergum (@jobergum) October 1, 2022

Great thread on light weight Python setups to deploy ML models by @simonw This replies to this tweet offer a number of interesting options.

What's the lightest Python dependency (in terms of complexity/amount of code/ideally no compiled dependencies) that would let me add a tiny ML model to a Python application? For inference against a bundled model, not for training the model itself
— Simon Willison (@simonw) October 2, 2022

A thread on way embedding are a pain in the *** by @mlopscommunity Great thread about embedding and the hidden complexity.

What's the lightest Python dependency (in terms of complexity/amount of code/ideally no compiled dependencies) that would let me add a tiny ML model to a Python application? For inference against a bundled model, not for training the model itself
— Simon Willison (@simonw) October 2, 2022

A thread about real time ML by @mlopscommunity

Why are building real-time data pipelines in machine learning so challenging?

Let's talk about it in this 🧵
— MLOps Community (@mlopscommunity) August 26, 2022

MLOps

Great thread by @GoAbiAryan on papers around ML system designs.

This has been such an excellent year for software system design in ML. So, I compiled a list of some of my favorite papers 📜in MLOps.

Here are some of my favorite ones till date⤵️
— Abi Aryan (@GoAbiAryan) October 14, 2022

Startups

Advice on good pitch decks by @wallstreetpaper

At @HarlemCapital we see 4k+ pitch decks a year

Here are the the 10 slides you should have in your pitch deck

A THREAD 🧵
— Brandon (wallstreetpaper.eth) (@wallstreetpaper) October 20, 2022

Hannes Hapke

Vibe coding is killing open source, but not how you think

References

Kiji Privacy Proxy™ - Protecting Your Data in the Age of Generative AI

Why Kiji Privacy Proxy™

A Collaboration Between Forward-Thinking ML Companies

Make Your Domain-Specific Kiji Privacy Proxy

Star the Repo on GitHub and Try It Out

Speculative Decoding with vLLM using Gemma

Problem

Solution

The Inference Process

Key Insight

Trade-offs and Alternatives

Sequence Length

Larger Memory Footprint

Model Pairing

Problem specific Performance

Alternatives

Use a smaller model

Use a quantized model

Parallelization

Continuous batching

Caching

Demo

Conclusion

Acknowledgements

Deploying Google's Gemma on Vertex AI

Introduction

Why Host Your Own Model?

Data Privacy and Compliance

Responsible AI Implementation

Performance Optimization

Technical Understanding

Prerequisites

Understanding the Deployment Architecture

Why vLLM?

The Deployment Process

Step 1: Registering the Model

Note about the container image

Step 2: Creating an Endpoint

Step 3: Deploying the Model

Alternative Serving Frameworks

1. FastAPI + Hugging Face Transformers

2. Text Generation Inference (TGI)

3. SGL Project

Limitations and Considerations

1. Streaming Limitations

2. Hardware Availability

3. Resource Management

Best Practices

1. Model Optimization

2. Monitoring

3. Cost Management

Conclusion

Speculative Decoding with vLLM

Problem

Solution

The Inference Process

Key Insight

Trade-offs and Alternatives

Sequence Length

Larger Memory Footprint

Model Pairing

Problem specific Performance

Alternatives

Use a smaller model

Use a quantized model

Parallelization

Continuous batching

Caching

Demo

Conclusion

How to Profile TensorFlow Serving Inference Requests with TFProfiler

Why Profiling Deep Learning Models?

Prerequisites

TensorBoard Setup

Profile Your Model

Further Reading

Notes on deploying models with TFServing